Thread

1. reducesuffering · 1791073971 · focus · HN ↗

   “I believe there’s about a 50% chance we all die because of the development of smarter-than-human AI systems, and that our actions over the next two to 10 years will determine the outcome.”

   There are a gargantuan number of extremely intelligent AI researchers and Turing Award winners saying the same thing. They are the ones closest to understanding the technology.

   Where there’s smoke there’s fire.

   1. biophysboy · 1791078148 · focus · HN ↗

      Where does 50% come from? It is meaningless if the probability model is not explained.

      1. 0xDEAFBEAD · 1791079843 · focus · HN ↗

         One could also use language like "a decent chance". But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways. For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.

         1. biophysboy · 1791081866 · focus · HN ↗

            > But research has shown that people translate vague phrases like "a decent chance" into probabilities in inconsistent ways.

            That is not a bad thing. It captures uncertainty, unlike the fake number.

            > For an ML researcher who is already used to dealing in next-token probabilities that aren't rigorously determined, just stating a probability estimate directly is very natural.

            Exactly, it is a rhetorical device to persuade a technically-inclined audience. It works because it implies that a quantitative model exists. I want a clear, incisive set of mathematical arguments. Otherwise, I’m ignoring predictions as the ramblings of arrogant idiot rich kids.

            1. danielmarkbruce · 1791082208 · focus · HN ↗

               Saying 50% likely very clearly implies there is no quantitative model to anyone who deals with probability, predictions, gambling, financial markets, ML/AI and so on. The lack of precision is something to pay attention to.

               You may decide that the person doesn't know what they are talking about, but that's a very different issue.

               1. biophysboy · 1791088611 · focus · HN ↗

                  Uninformative priors are a part of Bayesian models, which are useless if they cannot be updated with real or simulated data?

                  1. danielmarkbruce · 1791091115 · focus · HN ↗

                     I estimate a 0.1% chance he intended it to be an uninformative prior.

                     1. biophysboy · 1791111358 · focus · HN ↗

                        People start with this prior in the examples you gave when they are completely uncertain. It is not a good starting point for an event that has never happened and can only happen once.

                        1. danielmarkbruce · 1791127896 · focus · HN ↗

                           Maybe people you interact with do. In my world, people don't say 50% when they have no idea, they say they have no idea. Saying "no idea" might effectively be saying "uniform distribution over all probabilities" which yields 50%, but at least with people I interact with, the reverse certainly is not true.

                           Either way - let me clarify - this guy is guessing, using his brain, that it's a 50% chance. He is absolutely not saying "i don't know" or "uniform distribution" or anything of that nature.

         2. danielmarkbruce · 1791082261 · focus · HN ↗

            For anyone who gambles, or deals with financial markets it's also a very natural way to speak.
      2. danielmarkbruce · 1791082037 · focus · HN ↗

         It's not meaningless. He said he believes it's 50% likely. It's a remarkably clear statement, and the probililty model is his brain.

Unofficial Hacker News client; not affiliated with Y Combinator.