The framing of how we think about AI has really been cultivated by a quite homogeneous group of people. These people, like all people that belong to a sub-culture, are almost certainly prone to groupthink.
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
So what you're saying is that we lack a reproduction study?
Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:
Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.
So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).
throwaway13337 · · focus · HN ↗
It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
benlivengood · · focus · HN ↗
Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:
Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.
So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).