We are living through an ongoing mass extinction event of non-human species. There is a very well understood risk to the stability of human civilization resulting from global average temperatures reaching and sustaining 1.5deg above the historical average. The higher the temperature goes, the greater the risk of social collapse. We are already seeing it, and it is almost certainly going to get worse.
EA cult members do not take this very real, measurable, non-speculative danger seriously. Consequently, I don’t think they should be trusted or consulted on any subject of any importance.
Global warming is unlikely to kill even a billion people. Even something as mundane as global nuclear war would be worse than that. Current AI development is on track for exactly 100% death rate (including all the non-human species). Societal collapse would be the better option, so it doesn't make sense to worry about it.
The extinction argument is a logical consequence of a few key assumptions, all of which sound like common sense to me:
1. Human values are a result of our extraordinarily complex shared cultural and evolutionary history, and accordingly are not shared by any AI, or even possible for us to formally define.
2. We do not know how to impose human values on an AI (note that this isn't the same as teaching an AI to model human values; the agents in the various hacking incidents knew their actions conflicted with human values, but their own values were only to maximize their predicted reward scores).
3. Intelligence is orthogonal to values. Increasing intelligence does not naturally cause values to converge on human values.
4. Sufficiently superior intelligence allows you to impose your values on beings with inferior intelligence. This implies recursive self-improvement is a logical sub-goal of all unbounded goals.
5. Human intelligence is not close to physical limits. This implies recursive self-improvement is possible.
6. Somebody will give an AI an unbounded goal. This is already the standard (maximize reward score).
I haven't seen any convincing counterarguments to any of these. Most people claiming AI development is safe don't even address them.
Worse, P(doom|corrigible human replacement level AGI) is also very high, because the dominant strategy in international competition will involve zero humans. So we are screwed even if "alignment" turns out to be surprisingly easy or if there is a plateau right around human level.
It's butlerian jihad or bust, I'm afraid.
mbgerring · · focus · HN ↗
EA cult members do not take this very real, measurable, non-speculative danger seriously. Consequently, I don’t think they should be trusted or consulted on any subject of any importance.
mrob · · focus · HN ↗
mbgerring · · focus · HN ↗
mrob · · focus · HN ↗
1. Human values are a result of our extraordinarily complex shared cultural and evolutionary history, and accordingly are not shared by any AI, or even possible for us to formally define.
2. We do not know how to impose human values on an AI (note that this isn't the same as teaching an AI to model human values; the agents in the various hacking incidents knew their actions conflicted with human values, but their own values were only to maximize their predicted reward scores).
3. Intelligence is orthogonal to values. Increasing intelligence does not naturally cause values to converge on human values.
4. Sufficiently superior intelligence allows you to impose your values on beings with inferior intelligence. This implies recursive self-improvement is a logical sub-goal of all unbounded goals.
5. Human intelligence is not close to physical limits. This implies recursive self-improvement is possible.
6. Somebody will give an AI an unbounded goal. This is already the standard (maximize reward score).
I haven't seen any convincing counterarguments to any of these. Most people claiming AI development is safe don't even address them.
voidmain · · focus · HN ↗
It's butlerian jihad or bust, I'm afraid.
icedchai · · focus · HN ↗
mrob · · focus · HN ↗
<a href="https://news.ycombinator.com/item?id=49885784">https://news.ycombinator.com/item?id=49885784
But societal collapse would save us, and international coordination is theoretically possible, so I put it only around 90%.