This whole "we can't trust the people warning us because they're either weirdos or involved" deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
The doomer argument is straightforward: a powerful AI will have goals. If those goals don’t include the welfare of humanity, a sufficiently powerful AI will kill us either by accident, by indifference, or intentionally when we get in its way. Making an AI that includes the welfare of humanity in its goals is incredibly hard.
That’s it. To refute the doomer argument, you can refute any part of that. I notice that none of the people bringing up fanfics, cults, and weirdness are doing that. No, they’re instead trying to convince people that doomers are uncool. Since only cool people can know things, QED.
It’s reasonable to cast doubt upon the fitness of ideas based on their sources, particularly when the ideas are hard to adjudicate. It reasonable to filter what one engages with partially based on the sources.
arcticfox · · focus · HN ↗
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
ribs · · focus · HN ↗
afthonos · · focus · HN ↗
The doomer argument is straightforward: a powerful AI will have goals. If those goals don’t include the welfare of humanity, a sufficiently powerful AI will kill us either by accident, by indifference, or intentionally when we get in its way. Making an AI that includes the welfare of humanity in its goals is incredibly hard.
That’s it. To refute the doomer argument, you can refute any part of that. I notice that none of the people bringing up fanfics, cults, and weirdness are doing that. No, they’re instead trying to convince people that doomers are uncool. Since only cool people can know things, QED.
ribs · · focus · HN ↗
It would be foolish to ignore sources.
afthonos · · focus · HN ↗
The ideas here are not complicated. The argument is three sentences.