This whole "we can't trust the people warning us because they're either weirdos or involved" deal is such a strange angle when you can just start from zero, look at what's known to be happening right now, and draw a very straight line to catastrophe.
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
The doomer argument is straightforward: a powerful AI will have goals. If those goals don’t include the welfare of humanity, a sufficiently powerful AI will kill us either by accident, by indifference, or intentionally when we get in its way. Making an AI that includes the welfare of humanity in its goals is incredibly hard.
That’s it. To refute the doomer argument, you can refute any part of that. I notice that none of the people bringing up fanfics, cults, and weirdness are doing that. No, they’re instead trying to convince people that doomers are uncool. Since only cool people can know things, QED.
We don't have this kind of AI, and (as of right now), unless humans link the AIs we do have up to a bunch of killer robots (i.e. drones) then this is vanishingly unlikely.
Like, any scenario where AI kills us all requires us to give it the tools that it needs to do this (as well as a desire to do this, as right now they just attempt to generate text to solve a problem/question they are posed).
No one is saying we have it, they’re saying they’re building it.
And what mechanism to control these drones, which are autonomous or remote-controlled machines, do you imagine that humans will be able to use, but AI won’t?
And what in the history of the past five years, gives you confidence that not a single human will give the AI the access it needs to do bad things? Incompetence alone is sufficient to kill us, let alone humans that genuinely, or confusedly, want bad things.
It’s reasonable to cast doubt upon the fitness of ideas based on their sources, particularly when the ideas are hard to adjudicate. It reasonable to filter what one engages with partially based on the sources.
arcticfox · · focus · HN ↗
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
ribs · · focus · HN ↗
afthonos · · focus · HN ↗
The doomer argument is straightforward: a powerful AI will have goals. If those goals don’t include the welfare of humanity, a sufficiently powerful AI will kill us either by accident, by indifference, or intentionally when we get in its way. Making an AI that includes the welfare of humanity in its goals is incredibly hard.
That’s it. To refute the doomer argument, you can refute any part of that. I notice that none of the people bringing up fanfics, cults, and weirdness are doing that. No, they’re instead trying to convince people that doomers are uncool. Since only cool people can know things, QED.
disgruntledphd2 · · focus · HN ↗
We don't have this kind of AI, and (as of right now), unless humans link the AIs we do have up to a bunch of killer robots (i.e. drones) then this is vanishingly unlikely.
Like, any scenario where AI kills us all requires us to give it the tools that it needs to do this (as well as a desire to do this, as right now they just attempt to generate text to solve a problem/question they are posed).
afthonos · · focus · HN ↗
And what mechanism to control these drones, which are autonomous or remote-controlled machines, do you imagine that humans will be able to use, but AI won’t?
And what in the history of the past five years, gives you confidence that not a single human will give the AI the access it needs to do bad things? Incompetence alone is sufficient to kill us, let alone humans that genuinely, or confusedly, want bad things.
ribs · · focus · HN ↗
It would be foolish to ignore sources.
afthonos · · focus · HN ↗
The ideas here are not complicated. The argument is three sentences.