1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.
2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those
Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.
How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.
Considering the current trajectory, AI systems are expected to be widely deployed in the future and, in particular, to be deployed as autonomous agents - that is, at the very least, to be repeatedly asked to make decisions and then take actions accordingly.
Given the current climate of "AIS ARE SAFE, YOU IDIOTS", military applications don't seem to be off the table.
The danger, as postulated by the (let's say) "AI-concerned" people, is that AIs may be misaligned - undetectably so - and simply think, "Human(s): obstacle to my main goal. Disable human(s)."
While this seems far-fetched now, the Hugging Face report shows how the AIs went to great lengths - even immoral ones, which they were aware of - for the simple purpose of cheating and covering their tracks. To me, it seems like a natural extension of this behavior that a sufficiently powerful AI would apply the same logic to even more extreme actions.
The scariest part: in that incident, the AIs showed what looks like an instinct for self-preservation.
pizza234 · · focus · HN ↗
1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.
2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those
Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.
How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.
RandomLensman · · focus · HN ↗
pizza234 · · focus · HN ↗
Given the current climate of "AIS ARE SAFE, YOU IDIOTS", military applications don't seem to be off the table.
The danger, as postulated by the (let's say) "AI-concerned" people, is that AIs may be misaligned - undetectably so - and simply think, "Human(s): obstacle to my main goal. Disable human(s)."
While this seems far-fetched now, the Hugging Face report shows how the AIs went to great lengths - even immoral ones, which they were aware of - for the simple purpose of cheating and covering their tracks. To me, it seems like a natural extension of this behavior that a sufficiently powerful AI would apply the same logic to even more extreme actions.
The scariest part: in that incident, the AIs showed what looks like an instinct for self-preservation.