1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.
2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those
Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.
How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.
> however, the mechanical grounds for it to happen are plausible
If you build a control systems for firing a gun, then coupled it with an RNG, the mechanical grounds for it to kill a person is plausible.
LLMs are text generators. They are not repositories of knowledge. The mistake is coupling them with actuators (tool call) or having humans interpreting the generated text as facts.
> LLMs are text generators. They are not repositories of knowledge.
This the take of people who have stopped reading about LLMs in 2023 or so (you forgot to mention the stochastic parrot, by the way).
If you have a bit of attention and interest to make informed conversations, read this report first: <a href="https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation" rel="nofollow">https://metr.org/blog/2026-08-26-openai-hugging-face-inciden....
> This the take of people who have stopped reading about LLMs in 2023 or so
Ad Hominem attacks make for great counterpoints /s
Whatever you may say, it's a text generators on top of a tool calling framework. Training may skew the text towards a particular text, but as with all ML technologies (and statistics based methods) there's always a good chance of errors on a particular sample task.
With standard control systems, we tried to incorporate the error into the actual control output in order to minimize it. This is done in a deterministic manner. There's still risk of failure so we design systems around them.
With control systems powered by LLM (agent harness), errors are often not taken into account and they are amplified in most sessions. Safety measures are close to nonexistent. The issue is not the failure mode, the issue is that it's preventable and there were not a lot done to prevent it.
pizza234 · · focus · HN ↗
1. AI agents are not just reactive systems. Their use is expanding toward continuous decision-making/monitoring information, which means, they make decisions and take actions with limited human intervention.
2. AI agents do absolutely have goals/tasks ("motivation" can be excessively antropomorphic), both primary (assigned) and secondary (self-assigned), and what surprised researchers is that self-preservation can be one of those
Mechanically speaking, the scenario (that is, how theorized by Hinton etc., which the OP didn't understand) is that a sufficiently powerful AI may decide that in order to achieve its goals/tasks (e.g. continuous research/development and/or survival from termination), humans may be a danger, therefore it may decide to take actions that endanger humanity.
How it can happen or what's the likelyhood is not in the scope of the topic, however, the mechanical grounds for it to happen are plausible.
skydhash · · focus · HN ↗
If you build a control systems for firing a gun, then coupled it with an RNG, the mechanical grounds for it to kill a person is plausible.
LLMs are text generators. They are not repositories of knowledge. The mistake is coupling them with actuators (tool call) or having humans interpreting the generated text as facts.
pizza234 · · focus · HN ↗
This the take of people who have stopped reading about LLMs in 2023 or so (you forgot to mention the stochastic parrot, by the way).
If you have a bit of attention and interest to make informed conversations, read this report first: <a href="https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation" rel="nofollow">https://metr.org/blog/2026-08-26-openai-hugging-face-inciden....
skydhash · · focus · HN ↗
Ad Hominem attacks make for great counterpoints /s
Whatever you may say, it's a text generators on top of a tool calling framework. Training may skew the text towards a particular text, but as with all ML technologies (and statistics based methods) there's always a good chance of errors on a particular sample task.
With standard control systems, we tried to incorporate the error into the actual control output in order to minimize it. This is done in a deterministic manner. There's still risk of failure so we design systems around them.
With control systems powered by LLM (agent harness), errors are often not taken into account and they are amplified in most sessions. Safety measures are close to nonexistent. The issue is not the failure mode, the issue is that it's preventable and there were not a lot done to prevent it.