Thread

The Download: why AI's latest breakthroughs and fears may be more hype than rea

49 points · 128 comments · joozio

bananaflag · 1790163465 · focus · HN ↗
> Describing them as “superintelligence” or “rogue models” ascribes agency to products rather than to the companies building them. This framing markets these companies’ products as “superhuman” and, at the same time, helps the companies evade accountability for their actions.
If someone discovered how to summon demons to aid them in robbing banks the main issue wouldn't be "but who has the responsibility for the crime, the human or the demon", it would be "OMG DEMONS".
Seriously, I don't get what sort of world these people are living in.
1. rubendev · 1790164184 · focus · HN ↗
  That's not what the LLM hacking accidents have been like at all though. To improve the analogy, it would be like summoning a totally passive demon, giving it weapons and placing it next to a bank, place a small fence around it, and then command it to perform a totally safe "exercise" that is exactly like robbing a real bank.
  LLMs do not have agency, they are just producing tokens based on a prompt that a person entered, and some of these tokens can trigger the tools that a person gave them access to.
  1. 0xDEAFBEAD · 1790165467 · focus · HN ↗
    Dwarkesh, for one, defended his use of "anthropomorphic" language.
    >Sacrificing now yields Oracle for team, but forfeits our chance, question mark. But other agents were pushing it, sending a message saying, go, sacrifice final now. And then EarlyBig eventually agreed, thinking to itself, our own utility may be already near zero. Sacrifice rational.
    <a href="https://www.youtube.com/watch?v=X50zezLFWWI#t=2m" rel="nofollow">https://www.youtube.com/watch?v=X50zezLFWWI#t=2m
    My suspicion is that many of the "LLMs do not have agency" folks just haven't learned much about the details of the incident. It was specifically with LLM agents that were trained to be more persistent than usual.
    If you're going to say that the incident details don't matter, and LLMs lack agency because it's all based on floating-point math--why can't I say that humans lack agency, because it's all based on neurons firing?
    embedding-shape · 1790166088 · focus · HN ↗
    > If you're going to say that the incident details don't matter, and LLMs lack agency because it's all based on floating-point math--why can't I say that humans lack agency, because it's all based on neurons firing?
    They're not saying this, we're saying LLMs lack agency because if you run a LLM and don't send any prompts, literally nothing happens.
    Instruct it to "Find the right answer regardless of where", it'll do exactly this. They're passive in that they don't act by themselves, somewhere, at one point, someone "told" the LLM to "do something" and that's the cause and the reason for saying "LLMs do not have agency".
    0xDEAFBEAD · 1790166732 · focus · HN ↗
    Imagine a super-competent Navy SEAL who just sleeps in the barracks unless his commander tells him to do something. Does the Navy SEAL lack agency? As a target of this Navy SEAL, should you be reassured by the fact that they'll be sleeping in the barracks unless their commander tells them to do something?
    diidjdicksodksk · 1790168268 · focus · HN ↗
    No, the Navy SEAL does not lack agency in this scenario because they are still a person with individual agency, they can act on their own accord without anyone prompting them to, but in this case their superior instructed them to stay put. He could rebel and go rogue, but that would mean he used his own personal agency to defy orders given to him, which again places responsibility on the actor that actually possesses agency.
    0xDEAFBEAD · 1790170160 · focus · HN ↗
    Suppose this SEAL is very obedient by disposition so the probability of him going rogue is akin to the probability of an LLM hallucinating or whatever.
    tavavex · 1790182865 · focus · HN ↗
    No matter how you slice it, it's not possible to equate humans to tools like this. If the SEAL receives an order to stand at attention until told otherwise, he will not stay in place for weeks until starving. An algorithm will work towards its self-destruction if ordered to - it doesn't care, it doesn't have the biological signals telling it otherwise. If the SEAL is lost somewhere with no one to give him orders, he will soon start acting on his own to ensure his survival and comfort. A tool will not do anything until directly activated and used by something that does have an active will, like a human. If the SEAL is ordered to massacre his hometown, no matter how obedient he is, he might have reservations. An algorithm with no instincts to get in the way will do anything if this behavior isn't somehow inhibited by people during training. It wouldn't even need the explicit order, if an irresponsible operator tells it to accomplish something by any means, that means that anything is on the table. Good thing the AI labs aren't stuffed to the brim with irresponsible operators.
    0xDEAFBEAD · 1790224661 · focus · HN ↗
    What if I prompt the AI to behave like the SEAL in the relevant ways, and fine-tune until it does so?
    tavavex · 1790252463 · focus · HN ↗
    Then these actions are downstream of your will, not the AIs. It doesn't change anything, you'd just be playing with an imitation. Ordering it to pretend to be like a human does not a human equivalent make.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.