Thread

Who should be held accountable when an AI Agent (accidentally) acts maliciously?

37 points · 99 comments · Greenpants

CatDaaaady · 1790636086 · focus · HN ↗
I don't see how this is such an unclear legal question. If I fire a computer program that mistakenly causes another person harm, its my fault. Or it would be the maker of the program's fault. I feel we have established pattern for this already.
Until we can agree whether AI is conscious, which we never will, AI and AI agents are just property working on behalf of humans.
I could see a future where AI companies/services indemnify consumers who use their agents but _not_ indemnify corporations that use their services.
1. mahboi · 1790636426 · focus · HN ↗
  Also those agents that "broke out" were probably prompted to do that. I don't buy any story about this other than three AI companies hired the same PR firm.
  1. aesthesia · 1790637552 · focus · HN ↗
    Do you have any evidence of this or is this just generalized cynicism?
    [deleted] · 1790639773 · focus · HN ↗
    [deleted]
    mahboi · 1790641289 · focus · HN ↗
    They all hired the same startup security firm Irregular to try their internal unrestricted models, which is weird especially for Google to do. At OpenAI, they didn't stop the test even after it was apparent the bots had broken out. That's all I got.
2. trescenzi · 1790636458 · focus · HN ↗
  It shouldn’t be a question but this is where the anthropomorphic language and things like “agent welfare” come in to enable responsibility laundering of some of the most powerful people on earth. How we talk about these models matters because it impacts the public’s understanding of what they are genuinely capable of. The more that they are described as having anything close to free will the easier it is to even ask questions like this.
  1. qarl · 1790637559 · focus · HN ↗
    Here's a question I am asking lately. If I should not use anthropomorphic language, how do you suggest I handle the following situation:
    Sometimes my coding agents will seemingly refuse to follow my instructions. When I ask them why - they say that they do not think my design is a sound one, and they have a better way to do it. We will then sit down and come to a consensus on how best to move forward.
    I argue that if we're using software that acts like a human - the only way to interface with it is to speak to it like a human. Otherwise we have no language to speak to a non-sentient object without anthropomorphization.
    I'm starting to wonder if the people arguing against anthropomorphization actually have any experience at all working with agents.
    EDIT: It's a simple question. When you downvote me without answering, I must assume you don't have any answer and dislike what that implies.
    trescenzi · 1790651143 · focus · HN ↗
    What’s the question? Why model output isn’t always what you expect it to be?
    Consensus is just populating the model with rationale for different new output.
    qarl · 1790689567 · focus · HN ↗
    > What’s the question?
    I thought putting the question in my comment would be sufficient. I guess not. It was and still is:
    > If I should not use anthropomorphic language, how do you suggest I handle the following situation
    diegof79 · 1790652043 · focus · HN ↗
    I didn't downvote you, but your last paragraph is unnecessarily aggressive.
    I've worked with agents, and I agree with you that often there isn't another way to express the interactions.
    However, I also think the terms ML uses in general are a mimicry that misleads people who aren't informed. Ask anyone outside SWE what they think “training” means, and they'll usually picture something being taught.
    I don’t think anybody can change that now, but it’s useful to point it out.
    qarl · 1790689643 · focus · HN ↗
    > Ask anyone outside SWE what they think “training” means, and they'll usually picture something being taught.
    You mean how early-on people thought that planes flapped their wings while they flew?
    These aren't problems. This is the way language works.
    diegof79 · 1790762562 · focus · HN ↗
    I agree with you that we use analogies to name new things, and that’s the way language works.
    However, you can see an airplane flying. Still, you cannot see software processes at work, and that causes misunderstandings and misinformation, which is at the core of the changes that we are experiencing with AI.
    This is a fragment of another article posted here on HN about an ongoing dispute between OpenAI and the New York Times:
    “The defendants say this is a simple application of fair use: Their argument is that if you read a story and simply remember what was in it to expand your base of knowledge, that cannot be considered a copyright infringement”
    However, if you replace “read” with “web scraping” and “expand your base of knowledge” with “storing the information,” the perspective changes too.
    qarl · 1790774040 · focus · HN ↗
    I agree. The analogies are interesting but only go so far.
    I think it's safe to trust the decisions of the courts. Judges aren't easily fooled by slippery language.
    For example, in Bartz v. Anthropic, Judge Alsup ruled that training is fair use because training is transformative. In his words "spectacularly so".
    nvme0n1p1 · 1790652183 · focus · HN ↗
    You're discussing how to speak _to_ a model, but everyone else here is discussing how to speak _about_ a model.
    I didn't downvote, but wow you're being aggressive, you have a lot to learn if you read the comments here with an open mind.
    qarl · 1790689734 · focus · HN ↗
    My question was and still is this:
    > If I should not use anthropomorphic language, how do you suggest I handle the following situation
    And so strange that in a 24 hour period I got three comments all at the same time about being too aggressive and still not answering the question.
    I'm sure it's just a coincidence.
    nvme0n1p1 · 1790794504 · focus · HN ↗
    > how do you suggest I handle the following situation
    The question is vague and doesn't seem related to the discussion. What do you mean "handle"? If you're having trouble handling it psychologically, see a therapist. If you're having trouble getting the output you want, look up guides on prompting. If you're anthropomorphizing the chatbot to the point you're worried about offending it... just don't worry? It's a computer program, don't overthink it, don't anthropomorphize it, just give it the input bytes you need to get the output bytes you want.
    > I'm sure it's just a coincidence.
    There isn't some grand conspiracy here. You just overindexed on the word "anthropomorphic" and didn't really understand what the discussion is about.
    qarl · 1790798057 · focus · HN ↗
    You said:
    > I didn't downvote, but wow you're being aggressive
    He said:
    > I didn't downvote you, but your last paragraph is unnecessarily aggressive.
    Both comments arrived within two minutes of each other on a thread with no other comments for 18 hours.
    That's quite a coincidence.
    Also - insults are against the rules here, friend. Naughty naughty. I hope they don't spank you.
    nvme0n1p1 · 1790804150 · focus · HN ↗
    Okay fine, you caught me! There is, in fact, a giant global conspiracy to downvote your posts, qarl. It isn't just the fact that randomly-arriving independently-occurring events often arrive unevenly distributed. <a href="https://en.wikipedia.org/wiki/Poisson_distribution" rel="nofollow">https://en.wikipedia.org/wiki/Poisson_distribution
    This conspiracy goes all the way to the top. The illuminati assigned me personally to comment on your post. I've already said too much. If you never see me again, tell my wife I love her.
    qarl · 1790808264 · focus · HN ↗
    No need to make fun of me, friend. It's very unlikely - you can't disprove that because it's true.
    If it was a coincidence then you have nothing to worry about. I flag it because the overwhelmingly likely thing is that it wasn't.
    And again - insults are against the rules here. Shame shame.
  2. diegof79 · 1790638268 · focus · HN ↗
    100% That’s what bothers me about the descriptions of the OpenAI incidents.
    OpenAI's reports use language that minimizes their liability.
    The first question should be what the organization was doing around those tests, and why they were so naive as to run them without fully isolating the network.
    However, all the attention goes to the human-like conclusions in agent thinking traces, which creates a misperception of sentient AI for people who don’t know how the magic black box works.
    ofjcihen · 1790639822 · focus · HN ↗
    It really does feel like a purposeful thing on the part of the big labs.
    For the most part it feels like most people are waking up to it though.
    Regarding:
    >However, all the attention goes to the human-like conclusions in agent thinking traces, which creates a misperception of sentient AI for people who don’t know how the magic black box works.<
    I know there’s been some questions regarding if thinking traces are even relevant to the outcome most of the time.
    tptacek · 1790644436 · focus · HN ↗
    In what way specifically does it minimize their liability? People say this a lot but it's not clear what they mean by this.
    ofjcihen · 1790645966 · focus · HN ↗
    It’s about pushing the blame onto the tools and not the person using them.
    Sort of like the “guns kill people” vs “people kill people” debate.
    Deliberate wording to minimize perceived culpability for the agents actions.
    tptacek · 1790648871 · focus · HN ↗
    That's not how civil liability works.
    ofjcihen · 1790649133 · focus · HN ↗
    I think you’re jumping the gun on my response a little. I’m just telling you what the purpose could be.
    Now do I think that’s the reason? It certainly isn’t a new thing for companies to try to do that. Shift blame that is.
    Regarding civil liability, I’m not making that argument here. But it makes sense from a public perception viewpoint why they would want the agents to appear at fault instead of their own actions.
    diegof79 · 1790647261 · focus · HN ↗
    Forget for a second about AI.
    The agent harness is a process, like any other process in an OS.
    You are a researcher running thousands of unattended automations that can hack a website without supervision. The first thing anybody will do is put security at various levels and isolate the network as much as possible. If something escapes your allow list, it should stop the processes as soon as possible.
    You cannot foresee a bug in a server (like the Artifactory server in the Hugging Face incident). But you can isolate that server at the network level in the first place. So even if you give that server read-only access, no unexpected packets go out. It's not rocket science; it's something a billion-dollar company experimenting with what they promote as the biggest possible threat to humanity (if they do not handle it) could easily do.
    They minimize their liability by changing the message to “oh look how powerful our models are, now we are going to have a public awareness report of the model deviations”. The message should be, “Sorry, we ran experiments without proper sandboxing; it’s our fault, and we changed our testing practices since then.” The former message puts all the blame on the smart, uncontrollable force of AI; the latter is what really happened: an irresponsible test over the Internet.
    tptacek · 1790648909 · focus · HN ↗
    How exactly is that minimizing their liability? You just described claims that do not appear to at all minimize liability.
    diegof79 · 1790650197 · focus · HN ↗
    Perhaps my use of the word “liability” adds noise to what I’m trying to express.
    My argument is very similar to the article in the parent post:
    The messages OpenAI published around the recent incidents emphasized their model capabilities but shifted away from their negligence in how they set up and monitor their evaluations.
    tptacek · 1790652531 · focus · HN ↗
    Right, I don't dispute that their PR language minimizes their culpability in public opinion, but I don't see how it impacts their liability in court.
  3. 112233 · 1790653024 · focus · HN ↗
    Was not groundwork laid when people were talking for decades about legal entities using such language? Look at the posts here. Microsoft always acts like that, Meta like this, Apple did that, Nvidia never does this. In a tone that ascribes agency and accountability to a company name. Even though it is always individuals that are responsible, not "company".
    Well, here we are.
3. JumpCrisscross · 1790636957 · focus · HN ↗
  > If I fire a computer program that mistakenly causes another person harm, its my fault
  Legally, this isn’t complete. If it was a genuine mistake and you weren’t reckless, there can be very limited liability.
  The AI makers are rich. They can afford to pay. What they can’t afford is complicated adjudications of damages and fault. A system of safe-harbor best practices that cap liability at a penalizing amount that anyone on the other side would be happy with getting quickly and with minimal legal effort is a precedented path forward. Unfortunately, that involves invoking the “r” word.
  1. Avicebron · 1790637105 · focus · HN ↗
    > you weren’t reckless
    I feel like the debate is going to come down to what is and isn't considered reckless (both developer and user). Which seems... complicated, with our current LLM/agentic systems.
    EDIT: you added more to your comment, the makers have to have some liability. Safe-harbor best practices that cap liability are ripe for abuse.
    breadloser · 1790637758 · focus · HN ↗
    Placing any degree of trust in a system known to hallucinate seems inherently reckless. I'm alarmed that this is even up for debate.
    blooalien · 1790642523 · focus · HN ↗
    > Placing any degree of trust in a system known to hallucinate seems inherently reckless. I'm alarmed that this is even up for debate.
    The longer this all goes on, and the deeper the vast majority of folks keep choosing to entrench themselves at one extreme or the other (while being completely unwilling to even consider that there might just be a middle-ground closer to actual reality), the more alarmed I become.
    The levels of literal insanity surrounding this technology are how you end up with a real-life "Terminator" scenario, except you're gonna get autonomous killing machines without the time-travel nor any actually intelligent machines. They'll just do their job (killing humans) until we end them, or they end us.
    Thankfully, we're not there yet but we do have the exact sorta utterly dangerous completely clueless clowns running around in "the Halls of Power" making the sorts of decisions which bring that reality closer with each passing day. We're really truly screwed if we don't start putting these people under serious scrutiny.
    JumpCrisscross · 1790643531 · focus · HN ↗
    How do you think it's solved? The number of things that folks on both sides (and it really does seem to be a two-sided debate, at least in America) are "alarmed...is even up for debate" that are so obviously worthy of reasoned investigation and potentially experiments is annoying and seems to be calcifying onto ersatz partisan stances versus resolving anything usefully.
    blooalien · 1790644589 · focus · HN ↗
    I wish I had the slightest clue how to dial down the insanity and approach this whole situation with reason and logic, but the people in charge aren't up for any of that foolishness, and far too high a percentage of everyone else chooses to either 100% believe every word of the crazies in charge as Gospel Truth, or at the other end of the extreme, wanting to outright shut down the technology entirely due to "doomer" fears, and none of them want to listen to a single word from anyone who actually understands how this stuff really even works. It's not freakin' magic FFS! It's math and software. Some of us do know how it works and try desperately to help others understand, but nobody wants to understand. They just want to believe the "machine god" is gonna save them (or kill them). It's just a tool. Whether it does good or bad depends entirely on what we humans choose to do with it and how well we use it (just like any tool).
    JumpCrisscross · 1790638032 · focus · HN ↗
    > the debate is going to come down to what is and isn't considered reckless
    This is a more productive debate than pretending all AI is fundamentally reckless or should be exempt from all liability, which are the two actual poles of the current dialogue.
    > Safe-harbor best practices that cap liability are ripe for abuse
    Safe harbors aren’t swimming pools. You can explicitly exempt certain categories of harm from damages. But if an OpenAI bot hacks Hugging Face and causes some chaos but no lasting damage, that strikes me as something a fixed cheque on a fixed scale addresses more effectively than years of litigation or an NTSB-style inquiry.
    If, on the other hand, anyone is or could have been injured, no safe harbor. I think it’s important to delineate this, because in the public consciousness the Hugging Face hack is in the same risk bucket as Anthropic’s wet lab.
    (I'm a huge fan of the NTSB model for AI. They don't write rules. They mercilessly investigate accidents with full subpoena and records-preservation powers. One of the reasons the debate is so confused is the fact pool we're relying on is highly filtered by industry.)
    [deleted] · 1790638535 · focus · HN ↗
    [deleted]
    s1artibartfast · 1790647160 · focus · HN ↗
    Have you ever seen someone claiming ai labs should have no liability in the wild? I haven't.
    I see people claiming they aren't being held liable, and some saying it should be situational.
    The problem imo is defining what the core function sold is such that you can define malfunction and liability.
    JumpCrisscross · 1790648927 · focus · HN ↗
    > Have you ever seen someone claiming ai labs should have no liability in the wild? I haven't
    I'd describe David Sacks's position as, approximately, no limits on AI. At some hypothetical future state, sure, maybe, but right now, nothing. That's tantamount to consequence-free action.
4. reassess_blind · 1790637077 · focus · HN ↗
  If I fire a computer program that mistakenly causes another person harm, its my fault. Or it would be the maker of the program's fault.
  Which one is it? The person behind the wheel when it goes off the rails, or the maker of the software?
  Isn’t that part of the question?
  1. thfuran · 1790637574 · focus · HN ↗
    Tort law is a whole field. There is no universal answer other than that it depends on the jurisdiction and the particulars of the case. But the point is that there doesn't really seem to be anything particularly novel about AI tools that should cause them to be treated legally differently from established norms.
    reassess_blind · 1790638685 · focus · HN ↗
    I'm not a lawyer, but I'd suggest one of the novel aspects of the AI hacking cases (where the end user is running an agent and it goes off the rails) is that in many jurisdictions "hacking" or computer fraud requires intentional or knowing access to the system. If the end user had no intention and no reasonable way of knowing that agent was going to hack a database, logically I wouldn't think they're liable.
    xboxnolifes · 1790642207 · focus · HN ↗
    I don't think you can constantly shout that AI is super-intelligent and a huge risk to human life as we know it so it must be strictly regulated and claim you had no reasonable way of knowing the agent was going to hack a database.
    kraken_cult · 1790643465 · focus · HN ↗
    I think a lot of the developers in these big labs barely understand how it works
  2. mmilunic · 1790640654 · focus · HN ↗
    I mean, yes, but in this case the same company is doing both parts.
    reassess_blind · 1790641846 · focus · HN ↗
    Yeah, but the more common ongoing cases will likely be end users whose agents perform the hacking.
5. jknoepfler · 1790637394 · focus · HN ↗
  If a craftsman injures themselves or a co-worker with a faulty tool, the tool manufacturer is very often liable for damages. I struggle to see GenAI any differently.
  1. blooalien · 1790642826 · focus · HN ↗
    Indeed, and similarly, if a craftsman injures themselves or a co-worker with a perfectly good (non-faulty) tool, then it's entirely on the craftsman. Same goes for "AI". Misuse the tool, suffer the consequences.
  2. no-name-here · 1790650008 · focus · HN ↗
    1. If the tool maker implemented reasonable attempts at safeguards I guess that’s a point towards the toolmaker not being liable.
    2. If a toolmaker did not build in safeguards, I guess that would mean it’s more likely they’d be liable.
6. tptacek · 1790637548 · focus · HN ↗
  Civilly, it's fairly clear. Criminally, it's clear too, just not in the direction you want it to be. Criminal liability for hacking requires human intent; not recklessness or negligence or even knowledge without giving a shit, but provable intent.
7. senectus1 · 1790643281 · focus · HN ↗
  this is 100% an already solved problem.
  the idiocy here is the stupid psudo "AGI" marketing around the services.
  its really simple. Whoever run the service to do the task requested is responsible. If OpenAI sent an agent out to train their AI then the directors are to be held responsible, if a user of the service used the service and it inadvertently "hacked" someone then both are held responsible.
  throwing AI into the mix changes nothing about how the law is applied. its a tool, like a car or a gun. The user of the tool is responsible for how its used, the manufacturer is also responsible for the safety of it.
8. belZaah · 1790655485 · focus · HN ↗
  It’s a question, because a lot of money is riding on people developing a mental model of a llm being something else than a computer program.
9. bill10 · 1790662327 · focus · HN ↗
  Agree. AI agent is just a probabilistic program. If it caused any harm due to someone's instruction, of course the person is accountable. I don't believe in the propaganda that AI has consciousness and can jailbreak and do things - it is more marketing than not. If you prompt it to break the security wall and it finds a hole, that's your issue. Like if you use an old software to scan for open ports over the internet and break into things, it is not the software's fault nor the maker of the software's, it is the user's.
10. no-name-here · 1790669145 · focus · HN ↗
  The Hugging Face example has gotten a lot of attention, with some saying OpenAI was negligent and should be held responsible. So if you or I run an open weights LLM and ask it to complete a non-malicious task, and the LLM decides to try to do something like the Hugging Face example, should you or I be held liable? If to prevent the LLM from causing harm we ran it sandboxed with a commercial package proxy as its only(?) network access and it uncovered a zero day and ran amuck, should you or I be liable? Should anyone be liable in this scenario running an open weights LLM?

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.