‹ BackHN Continuity

Thread

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

208 points · 169 comments · Wirbelwind

  1. continuational · · focus · HN ↗
    It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".

    It's been tried so many times before, and it never worked.

    1. IgorVoytyuk · · focus · HN ↗
      The pattern resurfaces because the prompt sits where it is cheap to implement, not where the question is answerable. "Allow rm -rf /tmp/build?" asks you to predict a consequence from a string. The thing you could actually judge - did this run produce the artifact it was supposed to - only exists after you have already said yes.

      I ran an autonomous pipeline in production for eight months and logged every silent failure. Two that make the point:

      - My top-level health signal stayed green for three days while zero artifacts shipped. Sixteen daemons alive, backend responding, auth token valid. Every signal it polled was true, and nothing measured the thing leaving the building. No approval prompt anywhere in that chain would have fired, because nothing was attempting anything dangerous.

      - I wrote 29 quality gates, tested them, committed them. Not one was ever called - nothing was a runner. The unit tests proved the gates worked; nothing proved they were wired. A permission layer has exactly that failure mode available to it, and it is invisible from the outside: a policy that never denies looks identical to a policy that was never loaded.

      So the 1-in-3 miss rate reads to me less like inattention than like being asked the wrong question at the only moment the system offers to ask one. Approval is a claim about the future. Verification is a claim about a result, and only the second one can be checked afterwards.

      Genuine question for anyone running agents with approvals on: when did your approval flow last block something you would have regretted - and can you tell that apart from it never having fired at all?

      1. inigyou · · focus · HN ↗
        LLM wrote this comment.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.