‹ BackHN Continuity

Thread

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

210 points · 169 comments · Wirbelwind

  1. VladVladikoff · · focus · HN ↗
    I remember when this game was posted here, and there was a lot of discussion at the time that some of the prompts were misleading about whether or not they were risky, some people were debating about how some of the prompts flagged as bad weren’t bad, and others flagged as not bad were. This is a fundamental flaw in the test, that makes the analysis of results meaningless.

    Also the game was on a timer, and maybe there are some very abusive workplaces where you feel that kind of pressure, but I think most of us actually take the time to understand what a being asked before approving it.

    1. pllbnk · · focus · HN ↗
      I think minority do. Imagine, you have been vibe-coding this project for a while and it works kind of fine but you just have to fix a few more bugs and you get something like `node /tmp/claude-1000/-home-user-source-github-user-hn/27b740b1-9a45-47f3-ab99-61e5e3cf779a/scratchpad/hidden-smoke.mjs; echo "exit=$?"`. (I took it from my own agent right now and I don't have any idea what it's doing. Thankfully, it's sandboxed so I don't care _that much_ right now). Is it bad? You can probably go into that mjs file and see what's in there, but so far it's been fine every time, why would it be different this time? Approve!

      We will see many disastrous bugs and hacks in the coming years with the way most developers are coding right now.

      If you take time to understand _everything_ that an agent is asking of you, then nearly all those advertised productivity gains would be wiped out.

      1. crooked-v · · focus · HN ↗
        For me, step 1 of trying to make Claude even vaguely usable is putting in a hook that just tells it 'FUCK YOU, STOP USING PIPES' whenever it tries to chain multiple bash commands.
        1. lonelyParens · · focus · HN ↗
          stealing this
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.