‹ BackHN Continuity

Thread

Humans missed 1 in 3 threats approving AI agent commands across 40k game runs

212 points · 175 comments · Wirbelwind

  1. continuational · · focus · HN ↗
    It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".

    It's been tried so many times before, and it never worked.

    1. est31 · · focus · HN ↗
      I think it's partially for responsibility reasons. Your employee approved the bash call? not our fault then!
      1. Aurornis · · focus · HN ↗
        If a company advertised an LLM as perfectly safe and then it caused some damage, there would be a case against the company.

        Have you used LLM tooling? It comes with warnings and explains that the user accepts the risk. Different levels of warning are supplied for the different levels of autonomy you can enable. The user has to understand the risk as they enable it.

        This is not a new concept and it’s not an idea the LLM companies invented. It shouldn’t be surprising to anyone.

        1. chrisjj · · focus · HN ↗
          > This is not a new concept and it’s not an idea the LLM companies invented.

          I don't recall any prior computer software working so badly that it needed a disclaimer like "Claude is AI and can make mistakes" on its front page. Let alone one so costly.

          1. mhjkl · · focus · HN ↗
            You must not use any open source software, because much of it comes with a whole paragraph of ”IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY“
            1. rrr_oh_man · · focus · HN ↗
              That would be fine if Claude was open source and/or unpaid
              1. rcxdude · · focus · HN ↗
                If you read the fine print of most commercial software you'll see some language to a similar effect. When's the last time Microsoft got sued due to an outage caused by a bug in Windows?
                1. chrisjj · · focus · HN ↗
                  Yes, the fine print.

                  Only this so-called "AI" needs it full size on the front page.

                  1. rcxdude · · focus · HN ↗
                    I would consider that more honest than more of a problem.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.