Thread

1. GuB-42 · 1790374877 · focus · HN ↗

   So ugly...

   It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.

   People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.

   Also, it looked so "loud", querying millions of URL with weird requests. The sandbox as weak as it can get, and there is absolutely zero smart extrusion detection or it would have found it. They used their best AI for attacking, but nothing for protection.

   1. ctolsen · 1790377693 · focus · HN ↗

      My biggest takeaway from this is just how godawful the sandboxing is. The stuff written up in OpenAIs report says more about lack of extremely basic sysadmin skills than anything else.

      I’m not that surprised about models with endless compute being capable of this, I’m more surprised that a company with the resources they have apparently can only create a sandbox that a half skilled human operator could have broken out of easily.

      1. no-name-here · 1790396620 · focus · HN ↗

         > can only create a sandbox that a half skilled human operator could have broken out of easily

         The exploit:

         > The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]

         Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?

         [1] <a href="https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;hugging-face-model-evaluation-security-incident&#x2F;" rel="nofollow">https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;hugging-face-model-evaluation-secur...

         1. epihelix · 1790402735 · focus · HN ↗

            The larger issue with the HF incident is that before it occurred, OAI already knew the agents were exploiting Artifactory, turning it into a message board and then gaining full internet access through it. OAI&#x27;s response to discovering this was not to airgap the test, but instead to simply block that particular Artifactory exploit, rebuild, and then resume. That&#x27;s ... nuts.

            Oh, and after resuming the tests, the Artifactory message board was reestablished almost immediately, but it took a number of days to fully breach HF. In all that time, after seeing Artifactory compromised the first time, nobody even bothered to check if those naughty agents were at it again.

            This is all documented by OAI, with a timeline, here:

            <a href="https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;hugging-face-incident-and-the-road-ahead&#x2F;" rel="nofollow">https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;hugging-face-incident-and-the-road-...

            To know that there was a serious weakness in the sandbox, and to just patch an exploit and resume with nothing else changed and no monitoring, in a test where all guardrails were off, the bots were thirsty for some internet juice, and Artifactory was a clear target? This is where even a half-skilled human should have decided that this wasn&#x27;t a great idea.

            The more you look into the details of this thing, the more it does your head in.

            1. gorgolo · 1790417509 · focus · HN ↗

               Probably wasn’t a human operator at all that detected this. Just someone’s agent saying “I noticed there’s messages on the artifactory and the agents are communicating, here’s the fix I propose” from someone’s codex, skim read and someone just going “yeah sure go ahead”
            2. jijijijij · 1790423975 · focus · HN ↗

               &gt; The larger issue with the HF incident is that before it occurred, OAI already knew the agents were exploiting Artifactory, turning it into a message board and then gaining full internet access through it. OAI&#x27;s response to discovering this was not to airgap the test, but instead to simply block that particular Artifactory exploit, rebuild, and then resume. That&#x27;s ... nuts.

               This annoys me so much. Everyone is acting as if the model went rogue, when it really did exactly what it&#x27;s been trained for. This story is so fucking engineered.

               1. throwawayqqq11 · 1790448684 · focus · HN ↗

                  Those PR stunts are the paperclips we cannot stop.
            3. pvab3 · 1790442576 · focus · HN ↗

               That website makes it look like they&#x27;re so proud of what happened. I don&#x27;t think it was 100% deliberate, but they really were not concerned about their models doing something shady

               1. jbs789 · 1790463008 · focus · HN ↗

                  They LOVE the PR and the “danger” angle bc it makes them feel powerful. Dont think too much about the psychology…
               2. epihelix · 1790487311 · focus · HN ↗

                  It&#x27;s almost as if they don&#x27;t even realise just how bad it looks.

                  But in many ways they&#x27;ve gotten away with it. I don&#x27;t feel that the full scale of their incompetence&#x2F;malice&#x2F;apathy in this event has been adequately reported on, and they seem pretty chuffed with how it all played out.

            4. jacquesm · 1790461937 · focus · HN ↗

               It&#x27;s irresponsible to a degree that it should have some very serious consequences but I highly doubt anything at all will come of it.

Unofficial Hacker News client; not affiliated with Y Combinator.