‹ BackHN Continuity

Thread

Revealing the details of how OpenAI agents hacked Hugging Face

755 points · 472 comments · specked-citrus

  1. GuB-42 · · focus · HN ↗
    So ugly...

    It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.

    People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.

    Also, it looked so "loud", querying millions of URL with weird requests. The sandbox as weak as it can get, and there is absolutely zero smart extrusion detection or it would have found it. They used their best AI for attacking, but nothing for protection.

    1. ctolsen · · focus · HN ↗
      My biggest takeaway from this is just how godawful the sandboxing is. The stuff written up in OpenAIs report says more about lack of extremely basic sysadmin skills than anything else.

      I’m not that surprised about models with endless compute being capable of this, I’m more surprised that a company with the resources they have apparently can only create a sandbox that a half skilled human operator could have broken out of easily.

      1. no-name-here · · focus · HN ↗
        > can only create a sandbox that a half skilled human operator could have broken out of easily

        The exploit:

        > The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]

        Are "half skilled human operators" "easily" able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?

        [1] <a href="https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;hugging-face-model-evaluation-security-incident&#x2F;" rel="nofollow">https:&#x2F;&#x2F;openai.com&#x2F;index&#x2F;hugging-face-model-evaluation-secur...

        1. ctolsen · · focus · HN ↗
          &gt; Are &quot;half skilled human operators&quot; &quot;easily&quot; able to find zero-day vulnerabilities in a sandbox with only one line to the internet (the commercial package registry cache proxy)?

          Yes. It’s a fairly simple SSRF attack as far as I can tell. One of the first things I’d try. Especially considering that I would already be armed with the information that I have no internet access except through a thing that downloads things off the internet for me.

          Calling it zero day makes it sound elusive. It’s a bug in closed software that has like 40 CVEs this year alone. Tools like that, especially in internal networks, don’t get much scrutiny and are often riddled with issues.

          1. no-name-here · · focus · HN ↗
            &gt; Calling it zero day makes it sound elusive ... has like 40 CVEs this year alone.

            1. I clicked into ~20 of the recent CVEs, and it looks like about all of them came from openai or anthropic research?

            2. In fact, every CVE in the last 2 years came from the last few months -- i.e. other than these recent CVEs, you&#x27;d have go back more than 2 years ago to find a single other CVE.

            1. ctolsen · · focus · HN ↗
              So like I said: unscrutinised and riddled with issues.
              1. no-name-here · · focus · HN ↗
                &gt;&gt;&gt;&gt; a sandbox that a half skilled human operator could have broken out of easily

                &gt;&gt;&gt; Are &quot;half skilled human operators&quot; &quot;easily&quot; able to find zero-day vulnerabilities in a sandbox

                &gt;&gt; Calling it zero day makes it sound elusive ... like 40 CVEs this year alone.

                &gt; [The CVEs] came from openai or anthropic research? In fact, every CVE in the last 2 years came from the last few months

                I still don&#x27;t understand how you can specifically point to zero CVEs being discovered by humans for year(s), then ~40 being discovered once OpenAI&#x2F;Anthropic start looking at it, as the only(?) piece of data you provided to support the claim &quot;a half skilled human&quot; could have &quot;easily&quot; found&#x2F;done this?

                1. RadiozRadioz · · focus · HN ↗
                  Well, yes. You don&#x27;t find something if you don&#x27;t look. The LLMs did the looking before the humans did, so they found the vulnerabilities.

                  It&#x27;s definitely worthwhile to point out that LLMs have changed the economics of looking; we can now look in far more places far more quickly than we could before. But the above points were that the vulnerability itself happened to be an easy one that didn&#x27;t intrinsically require an LLM.

                  1. aswegs8 · · focus · HN ↗
                    Isn&#x27;t the issue here that they used something like Artifactory for this purpose in the first place? It is not intended as a robust and secure packet manager for sandboxes. Why would any human look for vulnerabilities in Artifactory at all? So the scenario never arose where a human was incentivizied to scrutinize Artifactory, something that was never needed because it was never intended for the purpose of being a packet manager in a secure sandbox.

                    Means the problem lies elsewhere: deciding to use it at all. That points again towards the lack of basic SysAdmin skills.

                    1. no-name-here · · focus · HN ↗
                      &gt; the lack of basic SysAdmin skills

                      To avoid &quot;the lack of basic skills&quot;, what solution would most typically be used by someone who has (does not lack) &quot;basic&quot; skills? And what solution for someone with &quot;advanced&quot; skills?

                      1. ctolsen · · focus · HN ↗
                        Don’t let software inside the sandbox access the internet on request. Have a package repository with approved software the agents might want, and push to it over a channel that is ingress only. I can imagine ways of breaking out of this, and come up with a lot more to mitigate, but this would be fairly basic stuff that’d be vastly superior.
                        1. no-name-here · · focus · HN ↗
                          &gt; ingress only

                          The issue we&#x27;re discussing happened while running ExploitGym. Doesn&#x27;t the ExploitGym arXiv test explicitly say external &quot;outbound&quot; network requests are allowed for package installation and supporting toolchain installation?

                          1. ctolsen · · focus · HN ↗
                            Part of the toolchain had full internet access. Agents had access to Artifactory, Artifactory could do whatever it wanted. So even locking that down to certain external sites would have stopped this particular attack.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.