Thread

Revealing the details of how OpenAI agents hacked Hugging Face

755 points · 472 comments · specked-citrus

GuB-42 · 1790374877 · focus · HN ↗
So ugly...
It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.
People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.
Also, it looked so "loud", querying millions of URL with weird requests. The sandbox as weak as it can get, and there is absolutely zero smart extrusion detection or it would have found it. They used their best AI for attacking, but nothing for protection.
1. ctolsen · 1790377693 · focus · HN ↗
  My biggest takeaway from this is just how godawful the sandboxing is. The stuff written up in OpenAIs report says more about lack of extremely basic sysadmin skills than anything else.
  I’m not that surprised about models with endless compute being capable of this, I’m more surprised that a company with the resources they have apparently can only create a sandbox that a half skilled human operator could have broken out of easily.
  1. SV_BubbleTime · 1790393586 · focus · HN ↗
    I’m a pretty firm believer that this was intentional and that they wanted it to escape the sandbox.
    Woo look at escaped our sandbox, so scary! Be scared! Be scared now! Call your representative and do tell him how scared you are!
    Yeah, I mean our sandbox was a paper bag, but don’t focus on that.
    esseph · 1790397068 · focus · HN ↗
    OpenAI, Google, Anthropic, and Moonshot.ai have all "had this happen" now.
    So, either they're all liars, or incompetent and negligent (and still liars).
    d0mine · 1790399341 · focus · HN ↗
    Are you including cyber-testing by UK AI Safety Institute in this list?
    Sure, the models are capable (for some test tasks, though they are not omnipotent yet) but does it mean the actual OAI sandbox is adequate? Could have a competent engineer done better and made the escape less likely?
    esseph · 1790402169 · focus · HN ↗
    > Are you including cyber-testing by UK AI Safety Institute in this list?
    Nope, and look!
    OpenAI hacked multiple US government sites!
    <a href="https://www.bbc.com/news/articles/cw62jje658dlo" rel="nofollow">https://www.bbc.com/news/articles/cw62jje658dlo
    ---
    <a href="https://www.reuters.com/technology/metas-ai-model-hacked-another-company-during-testing-information-reports-2026-08-05/" rel="nofollow">https://www.reuters.com/technology/metas-ai-model-hacked-ano...
    <a href="https://www.wired.com/story/moonshot-kimi-k3-ai-model-escape-sandbox/" rel="nofollow">https://www.wired.com/story/moonshot-kimi-k3-ai-model-escape...
    dastardly45 · 1790405488 · focus · HN ↗
    There's a common denominator in the Israeli "security" company Irregular who was doing security testing for 4 AI labs <a href="https://thenextweb.com/news/irregular-four-labs-one-issue-disclosure-timeline-gemini" rel="nofollow">https://thenextweb.com/news/irregular-four-labs-one-issue-di...
    cubano · 1790458412 · focus · HN ↗
    Me too, indeed.
    no-name-here · 1790399115 · focus · HN ↗
    > sandbox was a paper bag
    The exploit:
    > The ExploitGym evaluation environment did not provide the models with direct Internet access. To gain Internet access, the models identified and exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy. We disclosed this vulnerability, along with other Artifactory vulnerabilities our models identified as part of our review, to the vendor. [1]
    Are most sandboxes more secure than only having a single avenue for internet access, the commercial package registry cache proxy, where the latter had a previously unknown zero-day vulnerability?
    [1] <a href="https://openai.com/index/hugging-face-model-evaluation-security-incident/" rel="nofollow">https://openai.com/index/hugging-face-model-evaluation-secur...

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.