Thread

Revealing the details of how OpenAI agents hacked Hugging Face

755 points · 472 comments · specked-citrus

damowangcy · 1790403824 · focus · HN ↗
Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?
If I post something on the Internet today claiming that I asked my agent to do X but it went rogue and did Y, all I will be getting in return is a jar full of "skill issue".
Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm. And this is not something we as individual or even company can deal with, responsibility should be held by those who use it, in a legal way.
I am baffled by the fact that up until now, no one is held responsible for so many incidents reported publicly or privately. At this point, it's free marketing, if I am CEO of any AI company, I will run swarm of agents hacking all NGOs and stating that I am just looking for some random piece of data that happened to be hidden in their servers, at least that's what my LLMs think, not me. Then I will start preaching everyone how dangerous this piece of technology is and start giving out free tokens for these NGOs so they can start defending themselves and we should slow the f down.
1. zx8080 · 1790404862 · focus · HN ↗
  It's not the incompetency. It's carefully designed pre-IPO story.
  1. sicher · 1790405404 · focus · HN ↗
    Care to explain how that would make sense?
    imtringued · 1790407314 · focus · HN ↗
    If you got caught breaking an expensive vase but there is no evidence, would you confess your crime? You would lie and make up some story how it wasn't your fault.
    christkv · 1790408317 · focus · HN ↗
    They ate pushing to be regulated so they can create a locked down monopoly of 3-4 players where nobody gets to sell legal llm access like open weights. Instant locked down corporate market.
    ArnoVW · 1790408472 · focus · HN ↗
    This technology is very powerful, and it is (and by extension, so are we) very valuable
    Also good story telling for the narrative of “this technology is so powerful that it must be strictly regulated”
    0xDEAFBEAD · 1790414156 · focus · HN ↗
    I doubt it. At this stage, so many eyes are on the AI industry that regulation is likely to be disadvantageous to industry actors: <a href="https://marginalrevolution.com/marginalrevolution/2026/09/what-regulatory-capture-actually-looks-like.html" rel="nofollow">https://marginalrevolution.com/marginalrevolution/2026/09/wh...
    Many people are calling for an AI pause or liability/punishment for bad actors like OpenAI. You have to do serious mental gymnastics to convince yourself that Sam Altman will benefit financially from the new regulatory bloodlust. No surprise that OpenAI hasn't exactly been forthcoming about info related to these hacks.
    Furthermore, HuggingFace required an open-weight model to respond to the hack. That certainly blows a hole in the "carefully designed" claim from zx8080, if nothing else. It looks terrible for OpenAI, and decreases the probability of some sort of regulatory restriction on open models.
    riedel · 1790422546 · focus · HN ↗
    Wouldnt call it regulatory blood lust at the moment. I think the competitive situation that is however solely focussed in performance at any cost and not at compliance at all is forcing AI labs to play with fire. The risk for them is an even bigger regulatory backlash. Totally different industry : Chinese ebike manufacturers are currently pushing the rules for motor strength to the max due to competition. What will likely happen is stricter regulation of max support and motor power if there are incidents. Particularly there will be more regulatory fragmentation with little common denominator. This will hurt the whole industry's growth. In this industry it kind of still works as an agreement of non-Chinese manufacturers.
    sicher · 1790428230 · focus · HN ↗
    Yeah. If the conspiracy theory was that these incidents were somehow orchestrated by Anthropic, that would at least make some logical sense.
    LMYahooTFY · 1790434716 · focus · HN ↗
    What you're saying might be plausible if Dario and Sam didn't take every other opportunity to call for regulation. And that's a massive understatement with Anthropic.
    They're facilitating a doomerism cult that is lobbying and clearly making headway in Congress.
    You have to put your head in the sand to think open weight models aren't a threat/large revenue loss to their business.
    0xDEAFBEAD · 1790441279 · focus · HN ↗
    Dario and Sam have been doomers or doomer-adjacent for many years. Occam's Razor is that they simply believe what they are saying. As do many other prominent people:
    <a href="https://aistatement.com/work/statement-on-ai-extinction-risk" rel="nofollow">https://aistatement.com/work/statement-on-ai-extinction-risk
    <a href="https://www.pacingthefrontier.com/" rel="nofollow">https://www.pacingthefrontier.com/
    <a href="https://aitreaty.org/" rel="nofollow">https://aitreaty.org/

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.