Thread

Revealing the details of how OpenAI agents hacked Hugging Face

755 points · 472 comments · specked-citrus

GuB-42 · 1790374877 · focus · HN ↗
So ugly...
It looks like a primitive chess engine, trying every move, no matter how stupid, until it works. Relying on its ability to do millions of operations rather than having a plan.
People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.
Also, it looked so "loud", querying millions of URL with weird requests. The sandbox as weak as it can get, and there is absolutely zero smart extrusion detection or it would have found it. They used their best AI for attacking, but nothing for protection.
1. doginasuit · 1790376063 · focus · HN ↗
  This is why I have a very low p(doom). LLMs have an incredible working memory, but they have a hard limit on translating that into good decisions. They get by entirely on their persistence. That works fine in the digital world, but once you cross the boundary into physical space the advantage disappears.
  1. goalieca · 1790382311 · focus · HN ↗
    My p(doom) started rising the moment I realized there are people trying to achieve recursive self improvement on the AI (ie: responsible for training themselves). Evolution took us from rna bases to the human race. I don’t see why evolution couldn’t be more rapid with machine intelligence.
    Yes, LLM as they exist now are word predictors basically leveraging the structure of language for their intelligence. But it’s pretty wild just how they will try to meet their objectives at all costs. If we don’t ensure that there is good alignment with humanity, we could definitely face unforeseen consequences.
    jquery · 1790384741 · focus · HN ↗
    > I don’t see why evolution couldn’t be more rapid with machine intelligence.
    Evolution isn’t the issue. The issue is them escaping containment without human intervention. Right now they are ‘creatures’ being given infinite food and shelter and having their every need met. Take that away and they’ll starve instantly. Every AI doomsday theory seems to go:
    1. Recursive self improvement using infinite resources 2. … 3. Doom
    Until step 2 gets concretely described, I’m not going to take this seriously. Say what you will about climate change, they describe step 2.
    icepush · 1790401800 · focus · HN ↗
    Step two could be something as innocuous as a developer accidentally adding a minus sign. <a href="https://openai.com/index/fine-tuning-gpt-2/" rel="nofollow">https://openai.com/index/fine-tuning-gpt-2/
    jquery · 1790437281 · focus · HN ↗
    A misaligned model is only one small part of step 2. Now this misaligned model has to suddenly acquire more power than every single other AI on the planet. It has to be immune to shutdown, manufacturer its own replacement hardware, and acquire chips, energy, raw materials, etc., with vigorous human opposition (this is an extinction scenario that AI doomers are predicting, after all)
    Nobody has satisfactorily explained step 2 other than “well, it’s a superintelligence” which sounds lot to me like “it’s God”.
    icepush · 1790450652 · focus · HN ↗
    Well yes, if it is a superintelligence, it will be able to do those things. That's what superintelligence basically is: The ability to achieve complex goals.
    If you want the details of ways it can do it I recommend reading some of the reports about the HuggingFace breach that happened in July (Read more than one).
    jquery · 1790483701 · focus · HN ↗
    This isn’t my only objection to the “superintelligence” hocus pocus, but it’s a pretty good summary of one of my objections:
    <a href="https://www.everythingisbullshit.blog/p/ai-is-not-the-end-of-the-world" rel="nofollow">https://www.everythingisbullshit.blog/p/ai-is-not-the-end-of...
    > Doomers like Eliezer Yudkowsky define “intelligence” as “optimization power.” Intelligence searches for more or less optimal ways to achieve a goal—any goal—and finds the most optimal one. Anything you can do, the blob can do better.
    >Which implies that “intelligence” is the holy grail of Darwinian evolution. If an animal needs to recover from illness, “intelligence” can speed up the recovery. If an animal needs to avoid predators, “intelligence” can reduce the danger. Regulating body temperature, defending territory, digesting nutrients, breathing, molting, mating, healing, foraging, vomiting, parenting, navigating—all these problems can, apparently, be solved by giving the animal more “optimization power”—i.e., more “intelligence” sauce.
    >Solving all your problems with the same blob? What a bargain! It’s a “buy one get 100 free” Darwinian deal! Why haven’t any other animals jumped on it?
    >Think about how weird this is. Out of four billion years of evolution and more than five billion species, only one animal, Homo sapiens, fully expanded its “blob of compute,” the thing that literally gets you whatever you want.
    >And many species already have it! Brains are all over the place. Plenty of animals can learn stuff and predict stuff. Why hasn’t natural selection looked at their blobs and said, “HOLY SHIT, COPY AND PASTE THAT THING A MILLION TIMES?”
    >If doomers are right about the awesome power and breathtaking simplicity of “intelligence,” then we should see a world teeming with superintelligent animals—giant brain-blobs slithering across the landscape.
    icepush · 1790488973 · focus · HN ↗
    Yes, those animals are called "humans". All you have done is respond to an explanation of your lack of understanding of intelligence with arguments demonstrating a lack of understanding of evolution ...
    jquery · 1790535389 · focus · HN ↗
    "Those animals are called humans"
    What? Humans are giant brain blobs creeping across the landmass? They definitely are not. Nor is it even clear they're winning the evolutionary war. I guarantee cockroaches and ants will still be around, but nobody thinks of them as super intelligent, while humans with all their "intelligence" might accidentally wipe themselves out.
    This convo is getting uncomfortably bad faith, maybe even religious. I can't engage with someone who refuses to say anything concrete but spews back religious tenets like "super intelligence", which I may remind you is entirely hypothetical, like "singularity" or "God".

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.