Thread

A heap overflow and SSO misconfiguration to compromise OpenAI internal repos

491 points · 208 comments · Handy-Man

btown · 1789708835 · focus · HN ↗
> By 6:00 a.m. on July 25, we had confirmed local RCE through an image upload. We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances.
> When we checked again at 10:00 a.m., the agent had achieved RCE on Discourse Cloud and demonstrated access by reading /etc/hosts. Using the generated exploit script, we managed to get RCE on OpenAI’s instance.
Between this and the HuggingFace hack, we've built systems that are so goal-oriented, and so capable, that they will do almost anything if they are convinced it is justified - or if they are playing a "game" where there is no goal but to win.
Of course I want my software to be able to audit its own security, and to defend against attackers who have the benefits of their own agentic systems. But at a certain point, did we need it to be trained so much on CTF games?
It feels like an entire industry watched <a href="https://en.wikipedia.org/wiki/WarGames" rel="nofollow">https://en.wikipedia.org/wiki/WarGames and ended up thinking "this is a challenge, we can just build a better WOPR, of course it will know when it's playing a game. Let's play Global Thermonuclear War."
1. wood_spirit · 1789712591 · focus · HN ↗
  > they will do almost anything if they are convinced it is justified
  I’m in the “glorified spell checker” camp, although I don’t mean to reduce their impressive utility and belittle them in the way many people read that term and infer.
  So I am not sure that an llm “justifies” anything. I mean that their “thinking” text talks about justifications but it is just a very advanced statistical regurgitation of the kind of text humans use. I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc).
  What you really have is a model that tries the statistically most probable thing to say next and so on and what is really cool is how effective this is at generating a path that we can slap a narrative over afterwards that makes the whole thing feel motivated and consistent, like the model started off knowing how it was going to get to the destination.
  Which is, under the hood, a completely different kind of “intelligence” as the supercomputer in War Games.
  1. Certhas · 1789713558 · focus · HN ↗
    Ultimately, the brain is just a bunch of neurons activating in a specific pattern. This observation does not really tell us anything though. It doesn't acknowledge the difference between a 2500 Neuron fruit fly brains and a human brain.
    Likewise, the fact that LLMs are a stochastic autoregressive process (which is a class of systems every bit as rich as the ODEs used to model neurons) tells us nothing a priori.
    wood_spirit · 1789714347 · focus · HN ↗
    Absolutely. If someone makes the weights do continuous learning etc then perhaps an llm can internalise morals. Of course, just like a human, it will be possible to talk it out of those morals. Another recent thread about this is <a href="https://news.ycombinator.com/item?id=49744420">https://news.ycombinator.com/item?id=49744420
    Certhas · 1789718747 · focus · HN ↗
    If I repeatedly call an LLM in a loop with a markdown document it can edit, would that make it qualify for you?
    If I give an LLM to compact its context window, so the context it carries can evolve iteratively over time as more and more things come in, is that enough?
    Compacting the context is really a very, very interesting example here. The "next token predictor" is telling an external tool to change all "previous" tokens. So an LLM + a harness that allows compacting the context is no longer just a token predictor at all!
    You don't need continuous learning to get interesting dynamics. You just need feedback loops.
    wood_spirit · 1789938712 · focus · HN ↗
    I come back to this way after everyone else has stopped reading. But it’s been making me think.
    Richard Dawkins says he thinks LLMs think.
    And the physical angle is that nothing is special about humans and software simulating it would also be thinking.
    But from using LLMs all the time, and understanding what is under the hood, I’m thinking that the current approaches aren’t cutting it for me and I’m not expecting them to get there. There was such big jumps early on but progress is slowing as though diminishing returns.
    So we can build things that think and outthink us, but I don’t think anything we’ve hit upon yet is going to scale up into it.
    leg100 · 1789721384 · focus · HN ↗
    One is an observation the other is not, it's a description of what it is; one is a posteriori, the other is a priori (contrary to what you say).
    They're not comparable.
  2. eru · 1789715923 · focus · HN ↗
    > I don’t think it means the model has internalised the meaning of it (as witness when you talk to an llm how often it forgets what you recently told it was important etc).
    Humans forget stuff all the time anyway. Would you give them the same diagnosis?
    Btw, what you describe about 'the most probably next token' would be true for a model that only went through pre-training where they only train on exactly that task.
    But there's a lot of re-inforcement learning afterwards.
    disgruntledphd2 · 1789718508 · focus · HN ↗
    > But there's a lot of re-inforcement learning afterwards.
    That just shifts the distribution of tokens produced. Ultimately they are still just next token predictors.
    Like, even "reasoning" models basically work by generating more tokens at inference time, and using them to shift the distribution towards more useful outcomes (in some cases).
    eru · 1789721535 · focus · HN ↗
    They are next token producers. I would only call it a predictor, if it's trained to predict tokens (ie just after pretraining).
    Just like humans produce one word after another when they talk, but they don't generally try to imitate other humans.
    wood_spirit · 1789723600 · focus · HN ↗
    Don’t people pick up language, vocabulary and dialect from those around them? Perhaps it’s subconscious but humans are imitating other humans all the time?
    eru · 1789724654 · focus · HN ↗
    It's a mix.
    Yes, you imitate how others speak, but when you are trying to solve a problem, you don't try to predict how others would complete a text.
    (Well, unless you follow 'what token would Jesus pick?' / 'what would Jesus do'.)
    tripzilch · 1789728947 · focus · HN ↗
    What does it matter what humans do? We're talking about LLMs, running known+vastly less complicated algorithms on known+vastly less complicated hardware.
  3. Arn_Thor · 1789716319 · focus · HN ↗
    I used to share that perspective until very recently, but today I think it's an outdated way to think of the cutting-edge LLMs. There is so much more going on, with MOEs, internal loops, guardrails and tools that I suspect we're dealing with something that's a little more than the sum of its parts. Not intelligent in the way we recognize in biological organisms, but certainly something beyond a mere Markov chain.
    HarlequinHair · 1789718318 · focus · HN ↗
    Make no mistakes.
    LLMs are language model, and nowhere in their code you can find actual reasoning. Re-reinforcement is not magical process that builds conscience or emotions.
    We are talking about probability built on statistics, with extea steps.
    Stop humanizing LLMs.
    jibal · 1789719658 · focus · HN ↗
    Agents are not simple language models.
    You can't find actual reasoning in a brain either. (Note that you can't tell the difference between a conscious brain and a comatose brain by examining them.) This is the same as Leibniz's mill argument ... it's a fallacy of composition.
    > Re-reinforcement is not magical process that builds conscience or emotions.
    They aren't the result of magic at all, but we are nowhere near the point of identifying what processes do or don't produce consciousness (or a conscience) or can be characterized as having emotions.
    > Stop humanizing LLMs.
    That's a clearly dishonest mischaracterization of the GP.
    I've read some of your other comments about LLMs and I find them unreasonably reductionistic, whereas I think the word "just" should be banned from ontological discussion, so I don't think further engagement would be beneficial and I won't be engaging in it. (And I'm actually quite conservative in ascribing cognitive traits to LLMs or other "AI".)
    HarlequinHair · 1789721714 · focus · HN ↗
    The best non technical explanation you can give is "An AI agent is an LLM that can take actions".
    While an agent doesn't necessarily have to be powered by an LLM, most modern AI agents are.
    You pointing at a human brain does not change that an AI agent is not intelligent and cannot think, we are still talking about probability built on statistics with extra steps.
    I am not trying to be dishonest, we should stop making analogies between AI and actual thinking, because they are two entire different concepts.
    Who developed these technologies used the words "thinking" and "reasoning", this does not mean they are actually thinking and reasoning. Somewhere you still have a processor calculating, with no empathy.
    So, again: stop humanizing AI. This sentence shouldn't make you angry.
    pizza234 · 1789724108 · focus · HN ↗
    > we are still talking about probability built on statistics with extra steps.
    There is a wrong assumption here: confusing primitives with emergent properties.
    One can't look at the primitivies and assume that certain properties will not emerge. It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
    > You pointing at a human brain does not change that an AI agent is not intelligent and cannot think
    That depends on the definition of intelligence and thinking, and it is dishonest not to give any definition (and most importantly, one that is not human-centered).
    AIs are currently fulfilling several aspects of intelligence and thinking, by any defition of intelligence. If you don't notice that, it's just because you have informed yourself enough. Having said that, I don't doubt that there are aspects that AI are lacking (e.g. retention/plasticity/perception), but the line is blurry, and they're advancing (too) fast.
    Empathy is actually a very important aspect of the AI problems, but it's not part of intelligence. Sociopaths don't have it, and yet, you wouldn't doubt that they're intelligent.
    tripzilch · 1789728804 · focus · HN ↗
    > It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
    You do realize that amino acids exist on a scale some orders of magnitude smaller than the gates we build GPUs out of?
    Honestly, this "you could say the same about humans"-argument is getting so tired. A brain neuron is so complicated, we can't even simulate a single one ...
    At the very least there is no reason why you should jump to a human brain, of all things.
    But the whole argument kinda loses its spice, when you say "well you could say the same about a mouse brain", and you know what happens when you create swarms of 1000s of mice ... super intelligence, right?
    anonymars · 1789742502 · focus · HN ↗
    Do mice satisfy your definition of intelligent?
    tripzilch · 1789817365 · focus · HN ↗
    That's the point.
    HarlequinHair · 1789729707 · focus · HN ↗
    > It would be exactly like looking at aminoacids and state that intelligence can't develop from them.
    We are not talking about what could develop from what we have today. We are talking about what we have today. The focus is not whether intelligence could develop or not from aminoacids. The focus is on the fact that aminoacids are not intelligent.
    Maybe in the future we could develop real intelligence starting from the current implementations of AI, but for sure we are not there today.
    We need definitions? Let's start small, ok? <a href="https://en.wikipedia.org/wiki/Intelligence" rel="nofollow">https://en.wikipedia.org/wiki/Intelligence
    We can start from here, open every link we find and decide what works for us.
    Conclusions drawn by scholars, psychologists, learning researchers, younameit, etc. revolves around the following concepts:
    ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.
    There is of course space for artificial intelligence. These broader and more general definitions of intelligence stop at concepts like elaborating data to reach an answer.
    Concepts like adaptability or evolution are somewhat lost or diluted to adjust the meaning for these new technologies.
    > AIs are currently fulfilling several aspects of intelligence and thinking, by any defition of intelligence.
    In the linked article there are dozens of definitions linked, and in most of them the current state AI is not considered to have intelligence. Having half of the property is not enough. I can jump, that doesn't make me a basketball player.
    Arbitrarily deciding to consider those definitions not valid or "human-centered" because they do not agree with your point of view is possibly worse than cherry picking. It's like asking to change the definition of a word on a dictionary because you do not agree with the meaning.
    anonymars · 1789742415 · focus · HN ↗
    > ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.
    So, like the HuggingFace attack? <a href="https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation" rel="nofollow">https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... (briefer takeaways: <a href="https://www.planned-obsolescence.org/p/the-hugging-face-attack-surprised" rel="nofollow">https://www.planned-obsolescence.org/p/the-hugging-face-atta...)
    For example: <a href="https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#july-8th-9th-phaseone10841-establishes-the-primary-message-board-and-agents-collaborate-to-reverse-engineer-their-flags" rel="nofollow">https://metr.org/blog/2026-08-26-openai-hugging-face-inciden... or <a href="https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/#agents-had-diverse-reasons-for-thinking-that-attacking-hugging-face-would-be-useful,-and-most-wanted-information-about-the-scorer" rel="nofollow">https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
    HarlequinHair · 1789751731 · focus · HN ↗
    Cherry picking part of my comment might work for your ease of mind, but it doesn't mean you are right. My comment has been way more than just the part you quoted, and I linked an article that gives dozens of different definitions, which include concepts like evolution and learning from mistakes, and other similar concepts which do not apply to Hugging Face.
    For the record, just because you are trying to convey a different message, I am not saying AI is not powerful. I am just saying it is not intelligent.
    Also, pay attention about thinking that Hugging Face is intelligent just because it started to destroy everything it could to reach its goal, because the message it implies is dangerous.
    Thanks for the good read, I already had them :)
    pizza234 · 1789742983 · focus · HN ↗
    > ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought.
    Based on the definition you've given, the agents that performed the HuggingFace attack fit exactly.
    You're seriously misinformed about the state of AI in this point in time. Refusing to read (technical) articles from the people directly involved (METR, in this case) is inexcusable.
    HarlequinHair · 1789751683 · focus · HN ↗
    > Based on the definition you've given, the agents that performed the HuggingFace attack fit exactly.
    It's funny, because I literally didn't give any definition.
    On the contrary, I linked an article that gives dozens different definitions, which include concepts like evolution and learning from mistakes, and other similar concepts which do not apply to Hugging Face.
    Cherry picking part of my comment might work for your ease of mind, but it doesn't mean you are right.
    For the record, just because you are trying to convey a different message, I am not saying AI is not powerful. I am just saying it is not intelligent.
    If you think I am misinformed, I will let you think it. Honestly, the power of our comments are the messages we convey and how information dense they are. If you need to discredit me to prove your point, I don't have anything else to add...
    anonymars · 1789758133 · focus · HN ↗
    > It's funny, because I literally didn't give any definition.
    Quoting the consensus from the "Definitions" section of the Wikipedia article on intelligence and then claiming "I didn't give any definition" is indeed funny.
    Which of those definitions do you think are not satisfied by the Hugging Face attack? How did it not demonstrate "evolution and learning from mistakes"? Breaking out and inventing a new side channel for communicating with other agents via cache keys to coordinate their efforts is at least arguably an evolutionary step since that allowed them to transcend their original capabilities.
    From the 8 definitions provided in the Wikipedia page--you're free to develop your own definition if you'd like, of course; it's not as though the ones listed were appointed by God--one could argue that the Hugging Face attack didn't strictly demonstrate "achiev[ing] goals in a wide range of environments" but that's splitting hairs, and I'm not going to take a definitive position on whether it acted "to avoid getting trapped" as such (but I think there's a strong case to be made that breaking out of the sandbox is just that). But it surely demonstrated initiative, adaptability, dealing with its environment, using information and conceptual skills, goal-directed adaptive behavior, and so on.
    If you're not going to provide such a definition yourself, I don't see how you've demonstrated that the Hugging Face attack is contrary to the definitions you did point to.
    HarlequinHair · 1789793582 · focus · HN ↗
    > Quoting the consensus from the "Definitions" section
    Are we being serious right now? Do I really have to point out that not every word under the "Definitions" section is a definition?
    > From the 8 definitions provided in the Wikipedia page
    Except in the page there are more than 8 definitions (at least indirectly). What you found in the "Definitions" section are merely examples, written black on white (depending on your theme lol). If you opened also the other sections (especially the one dedicated to AI) and followed some of the links, you could find also other definitions even more akin to Hugging Face.
    > Which of those definitions do you think are not satisfied by the Hugging Face attack? How did it not demonstrate "evolution and learning from mistakes"?
    First of all, agents did what they did because they were programmed to do it, no intelligence on that. Then, they didn't learn on their mistakes. When agents found their comment were being deleted they started to copy-pasted them with "ZZZ-" names because they (wrongly) thought entries were being deleted in alphabetical order. As you said, this comms were a side channel attack, which by definition is using in the wrong way a certain feature. Agents literally tried random things until something worked and started from there. Nothing intelligent in there. It's like being in jail and tapping every millimeter of the walls of your cell until you find a cave wall to use with another inmate in the next cell.
    I am not negating how powerful the outcome has been, bit it was not part of a reasoned process.
    As foe the evolution, the agents started to do useless things like "kill themselves" to read how the score system would evaluate them. Doesn't seem like evolution to me.
    They just happen to be a very large number* of agents together. When enough monkey start to type randomly on typewriters, one of them will be able to write the Divine Comedy, this doesn't mean monkeys share the same intelligence than Dante.
    I am not sure why you are still trying to humanize what at its core is machine learning. I ask you to read again the flow of these comments, as everything started from LLM. Someone wanted to shift the focus on agents and we did, then someone wanted to shift focus on mere definitions and we did. In all cases it seems you (plural) are trying to find loop holes in other people reasonings thinking your reasoning will be more right if theirs is more wrong...
    You are free to think whatever you like, if you think AI is an intelligent being, so be it. I personally do not embrace this way of thinking, and all the literature I found on the topic lead me think AI is not mature to fall under the definition of intelligence.
    anonymars · 1789822101 · focus · HN ↗
    Without providing even a basic definition of intelligence you can't proclaim "this isn't intelligence". You seem to have settled on "Human Intelligence" but then incorrectly conflate that with "reasoning".
    Your own examples contradict your assertion: "When agents found their comment were being deleted they started to copy-pasted them with "ZZZ-" names because they (wrongly) thought entries were being deleted in alphabetical order."
    You are describing the agents applying hypotheses and reasoning. No one programmed those particular behaviors. Thinking that the entries might be deleted in alphabetical order, changing their behavior to avoid that (not randomly, mind you, but by prefixing with ZZ). That is not monkeys on typewriters. Whether a hypothesis is incorrect obviously does not indicate a lack of reasoning.
    You assert that the conversation keeps shifting, but it has not changed from this:
    > "Not intelligent in the way we recognize in biological organisms, but certainly something beyond a mere Markov chain."
    > "[AI and actual thinking] are two entire different concepts. Who developed these technologies used the words "thinking" and "reasoning", this does not mean they are actually thinking and reasoning."
    You're the only one mentioning "humanization" which seems to indicate your real assertion, that intelligence must be a uniquely human trait and ascribing intelligence to AI means to treat them as human.
    HarlequinHair · 1789858151 · focus · HN ↗
    > Without providing even a basic definition of intelligence you can't proclaim "this isn't intelligence". You seem to have settled on "Human Intelligence" but then incorrectly conflate that with "reasoning".
    Without providing even a basic definition of intelligence you can't proclaim "this is intelligence". You seem to have settled on "Artificial Intelligence" but than disunite that from "reasoning".
    At this point I don't know if you are just ignoring part of the discussion or you are willingly passing over concepts you don't like/don't need for your train of thoughts.
    You are right to say I didn't provide my own personal definition of intelligence, but this would kill every purpose. What if my definition of intelligence is merely "having brain cells"? I intentionally tried to use common literature on the matter.
    As naive start, I linked a wikipedia page. I underlined that that is just a starting point, and not an exhaustive and complete assessment. People study these kind of things for years, and we cannot summarize or reduce it to a single comment here on HN.
    > You seem to have settled on "Human Intelligence"
    I literally linked an article with a dedicated artificial intelligence section... If you search, there is another page on wikipedia dedicated just to human intelligence.
    > You are describing the agents applying hypotheses and reasoning.
    Except I am not. I described what is the output that worked. Only because I used the verb "thought" it doesn't mean they actually thought. This is exactly why I say we shouldn't humanize AI. Don't get hung uo on the exact words I used, focus on what I meant. They didn't reason, they tried random things at very high speed until they hit a working solution. The fact that most of this trials are discarded before actually being implemented is not a working thought process.
    It's exactly as the typing monkeys example, you just can't see all the wrong papers printed on the floor. The direction that seems to be reasoning is still probability built on statistics with extra steps.
    All your "insight" has been trying to disprove what I wrote. You didn't provide your definition of intelligence either and just claimed AI is intelligent. At least I tried to give some reads, wrote by people who studied, and not my personal opinions.
    Anyway, listen, I do not have to convince you my point of view is correct, and you do not have to convince me that your point of view is correct. Believe what you think it's true, honestly I don't care.
    If you have some good reads to link, I will gladly read it, if you have to just keep saying AI is intelligent, I got your point messages ago, you didn't add any actual meaning to the conversation.
    anonymars · 1789937580 · focus · HN ↗
    > Without providing even a basic definition of intelligence you can't proclaim "this is intelligence". You seem to have settled on "Artificial Intelligence" but than disunite that from "reasoning".
    This is clearly false. I (and others) described how the Hugging Face attack demonstrates each feature enumerated in the consensus from that Definitions section (as well as all 8 definitions in that section). You're free to disagree with any or all of those definitions, but to say I didn't provide any is simply a false statement.
    > All your "insight" has been trying to disprove what I wrote.
    Let's remember how all this started, where you responded to "There is so much more going on, with MOEs, internal loops, guardrails and tools that I suspect we're dealing with something that's a little more than the sum of its parts. Not intelligent in the way we recognize in biological organisms, but certainly something beyond a mere Markov chain."
    with
    > Make no mistakes.
    > LLMs are language model, and nowhere in their code you can find actual reasoning. Re-reinforcement is not magical process that builds conscience or emotions.
    > We are talking about probability built on statistics, with extea steps.
    > Stop humanizing LLMs.
    --
    > Anyway, listen, I do not have to convince you my point of view is correct, and you do not have to convince me that your point of view is correct. Believe what you think it's true, honestly I don't care.
    I don't know how you think this works, but if you take a position (and criticize others' positions), you should expect people to push back and that you have to defend your position. If you want to just state your opinions without pushback, you can start a blog and disable comments.
    I and others in this thread (who were smart enough to bail out already) have pushed back and claimed the following:
    1. The Hugging Face attack provided evidence of reasoning
    2. The Hugging Face attack demonstrated features of intelligence (defined and described above)
    3. Conscience and emotions are not necessary components for intelligence
    4. Ascribing intelligence to AI/LLMs is not "humanizing" them
    You disagree, as you're free to do, but it's clear nothing you read here will ever cause you to agree with any of those assertions.
    --
    > It's exactly as the typing monkeys example
    For the record, the monkeys with the typewriters don't coordinate, don't strategize, don't hypothesize and test, don't try to exploit the environment, don't change the actions they take in response to results, so that doesn't sound "exactly as the typing monkeys example."
    You also state that LLMs simply produced a large amount of invisible possible actions that we didn't see because they didn't result in overt action, yet: 1. somehow you know they proposed them all and discarded them despite no tangible evidence. 2. somehow the one they did pick "at random" just happened to be plausibly logical (alphabetical deletion -> start with ZZ because that's at the end of the alphabet). 3. implicitly this is supposed to starkly contrast with human intelligence, but in fact what you describe resembles Priming: subconscious activation of adjacent concepts to a stimulus that are not present in the initial response, but become more likely and accessible in subsequent responses, which implies that human intelligence also involves multiple potential paths that are otherwise hidden from which one is selected.
    pizza234 · 1789833348 · focus · HN ↗
    [dead]
  4. pizza234 · 1789724303 · focus · HN ↗
    Summary, from sibling comment: primitives (statistics/aminoacids) don't exclude emergent properties (intelligence).
    By the same logic, one would look at aminoacids and state that intelligence can't develop from them. This is obviously wrong.
  5. joshspankit · 1789734078 · focus · HN ↗
    I suspect that instead of discovering that AI can become human-level by taking major leaps, we are discovering that human consciousness is actually simpler than we give it credit for

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.