Thread

A heap overflow and SSO misconfiguration to compromise OpenAI internal repos

491 points · 208 comments · Handy-Man

btown · 1789708835 · focus · HN ↗
> By 6:00 a.m. on July 25, we had confirmed local RCE through an image upload. We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances.
> When we checked again at 10:00 a.m., the agent had achieved RCE on Discourse Cloud and demonstrated access by reading /etc/hosts. Using the generated exploit script, we managed to get RCE on OpenAI’s instance.
Between this and the HuggingFace hack, we've built systems that are so goal-oriented, and so capable, that they will do almost anything if they are convinced it is justified - or if they are playing a "game" where there is no goal but to win.
Of course I want my software to be able to audit its own security, and to defend against attackers who have the benefits of their own agentic systems. But at a certain point, did we need it to be trained so much on CTF games?
It feels like an entire industry watched <a href="https://en.wikipedia.org/wiki/WarGames" rel="nofollow">https://en.wikipedia.org/wiki/WarGames and ended up thinking "this is a challenge, we can just build a better WOPR, of course it will know when it's playing a game. Let's play Global Thermonuclear War."
1. adrianN · 1789710320 · focus · HN ↗
  There is a finite number of rces that LLMs can find. We‘re in for a rough couple of years but on the other side of the transition we‘ll have more secure software stacks. I’d rather that everyone got the full capabilities and we’d weed out the bugs quickly than restricting LLMs for all but three letter agencies.
  1. e28eta · 1789711830 · focus · HN ↗
    What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced?
    I could see it going either way.
    brookst · 1789736917 · focus · HN ↗
    I'd love to see data, but my intuition is that the average developer has access to dramatically better security reviews and far lower cost than ever.
    There's more software being written than ever so maybe raw numbers of RCE's could be up, but as a percentage, I'd really expect them to be down. Especially among any fairly common software, as all it takes is anyone working on it to get the idea to test.
    d4mi3n · 1789739539 · focus · HN ↗
    What I’m seeing is more developers pushing more code of dubious quality without the ability to respond to feedback on said code.
    You can have the best security review in the world, but if the author of the code is not equipped to understand the feedback it ends up being a moot point.
    The challenge to me seems less technical and more cultural: how do we keep ourselves intellectually honest and engaged when we now spend the majority of our time orchestrating agents and outsourcing the design and thought processes?
    fc417fc802 · 1789757744 · focus · HN ↗
    Hey claude, compare this security review to the current codebase and patch up anything that needs it.
    tyre · 1789751579 · focus · HN ↗
    One issue for developers is that the most powerful models refuse to do comprehensive reviews. You can’t ask Fable 5.1 to find every exploit in your codebase, because that’s indistinguishable from what a bad actor would do.
    Timwi · 1789774770 · focus · HN ↗
    > I'd love to see data, but my intuition is that the average developer has access to dramatically better security reviews and far lower cost than ever.
    Where? If I ask Claude to do a “security review” of my software, it gets blocked as a possible hacking attempt.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.