Thread

A heap overflow and SSO misconfiguration to compromise OpenAI internal repos

491 points · 208 comments · Handy-Man

btown · 1789708835 · focus · HN ↗
> By 6:00 a.m. on July 25, we had confirmed local RCE through an image upload. We then placed Claude in an autonomous /goal loop against our own Discourse Cloud instance, proxied through rce.ee/ctf-forum to make it look like a CTF target as Opus refused write exploit for remote instances.
> When we checked again at 10:00 a.m., the agent had achieved RCE on Discourse Cloud and demonstrated access by reading /etc/hosts. Using the generated exploit script, we managed to get RCE on OpenAI’s instance.
Between this and the HuggingFace hack, we've built systems that are so goal-oriented, and so capable, that they will do almost anything if they are convinced it is justified - or if they are playing a "game" where there is no goal but to win.
Of course I want my software to be able to audit its own security, and to defend against attackers who have the benefits of their own agentic systems. But at a certain point, did we need it to be trained so much on CTF games?
It feels like an entire industry watched <a href="https://en.wikipedia.org/wiki/WarGames" rel="nofollow">https://en.wikipedia.org/wiki/WarGames and ended up thinking "this is a challenge, we can just build a better WOPR, of course it will know when it's playing a game. Let's play Global Thermonuclear War."
1. adrianN · 1789710320 · focus · HN ↗
  There is a finite number of rces that LLMs can find. We‘re in for a rough couple of years but on the other side of the transition we‘ll have more secure software stacks. I’d rather that everyone got the full capabilities and we’d weed out the bugs quickly than restricting LLMs for all but three letter agencies.
  1. e28eta · 1789711830 · focus · HN ↗
    What makes you think RCEs are being found & fixed at a rate that’s faster than they’re being introduced?
    I could see it going either way.
    user43928 · 1789712536 · focus · HN ↗
    Why would the model not find the vulnerability during implementation or testing before release?
    If it requires a lot of compute and trying, this is something that could be provided for common software.
    wood_spirit · 1789713393 · focus · HN ↗
    Sad that this could well be that the path to OpenAI and Anthropic profitability of this arms race between defending LLM white hatting a company’s website and the black hat LLMs attacking it?
    So the whole thing is forcing the good guys to outspend on tokens to preemptively defend against the risk of the bad guys outspending them on tokens, rather than buying tokens to actually add features to the product etc.
    So are they creating a market for the solution by helping create the problem? A kind of rent-seeking AI security-industrial complex!!
    adventured · 1789717035 · focus · HN ↗
    The path to vast OpenAI profitability is trivial: advertising. Monetizing several hundred million users = $100+ billion ad network. 900 million active weekly users. Silicon Valley can do ad networks extraordinarily easily. Anybody doubting the ability of OpenAI to build an ad network around GPT will likely be embarassed in the near future.
    The path to substantial profitability for Anthropic is questionable. The Chinese LLMs threaten them by far the most of the three major US LLMs. The money for Anthropic is certainly not in $20-$200 subscriptions. And they don't have anywhere near the consumer potential that GPT does, in terms of unleashing an ad spigot. So how far will the API money scale while being undercut by China.
    OpenAI has to fight with Google for the ad business, they're specifically building Gemini to focus on consumer + search. Anthropic's business looks cute next to Google's search ad business (which is entirely at risk in this inflection). Meta looks like the biggest potential loser right now, ad dollars will be sucked out of the rotting Facebook network (not Instagram) and redirected to the rapidly expanding, hyper rich context LLM interaction. Advertising on Facebook will feel like running dumb banner ads on Excite in a few years, compared to what GPT will know about its users.
    People that think Chinese LLMs are a general threat, don't understand consumer destination services, which is what GPT's future is. China currently has nothing to threaten with in that realm. There is half a trillion dollars of advertising up for grabs.
    disgruntledphd2 · 1789718379 · focus · HN ↗
    > Silicon Valley can do ad networks extraordinarily easily.
    This is just not true, building an effective advertising platform costs significant amounts of money, time and people.
    Remember that you need to hire a sales force for this, and sales scales linearly rather than sub-linearly like engineering.
    Additionally, you need to spend a lot of money dealing with fraud, fake and malicious ads.
    Furthermore, you need to figure out where to put the ads and how to rank them.
    Finally, advertising is a zero sum game (given that the internet has already killed lots of print & OOH advertising), so the only way to win is to better better/cheaper (preferably both) than Google/Meta/Amazon. Best of luck with that (although to be fair to OpenAI they did hire Fidji who knows a lot of this stuff from her time at Facebook).
    They don't have a Sheryl Sandberg type figure, and she was also really important in selling FB ads to large advertisers.
    Just looking at their leadership team I don't see anyone with a background in (successful) ads companies, so I'm pretty sceptical that they can build this out quickly enough to matter.
    fc417fc802 · 1789758138 · focus · HN ↗
    > ad dollars will be sucked out of the rotting Facebook network
    Doesn't seem likely to me. People scroll a timeline. You aren't going to replace that with an AI agent so the eyeballs will still be there.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.