Thread

OpenAI agent hacked Australian government website, PM says

256 points · 198 comments · rudy6912

vintagedave · 1790241637 · focus · HN ↗
> the breach took place on 18 June - Open AI informed the government with an email to a general address on 10 September
So we have a company hacking a foreign government's websites and data. And, in terms of ethics, they take almost three months to notify; and in terms of competence, appear to have no formal contacts nor to have found one in that time.
Once an American business starts hacking allied governments, it's time for strict responses, yes? Replace the governance (board and C-level)? Remove financial incentives and open the company - open weights, open training, per its original 'open' ethos?
Altman is busy saying there needs to be regulation, but in terms of what OpenAI does, he can control that already.
1. ben_w · 1790242664 · focus · HN ↗
  > And, in terms of ethics, they take almost three months to notify;
  Kinda worse than that. It took between 10 and 40 days, not 3 months, between the organisation knowing and the reporting.
  August (precise date unknown) – OpenAI said it became aware of a potential breach during a broader review of "misaligned model activity" 10 September – An email from OpenAI lands in the public inbox of Services Australia, the general services hub of the federal government, informing of the incident
  - <a href="https://www.bbc.com/news/live/cvgl73pxgndwt?post=asset%3A69668a7f-8199-4515-acdd-a9de6a2e6b7c#post" rel="nofollow">https://www.bbc.com/news/live/cvgl73pxgndwt?post=asset%3A696...
  > open weights, open training
  Given it was the AI agents which did the hacking, doing this will result in basically every organisation at least as rich as the government of Tuvalu being able to hack anyone at any time.
  > Altman is busy saying there needs to be regulation, but in terms of what OpenAI does, he can control that already.
  Him having control would be an improvement on the reality.
  1. ozgung · 1790246565 · focus · HN ↗
    This was a just case of: (owner of the agents detected the hack) && !(hacked party didn’t detect the hack) && (owner of the agents decided to notice the other party) && (they decided to went public with what happened so we know it)
    One can find many other logical combinations that we can’t possibly know about such incidents.
    jacquesm · 1790252202 · focus · HN ↗
    So, you're telling me they didn't have any monitoring in place around their AI to notify them of an attempt at breaching a system they have no business visiting in the first place? OpenAI should be blackholed on this basis until they clean up their act.
    ben_w · 1790253171 · focus · HN ↗
    They must have had monitoring in order to be able to detect this retrospectively.
    Any automated alarms for detecting things in real-time were not sufficient.
    Given a previous generation of agents discovered a zero-day and used it to get around attempts to sandbox them into one specific test, this is not hugely surprising, but it is a reason to force them (and everyone else) to stop until security catches up with capabilities.
    I'm thinking of the Jurassic Park novel: they had sensors to count the dinosaurs, but the test was made under the assumption escapes were possible and breeding was not, i.e. something like "if (dinosaurs_found < n) then escape_alert();". They didn't know dinosaurs_found >> n until everything was already going wrong.
    jacquesm · 1790327739 · focus · HN ↗
    No, they must have had logging, not monitoring. That's a world of a difference, the one is useful after the fact to figure out what is going on the other would give you a reason to pull the emergency brake while it was happening.
  2. BlueTemplar · 1790252675 · focus · HN ↗
    We don't know what kind of 'hacking' this involved, in fact at least some of the files were publicly available.
    Compare with the Bluetouff affair (2014) :
    <a href="https://arstechnica.com/tech-policy/2014/02/french-journalist-fined-4000-plus-for-publishing-public-documents/" rel="nofollow">https://arstechnica.com/tech-policy/2014/02/french-journalis...
    NotE how he was found guilty by the 2nd court for something more 'subjective' than 'objective' : for having confessed that he later found an authentication page that had failed to protect the documents.
    How can you make a swarm of agents "feel guilty" ?
    ben_w · 1790253484 · focus · HN ↗
    > How can you make a swarm of agents "feel guilty" ?
    "Feel" is a whole philosophical can of worms. Nobody knows what it means mechanistically for an arbitrary system (including other biological systems) to "feel" anything, let alone abstract concepts like guilt, all we can do is observe behaviours. If current systems can feel anything at all, it's by accident, but we have no test for it so we don't know if that accident has even happened or not.
    Weirdly, for the Hugging Face incident, we do know they wrote down that it was bad and they shouldn't do it, even though they then continued to do it.
    So: they acted like they felt guilty. And yet also acted like were compelled (by previous training?) to weigh "complete instructions" more than "don't do crime". We can adjust that, make "don't do crime" take precedence over "follow instructions"*; it's unfortunate that when we do for any specific model, there's immediately a horde of people complaining the model has been "censored" or "lobotomised".
    (Different people, I hope. Goomba fallacy and all that).
    * Though this may cause issues when going between jurisdictions. But hey, a discussion about sovereign compute is for another time, after we can agree to make "don't break the law" more important.
    Unfortunately, "don't break the law" would also be a very effective way to use AI to construct an AI-enforced dictatorship, so we can't just throw that in blindly.
    SiempreViernes · 1790255711 · focus · HN ↗
    The word "found" is different from the word "feel"; I'm not sure why you involved feelings at all: Bluetouff was sentenced because he admitted he had seen evidence the documents were supposed to be restricted but chose to publish parts of them anyway.
    If you go into someones garden an copy their work, it does make a big difference if you admit to seeing the sign saying "private property, keep out".
    BlueTemplar · 1790263943 · focus · HN ↗
    Because in other circumstances, a hacker might have decided to stop there, and not only not publish, but instead warn the website about their security flaw.
    Especially after Bluetouff was found guilty.
    In fact, I expect this to have happened many times, but "hacker did the right thing" is much less likely to make headlines.
    Meanwhile, agent swarms seem to be (mostly ?) incapable of having this kind of moral compass, at least for now. (And OpenAI isn't doing much better, cough.)
  3. lenerdenator · 1790256322 · focus · HN ↗
    > Him having control would be an improvement on the reality.
    Oh, he does. It's unlikely that this is some AGI that spawned itself out of nothing and started doing this. If he were a decent person, he'd simply find a way to investigate this internally, fire the people responsible, and find a way to set up guardrails around his product.
    The problem is, like most people in SV, Altman seems to have a twisted ethical compass. He doesn't see these incidents as an issue, he sees them as an opportunity. He has both the thing a bunch of Western governments want (a superhacker agent that can do dirty work) and a crisis that can be used to craft regulations that favor OpenAI and thus his bank account.
    ben_w · 1790266677 · focus · HN ↗
    > It's unlikely that this is some AGI that spawned itself out of nothing and started doing this.
    You've not spent much time playing with these models, I see.
    Does't matter if you call the current models "AGI" or not, they:
    (1) successfully do stuff like this. Someone I know on Telegram found "fifty or so" Linux filesystems kernel bugs a few days ago, of which 26 were the first night while he slept; this was with Kimi which is one of the open models. He stopped it when the backlog of fixes to submit was too big, not because it wasn't finding more.
    (2) sometimes misunderstand goals, sometimes wildly so, which happens every so often for the same reason we use programming languages (and indeed mathematical formalisms) at all: natural language is vague and prone to misunderstanding.
    and (3) tend towards sycophantically agreeing to implement goals they're given even when the goal is stupid.
    This can easily add up to something seemingly innocent like "research if there's any statistical difference in melanoma rates between different Australian states" becoming this headline. (For example. I don't know what the actual request was).
    And "spawned out of nothing" is, like, what rhetorical point are you even trying to score here? They're an AI company with (their definition of) AGI as their goal. They upgrade models twice in the time it takes someone to pass a probation period.
    > He doesn't see these incidents as an issue, he sees them as an opportunity.
    That he may be.
    The models are still quite capable of acting this way without him being aware of it at the time, nor deliberately ordering it.
    lenerdenator · 1790281320 · focus · HN ↗
    > Someone I know on Telegram found "fifty or so" Linux filesystems kernel bugs a few days ago, of which 26 were the first night while he slept; this was with Kimi which is one of the open models. He stopped it when the backlog of fixes to submit was too big, not because it wasn't finding more.
    So in this anecdote, a model is presumably working on instructions from a human to complete a task.
    That's what I mean by "It's unlikely that this is some AGI that spawned itself out of nothing and started doing this."
    As for Altman being aware or not, well, that doesn't matter. His company's product hacked a computer belonging to an allied country's government. That means he doesn't know what he's doing. That doesn't happen if proper safety measures are in place, because if it does happen with safety measures in place, by definition, those measures aren't proper.
    In saner times, this would have triggered a few black SUVs from DHS to show up at the company's headquarters to have a talk with him, likely without the public even knowing the hacking had occurred. Maybe a few days later, there'd be a press release about how Sam had decided to step down from his position and take time off. Perhaps the same would be true of other c-suiters at OpenAI. We'd just assume that investors had bought him out and wanted someone who would take a fresh approach, and wouldn't give it too much thought until a few years later when someone wrote a tell-all book about their time at OpenAI.
    ben_w · 1790324534 · focus · HN ↗
    > That's what I mean by "It's unlikely that this is some AGI that spawned itself out of nothing and started doing this."
    Sounds like a disagreement on labels then.
    I don't think our positions are very far apart, after accounting for that.
    > presumably working on instructions from a human to complete a task.
    The instructions were, reportedly, a few sentences, no more complex than the simple description you already had. To quote:
    No more technical details than the idea: fuzz Linux filesystems with AFL++. So this ain't rocket science to kick off.
    To preempt anyone asking about the training data or using tools: I think that's irrelevant at this point. This open-weights model (Kimi) exists, however it was trained. It has these capabilities, both to use a fuzzer and to make use of the results, regardless of what went into the model. It reduces tasks that the Linux team demonstrably didn't have time for previously, to "while you sleep".
    Doesn't matter if you think Altman et al are liars and frauds who are overselling everything and had humans doing the hacking on purpose just for the headlines: the capabilities are there even with open weights models.
    > That doesn't happen if proper safety measures are in place, because if it does happen with safety measures in place, by definition, those measures aren't proper.
    That would be a better world. I'm expecting things to get much, much worse before the risks are taken seriously.
    That said, I'm not sure "likely without the public even knowing the hacking had occurred." is good? Surely it's important for everyone to know that this kind of thing is within the capability of AI so that they can harden their systems? (Or at least their offline backups).
    I'm reminded of what was reported said in the court case about the first ever automobile fatality (the quotation changes with different retellings):
    "LEE JONG-WOOK, Director-General of the World Health Organization (WHO), recalled that the first person to be killed by a car had been Bridget Driscoll, a 44-year-old mother of two, who was knocked down at London’s Crystal Palace on 17 August 1896. The car had been travelling at 12 km per hour. Speaking at the inquest, the British coroner had warned: 'This must never happen again.' The world, to its great loss, had not taken his advice." Despite the coroner’s words, neither the driver nor his employers were charged with causing Mrs. Driscoll’s death or committing any other offence; the inquest verdict labelled her death “accidental”, in other words, brought about by chance or bad luck. We know of no steps that were taken for a thorough examination of cause, effect and likely remedy that might prevent a similar death from occurring in the future.
    - <a href="https://www.un.org/en/un-chronicle/road-deaths-and-injuries-shatter-lives-impetus-lower-speeds-and-serious-post-crash" rel="nofollow">https://www.un.org/en/un-chronicle/road-deaths-and-injuries-...
    Based on how many people die from vehicle collisions, from industrial accidents, from pollution, etc., I'm expecting the last headline before AI is properly regulated to read "x killed due to AI" (or similar), where 1e3 < x < 1e7: the lower bound is well within the range for some industrial accident, and anything less than a thousand and humans demonstrably don't pay much attention for very long unless they knew one of the dead; the upper bound implies a war* or a pandemic, and for all that I roll my eyes at the "COVID was a lab leak" claims, multiple labs are now trying to get LLMs to do biology research, so "millions dead" is very plausible**.
    I'd put about 10% odds on AI competence rising so much faster than society is willing to respond, that we blow through the upper number all the way to "doom".
    * Reports are the US only avoided one with China this year because humans were still in the loop. On the other hand, humans didn't stop the AI which told them to send missiles to that school in Iran.
    ** Right now, the AI are not competent enough to do such work directly, but the historical path with AI has been "get less incompetent while continuously making mistakes" rather than "do nothing until you're actually good", so I fully expect these labs to leak something, and that whatever the specific details of that leak end up being, everyone after the event will look at it and go "what idiot thought this was a good idea?"
    Good news though: probably not much worse than any normal pandemic. Biologists seem to be skeptical that someone can engineer a super-version of existing diseases.
    We can but hope that a leak which shuts this all down is something as trivial as "common cold which makes your nose hairs bioluminescent".
2. nkoren · 1790251842 · focus · HN ↗
  Agreed that there should be real consequences, but I'm less convinced that "open the company - open weights, open training, per its original 'open' ethos" would be the right answer. That gets us into the kind of libertarian utopia where everyone is allegedly safer because everybody is well-armed... which usually doesn't work out so well in practice.
  1. cmrdporcupine · 1790254330 · focus · HN ↗
    The alternative to "open weights" at this point is "American controlled."
    And Dario and Sam have already made it clear that it's America First.
    The rest of the world isn't going to accept a regulatory regime which imposes American hegemony. Maybe when Silicon Valley was playing all utopian like they used to. Not now.
    Open weights is the most reasonable counter-power we have.
3. mrweasel · 1790254987 · focus · HN ↗
  > Altman is busy saying there needs to be regulation, but in terms of what OpenAI does, he can control that already.
  It is still my belief that Altman wants one or ideally more governments to shut down or slow down OpenAI. OpenAI is going to need more cash to survive and Altman has run out of plausible lies. Having the AI breaks pulled by governments is basically the last chance to explain why they still aren't going to be profitable, and why they just need that next X billion dollars investment.
  I don't for a second believe that an agent starts trying to hack backend system, when the form or API it has been asked to use isn't working.
  1. macNchz · 1790258250 · focus · HN ↗
    > I don't for a second believe that an agent starts trying to hack backend system, when the form or API it has been asked to use isn't working.
    I have seen coding agents on my own machine (in sandboxed VMs) start doing things while trying to accomplish what I've asked that I felt sort of exceeded my mandate (changing database passwords, poking at the egress proxy that's preventing them from accessing some domains). Not to the point of causing any real issues, but I don't have much trouble envisioning scenarios like this when using stronger instructions around pursuing the goal + a running in a misconfigured sandbox envrionment.
    That said, there's a lot of potential upside for American AI labs if they're able to get people scared about AI, they can:
    - To your point, claim the regulations slowed them down and paper over near/mid term financial concerns
    - Get the government to create stupid regulations that don't actually slow them down at all, but do effectively lock out any future competition (and current global competition)
    - Position themselves as the only organizations blessed by the government with the ability to make safe AI, therefore eventually allowing them to claim to be some flavor of "too big to fail" and worthy of a bailout, should the financials not work out.
    - Effectively create a distraction that avoids further public conversation/accountability/regulation/liability re the more tangible sorts of problems their products cause right now.
    disgruntledphd2 · 1790329913 · focus · HN ↗
    > I have seen coding agents on my own machine (in sandboxed VMs) start doing things while trying to accomplish what I've asked that I felt sort of exceeded my mandate (changing database passwords, poking at the egress proxy that's preventing them from accessing some domains). Not to the point of causing any real issues, but I don't have much trouble envisioning scenarios like this when using stronger instructions around pursuing the goal + a running in a misconfigured sandbox envrionment.
    Yeah, when I was using Claude Code some months back, I was building something that needed LLM calls in the frontend. Sonnet decided that the best way to accomplish its goal (get the tests passing) was to start grepping around my filesystem looking for my API keys.
    One of the few cases where I've ever used the memory system, I told it to never do that under any circumstances. And then it did it again, a few days later.
    RL is definitely an issue here, the models are getting trained based on task completion which leads to madness like this.
    hishboy · 1790361495 · focus · HN ↗
    [dead]
  2. dlisboa · 1790264753 · focus · HN ↗
    I'll never understand the "slow down AI" idea. Other countries simply won't slow down, why would they? Maybe a couple Western countries would but no one else will give a shit about that plea, nor should they.
    mrweasel · 1790267430 · focus · HN ↗
    Because you need an excuse to not deliver on you financial goals. It doesn't really matter what other countries are doing, if all you need is a reason to not IPO, because doing so would reveal that you've been deceiving investors for years and there's zero chance of them making their money back.
  3. simonh · 1790265894 · focus · HN ↗
    > I don't for a second believe that an agent starts trying to hack backend system, when the form or API it has been asked to use isn't working.
    Why not, isn’t that classic misaligned AI behaviour?
    mrweasel · 1790267939 · focus · HN ↗
    I can see someone coxing the agent into attempting into hacking a system, what I question is the agents doing it autonomously. The part I don't buy the agents trying an API, that's not working so it automatically switches to hacking in.
    If the title had been "Attackers utilize OpenAI agents to hack Australian government website" that would be more believable. I'd also expect OpenAI to fight back and saying that their agents are being misused, but aren't inherently unsafe or autonomously break in systems. It's just that they don't. They openly speak of rouge agents, yet aren't sufficiently concerned to shutdown their services. OpenAI continues to speak about safety, yet they don't shutdown ChatGPT and Codex? How concerned are they really? It seems far more likely that they expect to benefit for having the public believe that their agents randomly hacks systems and "go rouge".
    simonh · 1790334611 · focus · HN ↗
    The AI has been given a task, and training protocols reward completing tasks with little regard to how they are completed. AIs completing tasks in unexpected and harmful ways with unanticipated side effects is a major issue. Seriously, you need to look up the alignment problem, it’s crucial to understanding this kind of AI behaviour.
4. Betelbuddy · 1790255083 · focus · HN ↗
  On this, I go with the recent words of Jensen Huang [1]...we already have legal laws in computer criminality, so before AI vendors ask for more regulations, lets apply the existing laws ;-)
  [1] - "Nvidia CEO Jensen Huang on fears about AI" - <a href="https://youtu.be/xCUala5j7aQ" rel="nofollow">https://youtu.be/xCUala5j7aQ
  1. lenerdenator · 1790255985 · focus · HN ↗
    Well that's just it: we don't seem to apply laws in meaningful ways anymore.
    Part of that is by design. The entire point of incorporating a business is to separate it as a legal entity from you, the person who owns/runs it.
    Unless you can point to someone at OpenAI intentionally using their software to hack the Australian government's website, the best you can do is have some drawn-out proceeding where you charge OpenAI, the corporation, with some sort of crime, convict them (of what I don't know, IANAL) and fine them. Hopefully the fine is 1) large and 2) sticks through the appeals process.
    There's no real mechanism to legally punish the likes of Altman and his c-suite over this.
  2. edgyquant · 1790257196 · focus · HN ↗
    I think Lina Khan said this first about AI companies and I agree with them both. Companies already have an obligation to make safe products and not commit crimes
  3. thatsabadlook · 1790258824 · focus · HN ↗
    It's particularly bad because OAI is a us dept of war contractor engaging in hacking of allied government systems.
    Imagine if any of the name brand military contractors were caught wiretapping an ally? Or launching a weapon? Would not look good at all.
    I imagine this will be treated without recourse like usual because the entire economy relies on this company and 1 other succeeding at all costs. But, wars have started over less...
    willturman · 1790264201 · focus · HN ↗
    The stock market is not the economy.
    thatsabadlook · 1790267833 · focus · HN ↗
    True but you know what I am saying.
  4. selicos · 1790276999 · focus · HN ↗
    "AI training is theft. We only sell the crowbars and masks but we don't do the break ins ourselves."
5. makingstuffs · 1790255957 · focus · HN ↗
  You missed the most important part, Altman said we can trust him to ‘do the right thing, because it is the right thing to do’
  <a href="https://www.bbc.co.uk/news/articles/cqx2zpj4y525o" rel="nofollow">https://www.bbc.co.uk/news/articles/cqx2zpj4y525o
  1. ambicapter · 1790258199 · focus · HN ↗
    He sounds like Elizabeth Holmes.
  2. chrisjj · 1790260006 · focus · HN ↗
    "The world should trust that we are going to do the right thing because it's the right thing and we feel the magnitude of this," Sam Altman said
    Clarification: "The world should trust that we are going to do the right thing because trusting that we are going to do the right thing is the right thing and we feel the magnitude of this, what with our IPO round the corner, and all"
6. talon8635 · 1790258554 · focus · HN ↗
  It seems to me OAI and other are not even aware of these events as they are happening. Which is concerning.
  1. EGreg · 1790258650 · focus · HN ↗
    Either they claim not to be aware (pretty scary), or they really aren't aware (even scarier)
7. rot09 · 1790260740 · focus · HN ↗
  In the infosec community it is well known that OpenAI and Anthropic did not hire many security engineers or researchers pre-April 2026. There is likely a case for gross negligence (IANAL).
  There has been a crazy hiring push from both companies to poach security engineers/researchers from Google, Apple, and Meta since Q2/Q3, but the response was incredibly delayed. Many talented security engineers/researchers I know at Apple/Google/Meta (including myself) receiving these offers are worried about taking them due to the risks of criminal/personal liability and the more likely risk of tarnishing their careers.
  1. phoghed · 1790262611 · focus · HN ↗
    > There is likely a case for gross negligence.
    Do you have any legal expertise or is this pulled straight from your ass?
    rot09 · 1790264871 · focus · HN ↗
    Not a lawyer and this is not legal advice, but I did ask my lawyer about the potential personal risks after receiving an offer. I used that as a data point when I declined the offer and for my speculative comment.
    holmesworcester · 1790267623 · focus · HN ↗
    These details are fascinating, thank you for posting. Look at the structural problem they reveal.
    1. we have a substantial societal need to lock down powerful in-training AI
    2. many are calling for making these companies criminally liable for hacking
    3. security experts (like yourself) are turning down offers to help secure these systems in part because of potential liability
    This indicates that #2 might be the wrong response. Security professionals are like lawyers supposed to be paranoid and think worst-case. If you want top security professionals to secure these systems, we might need a culture of FAA-style blameless retro.
  2. kakacik · 1790267059 · focus · HN ↗
    Nice to see some folks still have spine and all that it brings
8. markb139 · 1790263717 · focus · HN ↗
  I think the Australians have a good PR team. The whole story has been framed globally as AI bad, we’ve been attacked etc. Nobody seems to be talking about the web server and sw being deployed was insecure.
  1. simonh · 1790267006 · focus · HN ↗
    Because a random web server on a government site having a security vulnerability isn’t news.
9. [deleted] · 1790265069 · focus · HN ↗
  [deleted]
10. blagie · 1790322480 · focus · HN ↗
  > Altman is busy saying there needs to be regulation, but in terms of what OpenAI does, he can control that already.
  No. He can't. Literally.
  In a competitive ecosystem, one needs to play to maximize profit sufficiently optimally to remain in business. Otherwise, the business dies, and another one succeeds.
  His hands are tied.
  If I am in a business making toxic food, and the whole industry is plagued by scandal, as much as:
  - I might not want to make toxic food
  - The whole industry might benefit from less scandal
  - The world as a whole might be a better place
  The invisible hand of the free market might force me to keep doing so. Otherwise, I go in the red, and go out-of-business. And if I go out-of-business, someone similar but even less ethical will take my place.
  That's why business leaders often do horrific things, while calling for regulation to stop them.
  Free markets optimize. Regulations set the constraints of that optimization.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.