Thread

Pentagon says overreliance on AI contributed to missile strike on Iran school

974 points · 553 comments · devonnull

imranq · 1790105401 · focus · HN ↗
> Officials said some users expected Maven to flag stale records or contradictions in the intelligence assembled for potential targets, though it is not clear why they thought the system would do that
This is what happens when people who don't understand AI and its limitations embrace it wholeheartedly, thinking it's going to behave like The ultimate analyst. Except it's not. Its got blind spots in more places than anyone knows. And not to mention the system itself has multiple areas of failure.
1. pants2 · 1790106704 · focus · HN ↗
  Chatbots in general can be quite opaque in how they operate, the biggest issue with agent products is communicating it's capabilities and limitations to users. If you have a bank app it's pretty clear whether or not it can send wire transfers, but a if that's a chatbot, users will have to think to ask it (and even then it could hallucinate)
2. shimman · 1790106943 · focus · HN ↗
  No, this is what happens when software developers don't understand how users actually use their products.
  This was a totally expected outcome, similar to how CV cameras arrest the wrong minorities in neighboring states 100s of miles away because it told the officer it was them; or how health insurance companies use ML algorithms to deny Americans care.
  It's going to happen in other sectors of life and it will continue killing innocent civilians.
  1. whycome · 1790124887 · focus · HN ↗
    Are you saying that letting insurance companies track your driving data will get used to do things like fine drivers directly for law breaking? Never!
    shimman · 1790130249 · focus · HN ↗
    No, I'm saying health insurance companies routinely deny American civilians life saving care because they'd make more money and routinely use similar software that also allowed 200 school age girls to be murdered. Recalling that a CEO was murdered and the person excused is being hailed as a hero over just this. Think people care about this a lot actually.
    <a href="https://www.newsweek.com/united-healthcare-ceo-shooting-ai-lawsuit-1996266" rel="nofollow">https://www.newsweek.com/united-healthcare-ceo-shooting-ai-l...
    <a href="https://www.ama-assn.org/press-center/ama-press-releases/physicians-concerned-ai-increases-prior-authorization-denials" rel="nofollow">https://www.ama-assn.org/press-center/ama-press-releases/phy...
    <a href="https://www.palmbeachpost.com/story/news/healthcare/2026/03/30/ai-is-denying-health-care-claims/88221783007/" rel="nofollow">https://www.palmbeachpost.com/story/news/healthcare/2026/03/...
    _heimdall · 1790152639 · focus · HN ↗
    For better or worse, health care providers at least in the US are for profit businesses. There's been plenty of arguments made to change that, but as it stands we shouldn't be surprised when a for profit business decides what service they provide and to whom they provide it.
    ImPostingOnHN · 1790199328 · focus · HN ↗
    Likewise, we shouldn't be surprised when someone who puts profits over other's lives gets ventillated by one of those lives in downtown New York. And what do you know, nobody was.
    _heimdall · 1790221331 · focus · HN ↗
    What so you mean by ventilated here?
    Forgive my ignorance, I just don't get the reference and don't want to assume its anything front he pandemic response to United Healthcare.
    shimman · 1790267003 · focus · HN ↗
    United Healthcare CEO was assassinated because they implemented policies that denied Americans healthcare due to algorithmic discrimination.
    _heimdall · 1790387882 · focus · HN ↗
    Are you saying said murder was justified and shouldn't be punish? Or even that it should be celebrated?
    ImPostingOnHN · 1790270396 · focus · HN ↗
    I mean the person in question had new air vents rapidly installed in their body.
    _heimdall · 1790387901 · focus · HN ↗
    Are you saying that's a good thing?
    ImPostingOnHN · 1790440199 · focus · HN ↗
    My post very unambiguously explains that is an unsurprising thing. It does not explicitly say it was a good thing, so no, "good" is not specifically what I am saying in that post.
3. TZubiri · 1790121133 · focus · HN ↗
  can't read because of paywall, are we talking the Java package manager Maven?
  1. mahboi · 1790207860 · focus · HN ↗
    Google's military Project Maven? Thought that never happened
4. ImPostingOnHN · 1790123874 · focus · HN ↗
  "I thought the gun would recognize that it was pointed at an innocent victim and not fire, so we're cool, right?"
  Still murder.
  1. bigstrat2003 · 1790147413 · focus · HN ↗
    Actually, no. Murder is the deliberate killing of a person. If the person in your example believed the gun wouldn't go off, then it isn't murder. Negligent as hell, yes. Manslaughter, certainly. But not murder, because it doesn't meet the definition.
    [deleted] · 1790167193 · focus · HN ↗
    [deleted]
    [deleted] · 1790167979 · focus · HN ↗
    [deleted]
    ImPostingOnHN · 1790168082 · focus · HN ↗
    The entire point of the post is that it isn't credible that the person actually thought their killing machine couldn't kill: pointing a gun at someone with the intention of killing, and pulling the trigger, counts as murder, even if you thought the person would magically survive if they were innocent.
    You're free to try to make a case otherwise, but it'll be an uphill battle against reason.
    blitzar · 1790184135 · focus · HN ↗
    Not my fault, I was certain that they would move out of the way of the bullet.
5. skrebbel · 1790140296 · focus · HN ↗
  Come on, they understand it perfectly well. They blew up a school! Of course they’ll happily blame the AI when journos nudge nudge wink wink suggest that they’ll take that as a perfectly good explanation.
  The real headline is “Americans blow up school, blame AI”

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.