Thread

Pentagon says overreliance on AI contributed to missile strike on Iran school

974 points · 553 comments · devonnull

legitster · 1790105097 · focus · HN ↗
> It found the U.S. “failed in its obligation to do everything feasible to verify” that the school was a military objective and that the failure “went beyond mere negligence.” The report said the United States “directed the strikes at the building of the school while being aware of a substantial risk of striking a civilian object and acting recklessly as regards the possibility that this would happen.”
Reading the details, "AI" doesn't really seem like the culprit -it's a scapegoat.
The intelligence that it was no longer a military target never entered the target database, the team that was responsible for vetting the target list was gutted, and said team was never even consulted.
The White House wanted 1000 targets and pulled from their database without any due diligence. Whether it was an AI call or an SQL query - this was from pure human maliciousness and incompetence.
1. xlayn · 1790107269 · focus · HN ↗
  Now, ask yourself if you want AI taking decisions around your life... it will pull numbers... someone will have a "goal" of 1000 "hits" and there you go, 2-3 months in jail working to check what went wrong...
  But don't worry, the other P AI system will ensure that the first one doesn't do mistakes...
  1. toomuchtodo · 1790113544 · focus · HN ↗
    Lawsuit Says UnitedHealth Used AI With a 90% Error Rate to Deny Care - <a href="https://www.yahoo.com/news/us/articles/lawsuit-says-unitedhealth-used-ai-173017952.html" rel="nofollow">https://www.yahoo.com/news/us/articles/lawsuit-says-unitedhe... - September 18th, 2026
    UnitedHealth uses AI model with 90% error rate to deny care, lawsuit alleges - <a href="https://news.ycombinator.com/item?id=38299921">https://news.ycombinator.com/item?id=38299921 - November 2023 (58 comments)
    <a href="https://litigationtracker.law.georgetown.edu/wp-content/uploads/2023/11/Estate-of-Gene-B.-Lokken-et-al_20231114_COMPLAINT.pdf" rel="nofollow">https://litigationtracker.law.georgetown.edu/wp-content/uplo...
    teravor · 1790118858 · focus · HN ↗
    that smells like bullshit to be fair. if you have a system that makes errors over 90% of the time then you have a system you can leverage to make fewer errors.
    the situation would have to be very particular/contrived for that not to be the case.
    toomuchtodo · 1790119302 · focus · HN ↗
    <a href="https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does" rel="nofollow">https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...
    1718627440 · 1790180894 · focus · HN ↗
    The word purpose exists to describe something that is independent of it's current behaviour. If you do not mean that, don't use that word.
    flossly · 1790119390 · focus · HN ↗
    What if you want errors to happen, e.g. deny care, bomb civilians; in that case 90% error rate is what you want, and plausible deniably so!
    ImPostingOnHN · 1790122630 · focus · HN ↗
    What makes you think they are financially motivated to make fewer "errors" (improper denial of care while keeping premiums)?
    degamad · 1790123581 · focus · HN ↗
    > the situation would have to be very particular
    The situation is very particular. When the system makes an error, I make a bigger profit. If the system made fewer errors, it would approve more things that cost me money.
    Why do you think I'm motivated to make fewer errors under that situation?
    dylan604 · 1790123908 · focus · HN ↗
    It sounds like something with a 90% error rate would be better to just do the opposite of what was determined and suddenly have a 10% error rate??
    eks391 · 1790130802 · focus · HN ↗
    That's not how type 1 & type 2 errors work. The article is describing type 2 (false negative) and inverting the output has no correlation with the percent of false positives (type 1)
    <a href="https://en.wikipedia.org/wiki/Type_I_and_type_II_errors" rel="nofollow">https://en.wikipedia.org/wiki/Type_I_and_type_II_errors
    benlivengood · 1790124735 · focus · HN ↗
    Nah, it's cumulative errors per claim. E.g. maybe the per-decision-factor rate is somewhere over 50% but overall most claims will have errors because so many factors go into coverage/reimbursement decisions.
    Dylan16807 · 1790126820 · focus · HN ↗
    > if you have a system that makes errors over 90% of the time then you have a system you can leverage to make fewer errors.
    Sure if you're flipping a coin.
    If you're calculating a number from 1 to 100 then depending on distribution a calculation that's wrong 90% of the time is probably useless.
    mschuster91 · 1790144985 · focus · HN ↗
    > if you have a system that makes errors over 90% of the time then you have a system you can leverage to make fewer errors.
    member the three words "delay, deny, depose"? Insurances - and not just in the US, Germany has similarly bad stuff going on - aren't making a profit if they just blindly accept claims. It is much more profitable to deny with a vague "AI" system (or a blanket automated deny) first, and only have a first look with an actual human agent at it when the customer complains or files a lawsuit.
    j4yav · 1790159055 · focus · HN ↗
    I can easily build you a system that will guess the price of a stock tomorrow wrong at least 90% of the time. Even 100% wrong if you'd like. How would you use that to make fewer errors in some other system?
    [deleted] · 1790178314 · focus · HN ↗
    [deleted]
    1718627440 · 1790180819 · focus · HN ↗
    If it is a binary choice you can just do the opposite.
    j4yav · 1790229897 · focus · HN ↗
    What’s binary choice we are discussing here and what’s the opposite that you’d do? Approve every claim that was denied by the AI, and vice versa? Or what’s the binary choice related to stock prices you’re referring to?
    1718627440 · 1790243400 · focus · HN ↗
    I agree, that most real world problems, are not binary choices. But in this case actually yes, just do the opposite.
    For that reason error rates above 50% don't make any sense. Building a system, that is reliably wrong, is as hard as building one that is reliable correct, because the systems are the same, just with the output inverted.
    The worst you can get with a system is 50%, at which point you can just flip a coin, because that is just pure random. Any deviation from that is going to be an improvement.
    [deleted] · 1790246747 · focus · HN ↗
    [deleted]
    j4yav · 1790271960 · focus · HN ↗
    I always thought insurance claims were more complex, some things may be wholly or partially covered on a line by line basis, and there should be justification provided. In certain cases they go to mediation or even court. It doesn’t seem binary to me but maybe it works differently where you are.
    1718627440 · 1790284735 · focus · HN ↗
    I said "if it is", not that it would be.
    j4yav · 1790310103 · focus · HN ↗
    Oh, haha, sure.
  2. 8note · 1790114124 · focus · HN ↗
    what makes ai special here? a person following an SOP that involves polling an incomplete dataset will have the same 2-3 months in jail
    stuaxo · 1790116429 · focus · HN ↗
    People often tend to stop in these situations.
    jimbooonooo · 1790116907 · focus · HN ↗
    The person can be sued for negligent, retaliatory, and otherwise crappy decision making.
    iAMkenough · 1790118007 · focus · HN ↗
    Judicially, you can hold a person accountable for "accidentally" killing 100+ children easier than you can a technology system. The technology system will always be used as the scapegoat for the government's atrocities.
    I'm not sure why the "people make mistakes too" crowd can't seem to grasp this concept.
    bradleyjg · 1790122057 · focus · HN ↗
    Do you have three examples from 20 years in Afghanistan of analysts being held judicially accountable for target selection errors?
    iAMkenough · 1790122120 · focus · HN ↗
    Seems like a weird goalpost.
    Doesn't invalidate my point. I didn't say it was easy to hold a person accountable, just easier than a techonology system, whose owners point the finger back at government.
    I think we both want to see justice for those 100+ innocent children murdered (as well as those innocent lives in Afghanistan and Iraq) by the U.S. Military's incompentence.
    There were never "weapons of mass destruction," although Congress was mislead to believe there was.
    In the Minab attack, Congress wasn't even given the option to authorize.
    bradleyjg · 1790122316 · focus · HN ↗
    You’re comparing the current system not to the status quo ante but to a utopian fantasy that only ever existed in your imagination.
    That’s a weird goalpost.
    iAMkenough · 1790124540 · focus · HN ↗
    Sorry, I was taught good Americans don't murder people unjustly. I guess that is a utopian fantasy to think good Americans were serving our country.
    So much for seeking justice for those who can't advocate for themselves. America kills the innocent is accepted fact, I guess. The ultimate pro-life society.
    pessimizer · 1790132424 · focus · HN ↗
    Do you believe every TV ad you see, too? If you do, do you think you should stand in judgement of people who don't?
    Thinking everyone who doubts is a cynic is your problem, and you're continuing to execute on it. When the next liar rolls in and says they fixed everything, you'll be the first to run in bashing people saying "no, I don't think he did."
    Do you not read? Do you not know anything about what America has done?
    Your lecturing is not virtuous.
    iAMkenough · 1790132962 · focus · HN ↗
    Disprove this: America kills the innocent is accepted fact, I guess. The ultimate pro-life society.
    iAMkenough · 1790133521 · focus · HN ↗
    Killing kids overseas is the best way to end up with dead Americans. That's not virtuous, it's just common sense.
    Without accountability and justice for those families that experienced unnecessary loss, they're coming for us to seek their own justice. Rightfully so.
    Can you not take a second to put yourself in their shoes as a parent? If the tables were turned and Iran's government "accidentally" bombed an elementary school in the U.S., I feel like you would have a different opinion on justification for revenge.
    In this instance on the very first day of strikes on Iran, the U.S. Military instigated future conflict by murdering hundreds of innocent children. Your children will pay the price in one way or another.
    Actions have consequences. Trump and his installed sycophants in Congress and the SCOTUS reject that reality, but their constituents aren't as gullible as they banked on.
    When they get their golden parachute, Presidential pardons, and taxpayer funded-protection, you are left holding the bag for bankrolling this atrocity.
    iAMkenough · 1790132364 · focus · HN ↗
    For the sake of argument, I'm also entertaining your idea of what happens after the U.S. spends 20+ years killing innocent family members overseas. And how that strategy has changed since the Iran conflict many years later?
    An end to anti-American radicalism brought on by the repeated pattern of carelessly murdering children of pro-democracy parents in these regions and cutting international aid to other regions?
    If it was my 12-year-old murdered by a "targeting error," I would dedicate my life to revenge.
    Cyph0n · 1790137610 · focus · HN ↗
    This is a war that is generally supported by both Republicans and Democrats - the latter mostly dissenting to the fact that it is being conducted without Congressional approval.
    It is also a war that extends in part from the Gaza genocide, which as we know was overseen and backed by a Democratic administration.
    In other words, this is a war enabled and sanctioned by our government collectively, and not a particular administration or party.
    iAMkenough · 1790144352 · focus · HN ↗
    Did representatives from the U.S. House and Sentate vote to authorize this?
    No. You're full of shit. It's Trump's Executive Branch demanding quotas from the Pentagon that led to bad information being used to murder 100+ children.
    Maybe if Republican and Democrat elected representatives had an oppportunity to authorize the Iran war, I would listen to you.
    Respectfully, your argument is bullshit.
    Produce voting records authorizing the war, or shut up.
    nixon_why69 · 1790158370 · focus · HN ↗
    They haven't voted to stop it, either. We're way past 90 days, it's a choice for congress not to exercise their power.
    Schumer could be raising hell and disrupting everything in the senate in order to force a vote, it would be incredibly popular and define the midterms, but he's choosing not to.
    iAMkenough · 1790166877 · focus · HN ↗
    The U.S. House has, after bad polling more Republicans are turning on Trump leading up to the midterms.
    nixon_why69 · 1790168185 · focus · HN ↗
    Took them a few tries to even get there, and the senate hasn't even taken it up for consideration. Chuck Schumer wants this war, even though 60% of Americans and like 90% of democrats are against it. He just needs deniability to blame it on Trump.
    iAMkenough · 1790168420 · focus · HN ↗
    Chuck Schumer wants farmers to get socialist handouts, and is a little nervous about the size of those checks as his farmer friends triple their diesel expenses needed to operate.
    Cyph0n · 1790203130 · focus · HN ↗
    Just came back to this. I am genuinely curious: were you expecting to start a discussion with such a disrespectful reply, or were you just ragebaiting?
    iAMkenough · 1790309277 · focus · HN ↗
    I'm genuienly curious if you want to give it the time. If not, I understand. Respect is earned, and I'm willing to listen.
    AngryData · 1790193797 · focus · HN ↗
    Unless congress votes on it this is an illegal war. There is no wiggle room here because Iran was not an imminate threat to the US and took no military actions against us.
    1718627440 · 1790180661 · focus · HN ↗
    It's not, if the status quo ante was not deemed to be moral in the first place.
    impossiblefork · 1790154597 · focus · HN ↗
    No, but the US has historically permitted even cold-blooded torture of innocents with no liability, and has asserted the state secrets privilege to prevent lawsuits and prosecutions, so it wouldn't have mattered.
    But that's a US-specific thing. With humans you at least have this theoretical liability-- that there is someone who could be punished.
    pjmorris · 1790123305 · focus · HN ↗
    AI is special because it is much speedier and doesn't have second thoughts. Both of those are disadvantages in this case.
    [deleted] · 1790123916 · focus · HN ↗
    [deleted]
    csomar · 1790129132 · focus · HN ↗
    > and doesn't have second thoughts.
    and so is the Trump administration?
    educasean · 1790130588 · focus · HN ↗
    Any technology can be blamed for “accelerating bad decisions”.
    DirkH · 1790180274 · focus · HN ↗
    AI is a different order of magnitude since unlike all other technology across all of human history you can hand off the task and decision-making itself entirely to the AI.
    It's the difference between a sword and a nuclear missile even though both are "technology"
  3. TheOtherHobbes · 1790129592 · focus · HN ↗
    You seem to have missed the point, which is that the decision to murder a large number of innocent people was made by humans at the top of the chain of command, and 'AI' is being used as a thin pretext for that.
    The pattern matches various other acts of senseless violence against innocents in South America.
    The message isn't "We did this accidentally. We should understand how it happened, so we can make sure it doesn't happen again.".
    It's "We do this to make the point that we can, no one can stop us, and if we're challenged we're going to lie about it."
    geraneum · 1790155463 · focus · HN ↗
    The decision was indeed made by AI and humans failed in their duty to fucking verify it. They green lit and now we have school children dead. No single component of this whole chain is defensible.
    lifeformed · 1790165262 · focus · HN ↗
    Depends on what "decision" means. If I say, "Claude, compile a list of 1000 buildings to blow up," and then skim the output briefly and then press the red button, I wouldn't say Claude "decided" anything.
    geraneum · 1790167692 · focus · HN ↗
    Of course it did! Among hundreds of thousands it decided those 1000 are the ones to blow up! No matter how we twist the meaning of “decision”.
    edoceo · 1790168505 · focus · HN ↗
    Who decided to bomb places? AI? Who decided to pick a fight? AI?
    ajsnigrutin · 1790171477 · focus · HN ↗
    We had a public tender for some 3d model of a railroad project a few years ago, three companies applied and for some reason (corruption) the most expensive one was selected to make the model.
    They blamed excel for the bad choice. Excel.
    I mean sure, you can always blame the tool, i can blame my cars rear parking sensors when i hit someone when backing up during parking, but come on... at some point, a human is responsible. Someone, some person, decided and greenlit the decision to bomb those places and should now be in jail for that.
    I mean, what's next? A building collapses, because the load bearing sections are too weak... do we blame CAD tools now?
    geraneum · 1790182131 · focus · HN ↗
    Non of the example you’ve falsely equivocated have the word “intelligent“ in them though, do they?
    joquarky · 1790186242 · focus · HN ↗
    Just keep moving those goalposts while ignoring your cognitive dissonance.
    geraneum · 1790192789 · focus · HN ↗
    Ah you must be one of those "Guns don't kill people, people kill people".
    ajsnigrutin · 1790211151 · focus · HN ↗
    Great example, if someone is killed by gunfire, does the gun go to jail or does the shooter (a person) take responsibility?
    geraneum · 1790231729 · focus · HN ↗
    I’m assuming you’ve never heard that slogan before. Google it. It’s an interesting read.
    archonis · 1790171628 · focus · HN ↗
    If someone made a table of one hundred potential targets and rolled ten ten sided dice to pick a row to target, do we blame the decision on the dice, the table, Wizards of the Coast, or the person who decided this was a good way to choose targets?
    geraneum · 1790181845 · focus · HN ↗
    You’re conflating who chooses the decision maker with the decision making device.
    So AI’s output is not so different than a roll of dice? Definitely the people who decided to put in such an incapable and unreliable tool should be held accountable. No dispute in that.
    GoblinSlayer · 1790174927 · focus · HN ↗
    Then you suddenly realize you outsourced your thinking to Claude, because it's tiresome to think by yourself.
    1718627440 · 1790180550 · focus · HN ↗
    The decision to blow up 1000 buildings, without any background check wasn't done by claude.
    sillyfluke · 1790169019 · focus · HN ↗
    You should have just kept the last two sentences.
    >The decision was indeed made by AI and humans failed in their duty to fucking verify it.
    The decision to leave the decision to AI was made by humans. It's on the moronic humans for not understanding the limitations and flaws of the software they are using and believing the marketing material without constantly verifying their results.
    How good or bad the software is an almost equally important but separate issue.
    geraneum · 1790190413 · focus · HN ↗
    > The decision to leave the decision to AI
    The decision, to make the decision to leave the decision…
    So much mental gymnastics.
    roenxi · 1790155876 · focus · HN ↗
    > It's "We do this to make the point that we can, no one can stop us, and if we're challenged we're going to lie about it."
    FWIW, forcing people to go along with transparent lies is a popular social manoeuvre. It signals both the power of a group and the power of the person leading the lie.
    Although it is a very frustrating tendency, because honesty and realism tends to be better for the people who have the numbers. It is hard to organise real efforts around lies.
  4. throw0101c · 1790176080 · focus · HN ↗
    > Now, ask yourself if you want AI taking decisions around your life... it will pull numbers... someone will have a "goal" of 1000 "hits" and there you go
    Before anyone does this please let me know beforehand so I can invest in paperclip factories…

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.