Thread

Pentagon says overreliance on AI contributed to missile strike on Iran school

974 points · 553 comments · devonnull

legitster · 1790105097 · focus · HN ↗
> It found the U.S. “failed in its obligation to do everything feasible to verify” that the school was a military objective and that the failure “went beyond mere negligence.” The report said the United States “directed the strikes at the building of the school while being aware of a substantial risk of striking a civilian object and acting recklessly as regards the possibility that this would happen.”
Reading the details, "AI" doesn't really seem like the culprit -it's a scapegoat.
The intelligence that it was no longer a military target never entered the target database, the team that was responsible for vetting the target list was gutted, and said team was never even consulted.
The White House wanted 1000 targets and pulled from their database without any due diligence. Whether it was an AI call or an SQL query - this was from pure human maliciousness and incompetence.
1. xlayn · 1790107269 · focus · HN ↗
  Now, ask yourself if you want AI taking decisions around your life... it will pull numbers... someone will have a "goal" of 1000 "hits" and there you go, 2-3 months in jail working to check what went wrong...
  But don't worry, the other P AI system will ensure that the first one doesn't do mistakes...
  1. toomuchtodo · 1790113544 · focus · HN ↗
    Lawsuit Says UnitedHealth Used AI With a 90% Error Rate to Deny Care - <a href="https://www.yahoo.com/news/us/articles/lawsuit-says-unitedhealth-used-ai-173017952.html" rel="nofollow">https://www.yahoo.com/news/us/articles/lawsuit-says-unitedhe... - September 18th, 2026
    UnitedHealth uses AI model with 90% error rate to deny care, lawsuit alleges - <a href="https://news.ycombinator.com/item?id=38299921">https://news.ycombinator.com/item?id=38299921 - November 2023 (58 comments)
    <a href="https://litigationtracker.law.georgetown.edu/wp-content/uploads/2023/11/Estate-of-Gene-B.-Lokken-et-al_20231114_COMPLAINT.pdf" rel="nofollow">https://litigationtracker.law.georgetown.edu/wp-content/uplo...
    teravor · 1790118858 · focus · HN ↗
    that smells like bullshit to be fair. if you have a system that makes errors over 90% of the time then you have a system you can leverage to make fewer errors.
    the situation would have to be very particular/contrived for that not to be the case.
    toomuchtodo · 1790119302 · focus · HN ↗
    <a href="https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_what_it_does" rel="nofollow">https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...
    1718627440 · 1790180894 · focus · HN ↗
    The word purpose exists to describe something that is independent of it's current behaviour. If you do not mean that, don't use that word.
    flossly · 1790119390 · focus · HN ↗
    What if you want errors to happen, e.g. deny care, bomb civilians; in that case 90% error rate is what you want, and plausible deniably so!
    ImPostingOnHN · 1790122630 · focus · HN ↗
    What makes you think they are financially motivated to make fewer "errors" (improper denial of care while keeping premiums)?
    degamad · 1790123581 · focus · HN ↗
    > the situation would have to be very particular
    The situation is very particular. When the system makes an error, I make a bigger profit. If the system made fewer errors, it would approve more things that cost me money.
    Why do you think I'm motivated to make fewer errors under that situation?
    dylan604 · 1790123908 · focus · HN ↗
    It sounds like something with a 90% error rate would be better to just do the opposite of what was determined and suddenly have a 10% error rate??
    eks391 · 1790130802 · focus · HN ↗
    That's not how type 1 & type 2 errors work. The article is describing type 2 (false negative) and inverting the output has no correlation with the percent of false positives (type 1)
    <a href="https://en.wikipedia.org/wiki/Type_I_and_type_II_errors" rel="nofollow">https://en.wikipedia.org/wiki/Type_I_and_type_II_errors
    benlivengood · 1790124735 · focus · HN ↗
    Nah, it's cumulative errors per claim. E.g. maybe the per-decision-factor rate is somewhere over 50% but overall most claims will have errors because so many factors go into coverage/reimbursement decisions.
    Dylan16807 · 1790126820 · focus · HN ↗
    > if you have a system that makes errors over 90% of the time then you have a system you can leverage to make fewer errors.
    Sure if you're flipping a coin.
    If you're calculating a number from 1 to 100 then depending on distribution a calculation that's wrong 90% of the time is probably useless.
    mschuster91 · 1790144985 · focus · HN ↗
    > if you have a system that makes errors over 90% of the time then you have a system you can leverage to make fewer errors.
    member the three words "delay, deny, depose"? Insurances - and not just in the US, Germany has similarly bad stuff going on - aren't making a profit if they just blindly accept claims. It is much more profitable to deny with a vague "AI" system (or a blanket automated deny) first, and only have a first look with an actual human agent at it when the customer complains or files a lawsuit.
    j4yav · 1790159055 · focus · HN ↗
    I can easily build you a system that will guess the price of a stock tomorrow wrong at least 90% of the time. Even 100% wrong if you'd like. How would you use that to make fewer errors in some other system?
    [deleted] · 1790178314 · focus · HN ↗
    [deleted]
    1718627440 · 1790180819 · focus · HN ↗
    If it is a binary choice you can just do the opposite.
    j4yav · 1790229897 · focus · HN ↗
    What’s binary choice we are discussing here and what’s the opposite that you’d do? Approve every claim that was denied by the AI, and vice versa? Or what’s the binary choice related to stock prices you’re referring to?
    1718627440 · 1790243400 · focus · HN ↗
    I agree, that most real world problems, are not binary choices. But in this case actually yes, just do the opposite.
    For that reason error rates above 50% don't make any sense. Building a system, that is reliably wrong, is as hard as building one that is reliable correct, because the systems are the same, just with the output inverted.
    The worst you can get with a system is 50%, at which point you can just flip a coin, because that is just pure random. Any deviation from that is going to be an improvement.
    [deleted] · 1790246747 · focus · HN ↗
    [deleted]
    j4yav · 1790271960 · focus · HN ↗
    I always thought insurance claims were more complex, some things may be wholly or partially covered on a line by line basis, and there should be justification provided. In certain cases they go to mediation or even court. It doesn’t seem binary to me but maybe it works differently where you are.
    1718627440 · 1790284735 · focus · HN ↗
    I said "if it is", not that it would be.
    j4yav · 1790310103 · focus · HN ↗
    Oh, haha, sure.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.