Thread

Pentagon says overreliance on AI contributed to missile strike on Iran school

974 points · 553 comments · devonnull

legitster · 1790105097 · focus · HN ↗
> It found the U.S. “failed in its obligation to do everything feasible to verify” that the school was a military objective and that the failure “went beyond mere negligence.” The report said the United States “directed the strikes at the building of the school while being aware of a substantial risk of striking a civilian object and acting recklessly as regards the possibility that this would happen.”
Reading the details, "AI" doesn't really seem like the culprit -it's a scapegoat.
The intelligence that it was no longer a military target never entered the target database, the team that was responsible for vetting the target list was gutted, and said team was never even consulted.
The White House wanted 1000 targets and pulled from their database without any due diligence. Whether it was an AI call or an SQL query - this was from pure human maliciousness and incompetence.
1. seizethecheese · 1790109013 · focus · HN ↗
  Important context is that the notes about the changing use of the location weren’t even in a connected system. Unclear if the human analysts would have even caught this, or if they would have known to check this unconnected system.
  > Some US intelligence analysts took notice, the officials said. Bloomberg reported in June that one analyst detected changes to the site as early as 2019, and logged remarks about the modifications in a system that wasn’t connected to the primary military intelligence database, which informs targeting
  1. pbhjpbhj · 1790115960 · focus · HN ↗
    >"Unclear if the human analysts would have even caught this" (seizethecheese)
    At the time a news outlet put out commercially purchased images of the target site before and after -- even a layman could see that it had a children's playground next to the building, and that it was separated from the compound.
    <a href="https://www.channel4.com/news/factcheck/factcheck-iran-school-satellite-imagery-shows-evidence-of-targeted-strike-says-expert" rel="nofollow">https://www.channel4.com/news/factcheck/factcheck-iran-schoo... has the info from around the time, but doesn't show the detailed images in the news report (video) I seem to remember seeing.
    I think it was maybe some earlier coverage from NYT, they published this retrospectively - <a href="https://www.nytimes.com/video/world/middleeast/100000010750321/how-we-analyzed-the-strike-on-the-iranian-school.html" rel="nofollow">https://www.nytimes.com/video/world/middleeast/1000000107503... (which is also on YouTube, <a href="https://www.youtube.com/watch?v=hDcim48W4Io" rel="nofollow">https://www.youtube.com/watch?v=hDcim48W4Io). It shows that even a cursory analysis - without the benefit of enhanced military grade satellite/flyover imagery - indicates the building targeted with pin-point accuracy is a school (eg <a href="https://youtu.be/hDcim48W4Io?t=98" rel="nofollow">https://youtu.be/hDcim48W4Io?t=98).
    In short, this seems entirely avoidable. I don't think you need any connected system, you just need an image of the target and eyes.
    If they trained a model to find targets and didn't train it to avoid civilian targets, and couldn't be bothered to have anyone check them, that's still a war crime. The analyst could probably see the kids playing there with the imaging tech they have available.
    This war was unlawfully started by the military at Trump's command, knowing it was not a lawful order; just as they knew firing on shipwrecked civilian sailors was murder. Seemingly it was started to win money betting on fuel prices.
    spwa4 · 1790117294 · focus · HN ↗
    The discussion is mostly about AI being evil. Humans, even the very large amounts of humans the US military employees, are fundamentally limited in capacity. So the odds of them seeing this sort of image, and canceling the targeting information are approaching a constant (and that constant is not 0)
    Now if you ACTUALLY used AI, as opposed to what happened here (Trump stooge getting request for 1000 targets and doing SELECT * FROM targets LIMIT 1000, except getting AI to write the query) could could drive the odds of this happening much closer to zero.
    seizethecheese · 1790121004 · focus · HN ↗
    My point is that with the context available to Claude, it’s unclear what human would have done. Both human and Claude should have had more context.
    calgoo · 1790152538 · focus · HN ↗
    The whole point of using the AI systems it to review vast amount of data. They would 100% have access to updated pictures if we as civs can pull that data from public sources. Its not like the school was new, it had existed for over 10 years!!
    [1]<a href="https://www.bbc.com/news/articles/c5yqqyly9n0o" rel="nofollow">https://www.bbc.com/news/articles/c5yqqyly9n0o

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.