Thread

Pentagon says overreliance on AI contributed to missile strike on Iran school

974 points · 553 comments · devonnull

legitster · 1790105097 · focus · HN ↗
> It found the U.S. “failed in its obligation to do everything feasible to verify” that the school was a military objective and that the failure “went beyond mere negligence.” The report said the United States “directed the strikes at the building of the school while being aware of a substantial risk of striking a civilian object and acting recklessly as regards the possibility that this would happen.”
Reading the details, "AI" doesn't really seem like the culprit -it's a scapegoat.
The intelligence that it was no longer a military target never entered the target database, the team that was responsible for vetting the target list was gutted, and said team was never even consulted.
The White House wanted 1000 targets and pulled from their database without any due diligence. Whether it was an AI call or an SQL query - this was from pure human maliciousness and incompetence.
1. oooyay · 1790118949 · focus · HN ↗
  I'm not sure if it was any kind of official rule, but I worked at times in a JTAC capacity for troops in contact. I mention the last bit because it's very high stakes where seconds and minutes count a great deal more than when you do coordinated tactical strikes against strategic objectives like weapons caches. When I did that kind of work we had to have three eyes-on forms of contact. Often that was the calling troops, the observer (like an Air officer), and an air asset like a drone. They all had to independently describe the target, its orientation, and surrounding activity.
  However you sourced your target is irrelevant to the activity at hand. When I first heard of this news I knew immediately someone skipped target verification or that it was simply no longer a policy of the DOD.
  1. somenameforme · 1790131005 · focus · HN ↗
    Famously, the last 'enemy' we killed in Afghanistan, alongside the 7 children and others surrounding him, was somebody who was working for a US aid organization. [1] A drone operator saw him putting bottles of water into his trunk, and somehow thought they were bombs, so they tracked him for 8 hours, then killed him and his family.
    [1] - <a href="https://www.bbc.com/news/world-us-canada-58604655" rel="nofollow">https://www.bbc.com/news/world-us-canada-58604655
    stult · 1790141919 · focus · HN ↗
    This happened after I stopped working for the DOD, but I am 99% sure the program in question during this incident was something I worked on. It was effectively an anomaly detection system for identifying any weird behaviors detectable in WAMI (wide area motion imagery, which is high resolution, low framerate drone video data that covers a large geographic area). As an example from the unclassified sample project they asked us to do as part of the bidding process, our code flagged a car doing donuts in a parking lot (the sample data was just from a random US city on a random day because this was an unclassified open bidding process). The system was not and never was intended to be a "terrorist" detection system. The idea was always to flag potentially interesting events for a human analyst to follow up on. The disconnect it took for someone to blindly trust the program as a target acquisition system boggles my mind. Someone doing donuts in a parking lot should not be automatically targeted with hellfire missiles, yet in practice that is how they were using the tool. I still lose sleep over it.
    4gotunameagain · 1790144961 · focus · HN ↗
    I'm sorry for your pain.
    This is a good lesson for young people reading this. If you work for the US military, you are not working for Defence.
    Your work will be used against innocent civilians in far away lands.
    dmos62 · 1790168095 · focus · HN ↗
    The answer is not to avoid engaging with a corrupt institution. US is a democracy: vote, be active politically, go into organizations that need change.
    4gotunameagain · 1790171088 · focus · HN ↗
    So you are suggesting that you can change the US military industrial complex by.. working for the NSA or Lockheed Martin ?
    Also it is quite questionable how much of a democracy the US is nowadays.
    Gerrymandering aside, the populace does not get to choose the presidential candidates. And the whole two party thing ? Or the fact that to have a chance you need a super PAC with hundreds of millions ?
    This seems like elitocracy..
    dmos62 · 1790172498 · focus · HN ↗
    Yes, that's what I'm suggesting. If you feel that the problem is higher up or lower down in the institutional hierarchy, then go higher up or lower down. That's the civil approach, as I understand it.
    4gotunameagain · 1790172682 · focus · HN ↗
    I think this is utterly infeasible.
    Apart from the fact the if you stay true to your morals you will be incapable of rising to a position of power in such corrupt institutions, the degree of institutionalisation necessary to do so will definitely erode your ethics, turning you into another cog.
    dmos62 · 1790176961 · focus · HN ↗
    In my opinion, you underestimate the power of an individual. I'm not saying you have to be Gandhi, but he was one person. A problem with your assessment, is that if you're right, or, if everyone believes you're right, you'll have to wait for some external force to come in and change things, like a calamity or revolt.
    MaxfordAndSons · 1790181355 · focus · HN ↗
    I agree in principle, but also agree with gp that in practice the things one has to do to rise to a position of power in corrupt institutions make it virtually impossible to be a credible agent of change, especially private/secretive institutions like military, intelligence, military-industrial corps. I guess politicians are the one role that is supposed to be able to operate without those institutional constraints but of course, (in the US) given corporate capture of political speech, political hyper-polarization, etc, in most cases they are just as or more constrained. It is a hard problem.
    somenameforme · 1790213969 · focus · HN ↗
    The Brits were bankrupting themselves with war spending, which culminated in WW2. After its conclusion, they started liberating every territory that wasn't directly profitable, including some that did not even want to be liberated. WW2 ended in 1945, India was liberated in 1947 alongside numerous other territories. Without WW2, India would likely have remained a British territory well past 1947. Beyond that, Gandhi wasn't one man - but the leader of a movement of millions.
    As for Russia, I think your perception there is also driven by 'messaging'. There's a million videos of people streaming walking tours and the like in Russia. Here [1] is one done 4 days ago. That's a country that in the 90s was on the verge of collapse and was extremely dangerous. You can find similar stuff on e.g. Twitch because there's this somewhat odd subculture of delivery drivers, taxi drivers, and others streaming their days. Perhaps in part out of bemusement from this Western perspective paired against reality.
    [1] - <a href="https://www.youtube.com/watch?v=89ckPUyGVA8" rel="nofollow">https://www.youtube.com/watch?v=89ckPUyGVA8

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.