Thread

1. legitster · 1790105097 · focus · HN ↗

   > It found the U.S. “failed in its obligation to do everything feasible to verify” that the school was a military objective and that the failure “went beyond mere negligence.” The report said the United States “directed the strikes at the building of the school while being aware of a substantial risk of striking a civilian object and acting recklessly as regards the possibility that this would happen.”

   Reading the details, "AI" doesn't really seem like the culprit -it's a scapegoat.

   The intelligence that it was no longer a military target never entered the target database, the team that was responsible for vetting the target list was gutted, and said team was never even consulted.

   The White House wanted 1000 targets and pulled from their database without any due diligence. Whether it was an AI call or an SQL query - this was from pure human maliciousness and incompetence.

   1. oooyay · 1790118949 · focus · HN ↗

      I'm not sure if it was any kind of official rule, but I worked at times in a JTAC capacity for troops in contact. I mention the last bit because it's very high stakes where seconds and minutes count a great deal more than when you do coordinated tactical strikes against strategic objectives like weapons caches. When I did that kind of work we had to have three eyes-on forms of contact. Often that was the calling troops, the observer (like an Air officer), and an air asset like a drone. They all had to independently describe the target, its orientation, and surrounding activity.

      However you sourced your target is irrelevant to the activity at hand. When I first heard of this news I knew immediately someone skipped target verification or that it was simply no longer a policy of the DOD.

      1. somenameforme · 1790131005 · focus · HN ↗

         Famously, the last 'enemy' we killed in Afghanistan, alongside the 7 children and others surrounding him, was somebody who was working for a US aid organization. [1] A drone operator saw him putting bottles of water into his trunk, and somehow thought they were bombs, so they tracked him for 8 hours, then killed him and his family.

         [1] - <a href="https:&#x2F;&#x2F;www.bbc.com&#x2F;news&#x2F;world-us-canada-58604655" rel="nofollow">https:&#x2F;&#x2F;www.bbc.com&#x2F;news&#x2F;world-us-canada-58604655

         1. stult · 1790141919 · focus · HN ↗

            This happened after I stopped working for the DOD, but I am 99% sure the program in question during this incident was something I worked on. It was effectively an anomaly detection system for identifying any weird behaviors detectable in WAMI (wide area motion imagery, which is high resolution, low framerate drone video data that covers a large geographic area). As an example from the unclassified sample project they asked us to do as part of the bidding process, our code flagged a car doing donuts in a parking lot (the sample data was just from a random US city on a random day because this was an unclassified open bidding process). The system was not and never was intended to be a &quot;terrorist&quot; detection system. The idea was always to flag potentially interesting events for a human analyst to follow up on. The disconnect it took for someone to blindly trust the program as a target acquisition system boggles my mind. Someone doing donuts in a parking lot should not be automatically targeted with hellfire missiles, yet in practice that is how they were using the tool. I still lose sleep over it.

            1. lokar · 1790176172 · focus · HN ↗

               Whenever you introduce (or grow) the presence of computer automation you need to consider the human &#x2F; social interactions. You are responsible for the whole system, not just your software.

               It is well understood that human diligence falls dramatically in the face of automation that is perceived to be highly reliable.

               There is a tendency for software people to say “it has a 5% error rate, so make sure to carefully check every time”. Even though we know that humans simply won’t check carefully the results of a 95% accurate system.

               1. 1718627440 · 1790180123 · focus · HN ↗

                  &gt; There is a tendency for software people to say “it has a 5% error rate, so make sure to carefully check every time”. Even though we know that humans simply won’t check carefully the results of a 95% accurate system.

                  This should really be framed differently during training. &quot;Assume you have 100 neighbors in your area&#x2F;house. We expect you to kill 5 innocent neighbors each time you use the system, and don&#x27;t actually think for yourself. (After using it once every day for two weeks, you will have killed half of your neighbors.)&quot;

                  Suddenly 5% doesn&#x27;t sound that small&#x2F;95% not as good to the laymen.

                  1. lokar · 1790182594 · focus · HN ↗

                     IMO you should increase the error rate synthetically, adding tests. So you get 10 targets to review, but 4 of them are known to be bad and the system is both checking the user, and also increasing the perceived error rate to encourage diligence.
                  2. naijaboiler · 1790197127 · focus · HN ↗

                     Brilliant! This is the logic behind the birthday problem where you only need 45 people present to have greater than 90% chance that 2 people share a birthday.

                     Even rare events become very likely when you repeat it. A 5% wrong chance means half are dead after 10 tries

                     1. 1718627440 · 1790243041 · focus · HN ↗

                        &gt; Even rare events become very likely when you repeat it.

                        True, but in my opinion 5 innocent dead out of hundred is not remotely small. For any serious automated decision anything larger than per million is completely unacceptable. (For reference, among Germans the amount of Jews killed by Nationalsocialists, is &quot;only&quot; 2.3‰ [160k vs. 69M] I don&#x27;t think anyone considers that to be small.)

                        1. tosapple · 1790314934 · focus · HN ↗

                           nationalsociopaths
                  3. 1718627440 · 1790243803 · focus · HN ↗

                     Btw., after using it for a year every day, the US population would be 2.52 people. And I think the US military plans to use their expensive systems for much more than a year.

Unofficial Hacker News client; not affiliated with Y Combinator.