Thread

One Piece of Flock Camera Data Put This Innocent Woman in Jail for 13 Days

257 points · 145 comments · HotGarbage

freddealmeida · 1790398947 · focus · HN ↗
Flock did not put the woman in jail. The police and DA put her in Jail. This is not a failure of the technology (do they have confidence levels?). This is a failure of how police are outsourcing their reasoning and decision making to an AI camera service that seems to fail at the tail.
1. CrendKing · 1790401755 · focus · HN ↗
  But Flock gives police overconfidence to commit actions instead of having due diligence. It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him, even there is clear evidence against the claim.
  This is also comparable to self-driving car: Waymo's system has achieve the level of "less error than human on average" long time ago, but they are still only doing slow expansion. Why? Because when AI is involved, human naturally expect them to be almost 100% correct, not just barely better than human, otherwise why bother use AI (and all the troubles to legalize for AI) in the first place? So in this case, I guess the police just blindly trust the AI being 99.99% correct.
  1. vconnor · 1790402118 · focus · HN ↗
    > It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him, even there is clear evidence against the claim.
    Yes, but it is still your fault. “I am dumb and I was deceived” is no valid defense, least of all for a police officer.
    CrendKing · 1790403387 · focus · HN ↗
    I'm not saying the police has no fault. Using AI assist without secondary verification is never good idea, at least not in 2026. But you can't argue Flock the technology itself, and especially Flock the company, are totally innocent, just like you can't say the "guns are just tools like hammer, it only depends on who uses them", and the gun industry.
    [deleted] · 1790404591 · focus · HN ↗
    [deleted]
    phendrenad2 · 1790405945 · focus · HN ↗
    So next time start your first post with that, otherwise you lead us on this wild goose chase only to find that you mostly agree, it's a foundational part of discourse.
    mdp2021 · 1790409595 · focus · HN ↗
    > the police has no fault
    Gross lack of due diligence: it has full fault.
    Untrained officials: graver systemic fault.
    > company, are totally innocent
    How clear is the uncertainty of the output?
    krisoft · 1790421845 · focus · HN ↗
    > How clear is the uncertainty of the output?
    It is not even clear from the article that there were anything wrong with the output. She has a car of that make. And it doesnt appear to be in dispute that she drove nearby.
    It simply is the case that “being nearby” in itself is not enough evidence that you were involved in a crash. Especially when the colour doesn’t match and the vehicle is not damaged.
    LorenPechtel · 1790522334 · focus · HN ↗
    The fundamental problem here is that the police are very bad about false positives. Flock gives them a ton of data--but that doesn't make it magically perfect. No surprise there was a similar car nearby--there are a lot of similar cars. Flock shows them the similar cars and they pick the closest without any investigation.
    usefulcat · 1790473239 · focus · HN ↗
    > But you can't argue Flock the technology itself, and especially Flock the company, are totally innocent
    For the sake of argument, let’s suppose that Flock is 100% evil/guilty/blameworthy in this case. Even in that case, two things are still true:
    A) Flock has no authority to arrest anyone B) Flock has no authority over the police
    The buck has to stop somewhere. It’s counterproductive to focus on what Flock did, compared to what the police did. And I say that as someone who is not a fan of Flock.
  2. nrmitchi · 1790403041 · focus · HN ↗
    Flock isn’t an “AI” though? It doesn’t put together a case and say “yes, this is the criminal”. It’s not a crime-solving AI.
    It’s a camera system, and data can be queried by police. They have to search for something. So the cops searched for a vehicle matching the description in the general area, seemed to take the first one, and called it a day.
    That’s not an AI-uncanny-valley. That’s the equivalent of (pre-flock/cameras) looking at vehicle registration list and assuming the one at the top of the list must be it.
    The automation-trust is a legitimate concern but it’s not applicable here.
    CrendKing · 1790403904 · focus · HN ↗
    I don't know the details on how Flock is internally designed, but I suppose AI is at least involved in the image recognition and correlation, at least to this scale. I also don't know exactly why Flock misidentifies Issacs' car in this case, because I don't find it in the article or the interview, but I assume it is due to a mislabeling of the plate number or the car damage imagery, which is related to AI. Correct me if I'm wrong.
    SpicyLemonZest · 1790435986 · focus · HN ↗
    Flock didn't misidentify Isaacs' car in this case. It correctly identified that her car was vaguely near the accident, and the police incorrectly concluded that it was the one involved in the accident.
    givinguflac · 1790420335 · focus · HN ↗
    Ahem, reality would like a word about flock not being AI.
    <a href="https://cybernews.com/privacy/flock-ai-os-investigate/" rel="nofollow">https://cybernews.com/privacy/flock-ai-os-investigate/
    newtonsmethod · 1790421434 · focus · HN ↗
    Flock is not an AI. That they have AI features on certain products does not mean "Flock" is an AI. That's the fallacy of composition.
    Flock does have AI components (this one seems LLM based). Basically every ANPR system will also use some level of AI.
    Also, on "reality would like a word". Why not just speak normally instead of using what seems like a high-school grade comeback?
    The main Flock related issue here is that officers treat results that come out of the systems as gospel. The same thing has happened with face recognition systems. There is also the broader issue of access controls to Flock, 5A, and whether places should be building up a technology so powerful (surveillance infrastructure can easily be used for evil).
    givinguflac · 1790509546 · focus · HN ↗
    I said that because your position is absurd, and clearly false. Flock is an AI company, period. They use a multitude of different ai types in their “AI OS”.
    And the issue is absolutely not just cops taking what they say as gospel; that’s a teeny tiny itty bitty fraction of the issue, which is in itself reductive. This is a massively multifaceted issue and to state otherwise means you don’t understand the scope.
    Perhaps you should actually read TFA. Here’s another one. <a href="https://cybernews.com/privacy/flock-ai-os-investigate/" rel="nofollow">https://cybernews.com/privacy/flock-ai-os-investigate/
    newtonsmethod · 1790595061 · focus · HN ↗
    I have made no comments on this thread before the one above.
    So saying you said something because my non-existent position is "absurd" and "clearly false" indicates massive levels of confusion.
    You then go on to imply I haven't read the article, using an acronymised swear (I'm not a pious person but clearly directing a swear to someone makes a message come off as aggressive), even though your understanding of what I said is based off non-existent messages. (This is quite ironic).
    Flock is a company. Companies are a type of organisation: it makes no sense to describe a company as AI, as AI is a type of computer system. It may make sense to say Flock is an AI-based company, but calling them "AI" in itself makes no sense.
    I can't find anything indicating that an LLM was used in this case, and this county seems to mainly have Falcon cameras (<a href="https://www.scribd.com/document/983922612/250730110407Flock-Safety-INV-63909-2025-05-09" rel="nofollow">https://www.scribd.com/document/983922612/250730110407Flock-...). Reverse engineering seems to indicate these are powered by non-LLM AIs like Tensorflow vision NNs. These have been around for ages, and notably no-one is alleging these models hallucinated etc.
    People also (perhaps wrongly) use "Flock" to refer to the camera system. CrendKing explicitly calls "Flock" a "camera system". A system of cameras is not AI, and the point of the comment seems to be that it's inappropriate to bring up LLM related AI fears about this system.
    baranul · 1790423355 · focus · HN ↗
    Correct, but what's more, there is a lack of accountability issue that's part of it. When people are not held accountable for the screw ups, then processes don't or are very slow to change. That's how you can just call it a day, regardless of the harm done to innocent citizens.
  3. boxed · 1790411635 · focus · HN ↗
    Cops regularly frame people and get caught only because of cameras that the cops are too stupid/incompetent to disable or avoid. This goes both ways.
  4. testplzignore · 1790416284 · focus · HN ↗
    Software developers approving AI PRs without reading them: This is the way the world works now. Get with the times.
    Police approving AI PRs without reading them: This is the way the world works now. Get with the times.
  5. DANmode · 1790442381 · focus · HN ↗
    > It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him, even there is clear evidence against the claim.
    It’s more like normal people don’t know that 97.8% accurate isn’t 100% - despite how good of a score that would be on a test.
    Law enforcement is predominantly made up of normal people.
    You might see where this is going.
    AngryData · 1790453619 · focus · HN ↗
    I would say US law enforcement are made up of mostly abnormal people with lower intellegences, higher aggressivness, and a propensity to lie and deceive. They tend to push actual normal people out for not playing ball with their own corruption.
    The number one demograph for domestic violence and abuse are cops, they are not just a randomly selected group of people that chanced into law enforcement as a career.
    prh8 · 1790533361 · focus · HN ↗
    Absolutely, law enforcement in the US is disproportionately not normal people. The system is designed to reject smart people who are not aggressive and have empathy. The system is designed to accept people who will uphold the design of the system.
    DANmode · 1790540407 · focus · HN ↗
    > The number one demograph for domestic violence and abuse are cops, they are not just a randomly selected group of people that chanced into law enforcement as a career.
    Be sure you don’t have the causation backwards - or that it doesn’t flow strongly both ways!
    It attracts certain people - and being around that crowd and power pushes some number of people over the line that were near it.
    See also: stimulant use, abuse, in the construction industry.
  6. zugi · 1790446994 · focus · HN ↗
    > It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him
    Closer analogy: The dumbest student in a class tells you 2 + 2 = 5 and you believe them.
    Even more apt analogy: You're the dumbest student in the class, the second-dumbest student tells you 2 + 2 = 5, but you lack the critical thinking skills necessary to know whether that student is smart or not so you just believe them.
  7. jaredklewis · 1790489599 · focus · HN ↗
    I don't feel we are asking too much of police and lawyers here: yes, you can use technology and tools (such as Flock) to help you do your job more efficiently, but that doesn't mean you can turn your brain off.
    Perhaps more facts will come out, but if we take the article at face value, the cops are at this woman's house, looking at her car and can see with their own eyes that the car right in front of them has obviously not been in an accident. All that is required is for these cops to rub two brain cells together and they will reach the conclusion that Flock misread a license plate.
    It's also worth emphasizing that this wasn't for a parking ticket. The extremely low level of care with which these public servants are conducting a manslaughter investigation is somewhat alarming.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.