Thread

One Piece of Flock Camera Data Put This Innocent Woman in Jail for 13 Days

257 points · 145 comments · HotGarbage

freddealmeida · 1790398947 · focus · HN ↗
Flock did not put the woman in jail. The police and DA put her in Jail. This is not a failure of the technology (do they have confidence levels?). This is a failure of how police are outsourcing their reasoning and decision making to an AI camera service that seems to fail at the tail.
1. p-e-w · 1790400556 · focus · HN ↗
  Indeed. It’s insane to see these arguments implying the contrary. The institution committing the violence is the one responsible for the outcome.
  If a random person walks into a police station and accuses a random stranger of murder, and the police put that stranger in jail based on that alone, the main problem is not the accuser.
  1. jimdavenport · 1790401063 · focus · HN ↗
    But in this case, it wasn't a random person, it was a police-vetted system. Of course the authorities are to blame, but Flock hold a lot of responsibility, too.
    p-e-w · 1790410757 · focus · HN ↗
    It shouldn’t have been vetted then. It’s again the fault of authorities who approved that.
    A scientist who uses an LLM to write a paper remains responsible for its contents. And authorities that outsource police work remain responsible for what comes from that.
    franga2000 · 1790415236 · focus · HN ↗
    Sure it shouldn't have been approved, but it was approvied through misleading marketing and a huge and expensive lobbying effort. Flock spends a ton of money convincing cops and legislators that their product works well, is useful and safe. That was always a lie and they have to be held responsible for it.
  2. [deleted] · 1790401109 · focus · HN ↗
    [deleted]
2. CrendKing · 1790401755 · focus · HN ↗
  But Flock gives police overconfidence to commit actions instead of having due diligence. It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him, even there is clear evidence against the claim.
  This is also comparable to self-driving car: Waymo's system has achieve the level of "less error than human on average" long time ago, but they are still only doing slow expansion. Why? Because when AI is involved, human naturally expect them to be almost 100% correct, not just barely better than human, otherwise why bother use AI (and all the troubles to legalize for AI) in the first place? So in this case, I guess the police just blindly trust the AI being 99.99% correct.
  1. vconnor · 1790402118 · focus · HN ↗
    > It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him, even there is clear evidence against the claim.
    Yes, but it is still your fault. “I am dumb and I was deceived” is no valid defense, least of all for a police officer.
    CrendKing · 1790403387 · focus · HN ↗
    I'm not saying the police has no fault. Using AI assist without secondary verification is never good idea, at least not in 2026. But you can't argue Flock the technology itself, and especially Flock the company, are totally innocent, just like you can't say the "guns are just tools like hammer, it only depends on who uses them", and the gun industry.
    [deleted] · 1790404591 · focus · HN ↗
    [deleted]
    phendrenad2 · 1790405945 · focus · HN ↗
    So next time start your first post with that, otherwise you lead us on this wild goose chase only to find that you mostly agree, it's a foundational part of discourse.
    mdp2021 · 1790409595 · focus · HN ↗
    > the police has no fault
    Gross lack of due diligence: it has full fault.
    Untrained officials: graver systemic fault.
    > company, are totally innocent
    How clear is the uncertainty of the output?
    krisoft · 1790421845 · focus · HN ↗
    > How clear is the uncertainty of the output?
    It is not even clear from the article that there were anything wrong with the output. She has a car of that make. And it doesnt appear to be in dispute that she drove nearby.
    It simply is the case that “being nearby” in itself is not enough evidence that you were involved in a crash. Especially when the colour doesn’t match and the vehicle is not damaged.
    LorenPechtel · 1790522334 · focus · HN ↗
    The fundamental problem here is that the police are very bad about false positives. Flock gives them a ton of data--but that doesn't make it magically perfect. No surprise there was a similar car nearby--there are a lot of similar cars. Flock shows them the similar cars and they pick the closest without any investigation.
    usefulcat · 1790473239 · focus · HN ↗
    > But you can't argue Flock the technology itself, and especially Flock the company, are totally innocent
    For the sake of argument, let’s suppose that Flock is 100% evil/guilty/blameworthy in this case. Even in that case, two things are still true:
    A) Flock has no authority to arrest anyone B) Flock has no authority over the police
    The buck has to stop somewhere. It’s counterproductive to focus on what Flock did, compared to what the police did. And I say that as someone who is not a fan of Flock.
  2. nrmitchi · 1790403041 · focus · HN ↗
    Flock isn’t an “AI” though? It doesn’t put together a case and say “yes, this is the criminal”. It’s not a crime-solving AI.
    It’s a camera system, and data can be queried by police. They have to search for something. So the cops searched for a vehicle matching the description in the general area, seemed to take the first one, and called it a day.
    That’s not an AI-uncanny-valley. That’s the equivalent of (pre-flock/cameras) looking at vehicle registration list and assuming the one at the top of the list must be it.
    The automation-trust is a legitimate concern but it’s not applicable here.
    CrendKing · 1790403904 · focus · HN ↗
    I don't know the details on how Flock is internally designed, but I suppose AI is at least involved in the image recognition and correlation, at least to this scale. I also don't know exactly why Flock misidentifies Issacs' car in this case, because I don't find it in the article or the interview, but I assume it is due to a mislabeling of the plate number or the car damage imagery, which is related to AI. Correct me if I'm wrong.
    SpicyLemonZest · 1790435986 · focus · HN ↗
    Flock didn't misidentify Isaacs' car in this case. It correctly identified that her car was vaguely near the accident, and the police incorrectly concluded that it was the one involved in the accident.
    givinguflac · 1790420335 · focus · HN ↗
    Ahem, reality would like a word about flock not being AI.
    <a href="https://cybernews.com/privacy/flock-ai-os-investigate/" rel="nofollow">https://cybernews.com/privacy/flock-ai-os-investigate/
    newtonsmethod · 1790421434 · focus · HN ↗
    Flock is not an AI. That they have AI features on certain products does not mean "Flock" is an AI. That's the fallacy of composition.
    Flock does have AI components (this one seems LLM based). Basically every ANPR system will also use some level of AI.
    Also, on "reality would like a word". Why not just speak normally instead of using what seems like a high-school grade comeback?
    The main Flock related issue here is that officers treat results that come out of the systems as gospel. The same thing has happened with face recognition systems. There is also the broader issue of access controls to Flock, 5A, and whether places should be building up a technology so powerful (surveillance infrastructure can easily be used for evil).
    givinguflac · 1790509546 · focus · HN ↗
    I said that because your position is absurd, and clearly false. Flock is an AI company, period. They use a multitude of different ai types in their “AI OS”.
    And the issue is absolutely not just cops taking what they say as gospel; that’s a teeny tiny itty bitty fraction of the issue, which is in itself reductive. This is a massively multifaceted issue and to state otherwise means you don’t understand the scope.
    Perhaps you should actually read TFA. Here’s another one. <a href="https://cybernews.com/privacy/flock-ai-os-investigate/" rel="nofollow">https://cybernews.com/privacy/flock-ai-os-investigate/
    newtonsmethod · 1790595061 · focus · HN ↗
    I have made no comments on this thread before the one above.
    So saying you said something because my non-existent position is "absurd" and "clearly false" indicates massive levels of confusion.
    You then go on to imply I haven't read the article, using an acronymised swear (I'm not a pious person but clearly directing a swear to someone makes a message come off as aggressive), even though your understanding of what I said is based off non-existent messages. (This is quite ironic).
    Flock is a company. Companies are a type of organisation: it makes no sense to describe a company as AI, as AI is a type of computer system. It may make sense to say Flock is an AI-based company, but calling them "AI" in itself makes no sense.
    I can't find anything indicating that an LLM was used in this case, and this county seems to mainly have Falcon cameras (<a href="https://www.scribd.com/document/983922612/250730110407Flock-Safety-INV-63909-2025-05-09" rel="nofollow">https://www.scribd.com/document/983922612/250730110407Flock-...). Reverse engineering seems to indicate these are powered by non-LLM AIs like Tensorflow vision NNs. These have been around for ages, and notably no-one is alleging these models hallucinated etc.
    People also (perhaps wrongly) use "Flock" to refer to the camera system. CrendKing explicitly calls "Flock" a "camera system". A system of cameras is not AI, and the point of the comment seems to be that it's inappropriate to bring up LLM related AI fears about this system.
    baranul · 1790423355 · focus · HN ↗
    Correct, but what's more, there is a lack of accountability issue that's part of it. When people are not held accountable for the screw ups, then processes don't or are very slow to change. That's how you can just call it a day, regardless of the harm done to innocent citizens.
  3. boxed · 1790411635 · focus · HN ↗
    Cops regularly frame people and get caught only because of cameras that the cops are too stupid/incompetent to disable or avoid. This goes both ways.
  4. testplzignore · 1790416284 · focus · HN ↗
    Software developers approving AI PRs without reading them: This is the way the world works now. Get with the times.
    Police approving AI PRs without reading them: This is the way the world works now. Get with the times.
  5. DANmode · 1790442381 · focus · HN ↗
    > It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him, even there is clear evidence against the claim.
    It’s more like normal people don’t know that 97.8% accurate isn’t 100% - despite how good of a score that would be on a test.
    Law enforcement is predominantly made up of normal people.
    You might see where this is going.
    AngryData · 1790453619 · focus · HN ↗
    I would say US law enforcement are made up of mostly abnormal people with lower intellegences, higher aggressivness, and a propensity to lie and deceive. They tend to push actual normal people out for not playing ball with their own corruption.
    The number one demograph for domestic violence and abuse are cops, they are not just a randomly selected group of people that chanced into law enforcement as a career.
    prh8 · 1790533361 · focus · HN ↗
    Absolutely, law enforcement in the US is disproportionately not normal people. The system is designed to reject smart people who are not aggressive and have empathy. The system is designed to accept people who will uphold the design of the system.
    DANmode · 1790540407 · focus · HN ↗
    > The number one demograph for domestic violence and abuse are cops, they are not just a randomly selected group of people that chanced into law enforcement as a career.
    Be sure you don’t have the causation backwards - or that it doesn’t flow strongly both ways!
    It attracts certain people - and being around that crowd and power pushes some number of people over the line that were near it.
    See also: stimulant use, abuse, in the construction industry.
  6. zugi · 1790446994 · focus · HN ↗
    > It is like the best student in your classroom tells you 2 + 2 = 5, and you naturally feel compelled to believe him
    Closer analogy: The dumbest student in a class tells you 2 + 2 = 5 and you believe them.
    Even more apt analogy: You're the dumbest student in the class, the second-dumbest student tells you 2 + 2 = 5, but you lack the critical thinking skills necessary to know whether that student is smart or not so you just believe them.
  7. jaredklewis · 1790489599 · focus · HN ↗
    I don't feel we are asking too much of police and lawyers here: yes, you can use technology and tools (such as Flock) to help you do your job more efficiently, but that doesn't mean you can turn your brain off.
    Perhaps more facts will come out, but if we take the article at face value, the cops are at this woman's house, looking at her car and can see with their own eyes that the car right in front of them has obviously not been in an accident. All that is required is for these cops to rub two brain cells together and they will reach the conclusion that Flock misread a license plate.
    It's also worth emphasizing that this wasn't for a parking ticket. The extremely low level of care with which these public servants are conducting a manslaughter investigation is somewhat alarming.
3. fellowniusmonk · 1790402246 · focus · HN ↗
  It is always both. A product that can be abused to this extent is a bad product.
  I have never released a single product that has seen people die through common user misuse.
  The standards for solutions shouldn't get _lower_ as the stakes get _higher_.
  If a private company can puts hundreds of tons of payload into orbit without ever causing someone to accidentally be imprisoned a company that sells cameras on a stick shouldn't either.
  1. [deleted] · 1790404143 · focus · HN ↗
    [deleted]
4. [deleted] · 1790404161 · focus · HN ↗
  [deleted]
5. latexr · 1790408242 · focus · HN ↗
  While you are technically correct (which we all know from Futurama is “the best kind”), that is neither helpful nor consolation to the affected, and I wish these kinds of comments didn’t dominate the top spot in HN discussions. They’re the “guns don’t kill people, people kill people” of tech.
  Yes, the system is broken even before we get to Flock (or whatever tech) but Flock makes it worse. If a bad tool is badly used to cause harm, it’s fair to criticise its existence. You can’t fix police abuse with a snap of the finger, so you need to start but taking away the tools which facilitate it.
  If it were your ass which had been thrown in solitary, would you be pleading with your lawyer to “go easy on Flock. Sure, they allowed a major disruption and trauma into my life, but they’re just a tool”?
  1. ArchD · 1790415104 · focus · HN ↗
    Then the right fix is to fix the system, e.g. imposing accountability. Removing tools is just a band aid. Why is the tool bad? A tool does what it's designed to do. You can use it properly or improperly. There should be no conflation of regular civilians with supposedly trained government officers.
    latexr · 1790416748 · focus · HN ↗
    Yes, of course the better fix is to fix the system but one thing at a time. You’re on a giant boat that has left port years ago and a leak just sprung; you should absolutely get back to port and fix the processes that allowed the leak to open but first you have to close it.
    If someone released a hammer that when held a certain way would cause your hand to explode and it kept happening over and over, the solution wouldn’t be just to tell people to hold it better, it would also be to recall the tool and fix it so the issue were less likely or impossible to occur. And if it can’t be done, then maybe the hammer shouldn’t be released again.
    This “it’s just a tool, the proper solution is to rework civilisation” defence is frankly as tired as it is ignorant. The way you fix the system as a whole is via a lot of small fixes.
    ArchD · 1790423517 · focus · HN ↗
    Then it seems that you are somehow implying that it takes many years to merely enact and impose real accountability rules. I think it's not a tech problem, or scientific problem, with unknown solution. It's a social/political problem.
    I don't it's about an innocent skill issue like you imply (holding a hammer the wrong way). It's an ethical issue of bad actors in the government, together with an accountability issue of the system preventing holding those bad actors to account.
    AI cameras are just one tool; there are other tools, too. Where do you draw the line for which tools deserve the removal for your kind of reason? Perhaps degun the police departments because guns also get abused? How do you decide in a principled way that AI cameras are somehow special compared to other tools?
    latexr · 1790445823 · focus · HN ↗
    > Then it seems that you are somehow implying that it takes many years to merely enact and impose real accountability rules.
    It doesn’t have to, but historically we can observe that the longer something has been entrenched as the norm, the harder it is to change (authoritarians regimes take advantage of this too, by making it appear some things are older than they are). Which is why it’s doubly important to resist when something new with bad consequences is introduced.
    > I don't it's about an innocent skill issue like you imply
    There was no “skill issue” implication. If anything, the opposite is the point.
    > AI cameras are just one tool; there are other tools, too.
    This is just whataboutism. Yes, there are several tools which are problematic. The submission is about this one, which is why we’re discussing it. We can’t discuss everything that needs to change in every conversation. That’s called focus.
    AngryData · 1790454131 · focus · HN ↗
    They can have their tools back after they show they can use them responcibly.
    If a kid has shown themselves to be repeatedly irresponsible, when they ask for a circular saw you don't just give it to them and turn away. You make them explain what it is for and ensure they know how to safely use it. And if they start swinging it around wildly in opposition to what you said, you take it away, they are obviously incapable of using it right.
  2. Buttons840 · 1790427096 · focus · HN ↗
    The people who say there are problems with Flock are perceived as excusing the human police officers.
    The people who say there are problems with the human police officers are perceived as excusing Flock.
    Who is responsible for the area inside a square, the width or the height? It is not an additive relationship.
6. neya · 1790409295 · focus · HN ↗
  Which makes Flock cameras even worse right? It provides false confidence to law enforcement to make decisions that can change someone's life for worse. I don't understand why people hate the NSA spying on them and yet support Flock cameras.
  1. SpicyLemonZest · 1790436979 · focus · HN ↗
    The question is whether Flock does provide false confidence to law enforcement, or whether it provides a media hook to cover the ways in which bad police departments routinely misbehave.
    A lot of people coming to this story seem to have the idea that it's unthinkable for someone to spend 13 days in jail when they have clear evidence they're innocent, so there must be some novel explanation for how this could happen. But as any criminal defense lawyer will tell you it's very thinkable. She was ordered to report to jail on a Friday afternoon, the standard stratagem cops use to ensure people they don't like spend at least a weekend behind bars. She was denied bail, even though she made no move to flee in the months where she knew the police thought she did it, because the courts in many places view mandatory pretrial detention as a punishment for presumptively guilty people.
7. rendaw · 1790409390 · focus · HN ↗
  If you give a toddler a gun and it shoots someone, it's the toddler's fault, not the gun's right? But you don't give toddlers guns. You don't say "well toddlers just need to be more careful".
  1. ArchD · 1790414923 · focus · HN ↗
    Are toddlers being hired to do government work nowadays, including as police officers?
    ninalanyon · 1790419152 · focus · HN ↗
    As far as reasoning abilities goes it would seem so.
    DANmode · 1790442427 · focus · HN ↗
    Worse, fully grown incompetent and or malicious people!
    joquarky · 1790480433 · focus · HN ↗
    Depends on if your use of "toddlers" includes metaphors.
8. atoav · 1790410825 · focus · HN ↗
  The technology is not a neutral thing here. Automation bias¹ exists and is especially problematic in this context.
  And if your country happens to have a police force that is historically bad at its job, giving them that power seems like a really stupid idea.
  ¹: <a href="https://en.wikipedia.org/wiki/Automation_bias" rel="nofollow">https://en.wikipedia.org/wiki/Automation_bias
  1. newtonsmethod · 1790421623 · focus · HN ↗
    I very much agree with what you say on automation bias.
    Certain technology just has too much potential to be misused / people repeatedly prove they can't use it properly. It also comes with risks of making authoritarian takeover easier.
    However, here, it seems they used Flock a bit like a regular ANPR. The actions they took independent of Flock show there are better stories to use to criticise this technology.
9. poulpy123 · 1790413650 · focus · HN ↗
  The police is of course responsible, but you can't wave the responsibility of flock away when it's inconvenient for them
  1. newtonsmethod · 1790421499 · focus · HN ↗
    How is Flock responsible in this case?? There may be cases where Flock is responsible, but I don't see it here.
    johannesrexx · 1790555017 · focus · HN ↗
    Flock made the matches (Flock mass surveillance system) and let children (the police) play with them. The result is that citizens get burned.
    newtonsmethod · 1790593845 · focus · HN ↗
    I don't think ANPR systems should be held responsible for matching number-plates or car models. That's their job, and police have had access to such tools for many many years. It seems relatively clear that Flock was providing a commonplace technology here, that was misused.
    LorenPechtel · 1790693989 · focus · HN ↗
    If the police are children then that's the real problem.
10. gentlerain · 1790414092 · focus · HN ↗
  True, this isn't about flock and it's not even about outsourcing reasoning.
  This is the good old police incompetence that you come across once in a while and shake your head.
  Where detectives lock on one angle of a case and completely refuse to entertain any other possible explanations
11. wolfi1 · 1790424397 · focus · HN ↗
  of course it's the absence of common sense in the law enforcement but on the other hand this technology is thought to be faultless and it takes resource to put it right
12. fluidcruft · 1790438013 · focus · HN ↗
  Would depend entirely how Flock is marketed and represented to the people who buy them and whatever training Flock provides.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.