A look at wrongful US arrests due to false positive facial recognition matches, and how “automation bias” can lead the police to ignore contradictory evidence
This report sits within a documented pattern of wrongful arrests tied to facial-recognition matches, including a Georgia case involving a bad match and a lawsuit over a New Jersey man’s detention after an erroneous match.
The central issue is not solely model accuracy: investigators can give an automated lead undue weight and discount exculpatory facts. Later coverage of departments making arrests with no other evidence suggests that safeguard failures can persist beyond isolated cases.
First-order effects
People identified by a false match can face arrest and detention even where contradictory evidence exists, while police investigations may narrow prematurely around the algorithm’s lead.
Police departments using facial recognition face greater pressure to treat a match as an investigative lead rather than proof and to document corroborating evidence before seeking an arrest.
Second-order effects
Defense lawyers can more directly challenge the reliability of an identification and the investigation’s disclosure of how it was generated, particularly where facial recognition was not disclosed to defendants as later reporting found in widespread police use of the software.
Vendors and police technology buyers face scrutiny not just over error rates, but over the human review, training, audit trails, and evidentiary procedures surrounding deployment.
Third-order effects
If agencies continue to operationalize facial recognition without independent corroboration, public-safety AI governance will increasingly turn on rules for human accountability and due process, not only technical benchmark accuracy.
A sustained record of wrongful arrests could push the market toward stricter limits on using automated identification in arrest decisions; the corpus does not establish which policy response will prevail.
The trend: This is part of a broader shift from evaluating public-safety AI by claimed investigative efficiency to evaluating it by the safeguards that prevent automated leads from becoming unreviewed coercive action.
Proponents view facial-recognition technology as an invaluable tool that can help make policing more efficient. But what happens when law enforcement puts too much faith in fallible A.I.? @EyalPress investigates. https://nyer.cm/kepPEkg
In 2019, the National Institute of Standards and Technology, published a study revealing that many facial-recognition systems falsely identified Black and Asian faces between ten and a hundred times more frequently than Caucasian ones. | The New Yorker https://www.newyorker.com/.…
The @NewYorker looks at cases where facial recognition technology has led police to arrest the wrong person and asks whether AI leads officers to ignore contradictory evidence. (It cites my book, because, of course.) https://www.newyorker.com/...
Too often, a facial-recognition search represents virtually the entirety of a police investigation. What happens when the technology makes a false identification? @EyalPress spends time with a man implicated by an algorithm. https://nyer.cm/gdLM5l5
In @NewYorker's special issue on AI, @EyalPress looks at what happens when law enforcement presumes that a deeply imperfect technology like facial recognition is infallible, and why people tend to trust a system even when they don't fully understand it. https://www.newyorker.com/…
Will the growing use of facial recognition technology in criminal investigations lead to more wrongful arrests? My article in the AI issue of @NewYorker. https://www.newyorker.com/...
“There's a tendency to place trust in a system that we don't fully understand.” Clare Garvie of @NACDL on police use of facial recognition. Article by @EyalPress chronicles the string of false arrests linked to bad facial recognition matches. https://www.newyorker.com/...
.@EyalPress reports on the damage that can be done when law enforcement puts too much faith in fallible facial-recognition searches. https://nyer.cm/V2daSe9