How facial analysis software that classifies gender misidentifies trans and non-binary people, is inaccurate with people of color, and could harm these groups
San Francisco (CNN)Artificial intelligence doesn't know what to make of Os Keyes. — The 29-year-old graduate student is dark-haired …
Context & Ripple Effects
This piece lands at the end of a two-year accumulation of evidence. It started with the 2018 benchmark showing 21%-35% error rates for dark-skinned women across Microsoft, IBM, and Megvii systems, then narrowed to Amazon, whose Rekognition was found to flag women as men 19% of the time — 31% for darker-skinned women — before mid-2019 investigations documented it misgendering trans, queer, and nonbinary users by design.
What the CNN reporting adds is the human layer on top of those benchmarks: Os Keyes and others show that binary gender classification structurally fails anyone outside its categories. That matters because the same year San Francisco became the first major U.S. city to ban law enforcement use of facial recognition — the accuracy debate and the policing debate are converging.
First-order effects
- Trans and non-binary people and people of color bear the direct cost today: they are the populations misclassified by tools sold by Amazon, Microsoft, IBM, and Megvii, and every downstream decision made on those classifications inherits the error.
- Amazon faces the sharpest exposure, since Rekognition now carries both documented racial error rates and by-design misgendering findings — a credibility problem in exactly the government-contract market it has been pursuing.
Second-order effects
- Vendors competing for police and agency contracts are pushed into an accuracy arms race, where a single published error-rate study can disqualify a bid — the pattern already visible when Rekognition's numbers were contrasted against IBM's and Microsoft's.
- Buyers gain leverage to demand audited performance across demographic groups before deployment, shifting procurement from marketing claims toward third-party benchmarks like the ones driving this coverage.
Third-order effects
- If the pattern holds, binary gender classification gets squeezed out of commercial products entirely — either withdrawn, rebranded, or restricted — while cities follow San Francisco's lead in legislating what vendors will not self-regulate.
- The longer-term structure points toward mandatory demographic-accuracy auditing as a condition of selling biometric analysis, turning civil-rights findings into compliance requirements rather than news-cycle embarrassments.
The trend: Commercial facial analysis is being forced from unchecked product feature to regulated infrastructure, with documented bias against marginalized groups acting as the trigger for both buyer pushback and municipal bans.