Study: facial recognition systems of Microsoft, IBM, and China's Megvii had 21%-35% error rate for dark-skinned women versus below 1% for light-skinned men
Facial recognition technology is improving by leaps and bounds. Some commercial software can now tell if a person in a photograph is male or female 99 percent of the time. Tweets: @fivefifths , @nytimestech , @dtunkelang , @tressiemcphd , @upulie , @nytimesbusiness , @lesliefeinzaig , and @ceciliakang Tweets: @fivefifths : Tbh I'm fine having a little more time being less recognizable to the surveillance state http://twitter.com/... NYTimes Tech / @nytimestech : In modern artificial intelligence, data rules. A.I. software is only as smart as the data used to train it. If there are many more white men than black women in the system, it will be worse at identifying the black women. http://www.nytimes.com/... Daniel Tunkelang / @dtunkelang : AI can only be as good — and as fair — as the data used to train it. Biased data produces biased results. http://www.nytimes.com/... Tressie Mc / @tressiemcphd : “We know.” -all black and brown people with an iPhone who have ever tried to wash their hands under a sensor-motion sink http://twitter.com/... Upulie Divisekera / @upulie : Shhhhh don't you get that it's a good thing http://twitter.com/... @nytimesbusiness : Only when she put on a white mask did the software recognize hers as a face. http://www.nytimes.com/... Leslie Feinzaig / @lesliefeinzaig : Well duh. Also: try speaking to Siri with an accent and see how wel she hears you. Facial Recognition Is Accurate, if You're a White Guy via @NYTimes http://www.nytimes.com/... @ceciliakang : Artificial Intelligence reflects its makers and the results are racial bias: http://www.nytimes.com/... via @stevelohr
Context & Ripple Effects
This study is the opening shot in a multi-year audit war over commercial facial recognition. It found that Microsoft, IBM, and China's Megvii all misidentified dark-skinned women at 21%-35% rates while staying below 1% error for light-skinned men — a gap the vendors' aggregate accuracy numbers had hidden.
The finding set the template for what followed: a year later Amazon's Rekognition showed even worse subgroup errors, and by late 2019 a federal-style benchmark of 189 algorithms confirmed racial bias across the industry. Vendors responded with the fixes Wired documented — diversified training data and per-group accuracy disclosure — but race-detection use cases kept expanding into market research, renewing researcher alarm.
First-order effects
- Microsoft, IBM, and Megvii face immediate credibility pressure on systems they sell as near-perfect, since their headline accuracy masked a 20x-plus error disparity for dark-skinned women.
- Any customer deploying these systems for identification — employers, retailers, security buyers — is now on notice that the technology's failure mode concentrates on a specific demographic.
Second-order effects
- Rivals get pulled into the same audit standard: once one vendor's subgroup errors are published, competitors' aggregate benchmarks stop being persuasive, forcing per-demographic testing industry-wide.
- Training-data practices become a competitive differentiator — vendors that diversify datasets and disclose group-level accuracy can market against those that don't, shifting procurement criteria toward audited performance.
Third-order effects
- If the pattern holds, commercial AI accuracy claims migrate from self-reported aggregates to independently verified subgroup metrics, making third-party audits a de facto requirement for selling biometric systems.
- Persistent demographic error gaps give regulators and litigants a concrete, measurable harm to target, pushing facial recognition toward either mandated accuracy floors or restricted deployment — the governance question the later race-detection coverage keeps reopening.
The trend: Commercial facial recognition is being forced from vendor-reported accuracy toward independently audited, per-demographic performance as the basis for trust and sales.