Healthcare AI has the same racial and gender biases as the data used to train it, which must be addressed if AI systems are to help make real medical decisions
Dave Gershgorn / Quartz : Tweets: @joyclee , @datasociety , @dmonett , and @davegershgorn Tweets: Doctor as Designer / @joyclee : “When you actually talk to real doctors and patients, suddenly the things that weren't apparent to computer scientists working in a basement with data become more evident” http://qz.com/... Data & Society / @datasociety : “'Bias at any point in data handling for precision medicine can lead to the recapitulation of longstanding health disparities,' wrote cultural anthropologist @KadijaFerryman and social psychologist @MikaelaPitcan.” http://qz.com/... Dagmar Monett / @dmonett : “Unfortunately, the medical datasets openly available for use by AI researchers are notoriously biased, especially in the US. It's not a secret: Health-care data is extremely male and extremely white, and that has real-world impacts.” @davegershgorn #AI #bias #data http://twitter.com/... Dave Gershgorn / @davegershgorn : Any discussion about bias in AI will be confusing, difficult, and uncomfortable, because bias is hidden and tricky until it's obvious and dangerous.My story about bias in healthcare data:http://qz.com/...
Context & Ripple Effects
Data & Society researchers Kadija Ferryman and Mikaela Pitcan's warning that 'bias at any point in data handling for precision medicine can lead to the recapitulation of longstanding health disparities' reads here as an early flag in what became a documented arc. Two years later, a STAT investigation found care-targeting software infusing racial bias into decisions about which US patients get stepped-up care — exactly the mechanism the researchers described.
The deeper problem the article surfaces is that medical data is messier than the web-scale datasets AI builders are used to: later reporting found healthcare AI underdelivering because medical data is complex and scarce, producing misleading results. Bias isn't a bug at the margins of these systems — it flows from the data itself.
First-order effects
- Hospitals deploying AI-powered decision support tools — often novel and unproven, with most patients unaware they're in use — risk encoding racial and gender disparities directly into decisions about who receives escalated care.
- Vendors of diagnostic and triage systems inherit a validation burden: their training data must be audited for demographic skew before outputs can be trusted for real medical decisions.
Second-order effects
- Clinician judgment becomes the last line of defense against biased outputs, but reporting on US hospitals shows some clinicians feeling pressure from administrations to defer to flawed algorithms — meaning organizational incentives can amplify rather than check the bias.
- Imaging systems like Google's mammogram reader show the adjacent risk: AI layered onto existing care patterns can worsen overtesting, overdiagnosis, and overtreatment for the populations already worst served.
Third-order effects
- If deployment keeps outrunning validation, expect regulators and payers to demand bias audits and demographic performance evidence as a condition of clinical use — shifting healthcare AI procurement toward documented fairness, not just accuracy claims.
- The structural outcome is a split market: systems trained on representative, well-curated medical data become defensible assets, while tools built on skewed datasets face liability and exclusion from care pathways.
The trend: Healthcare AI adoption is outpacing validation, turning training-data bias from a research concern into a live determinant of who gets care.