Study: researchers say an algorithm widely used in the US healthcare system to assess medical needs for millions of people is biased against black patients
A series of studies argue that by focusing on costs as a proxy for health, risk algorithms are ignoring the racial inequalities in healthcare access.
Hospitals and insurers running the algorithm must re-audit who it referred into high-risk care-management programs, where Black patients were effectively under-selected despite equal or greater illness.
Black patients wrongly scored as lower-need lose immediate access to extra services — case management, follow-up, specialist referral — that the score gates.
Second-order effects
Algorithm vendors face pressure to swap cost-based labels for direct clinical measures of need, forcing retraining and revalidation cycles their payer customers fund.
Payers' population-health savings math comes into question: if the tool misallocates care by race, projected savings from 'high-risk' targeting are built on a skewed denominator.
Third-order effects
If the pattern holds across the documented cases — Arkansas benefits screening, transplant prioritization, care-targeting software — clinical algorithms drift toward mandatory validation and bias audits before deployment, analogous to drug approval rather than software procurement.
Proxy variables become a regulated design choice: any model substituting an accessible metric (cost, utilization) for ground truth will need evidence the proxy holds across racial and socioeconomic groups.
The trend: US healthcare is moving from deploying risk algorithms at scale to auditing what they actually measure, with cost-for-need proxies emerging as the canonical failure mode.
Wow: A health algorithm used in hospitals dramatically underestimated the medical needs of black patients, reducing the care they could get. “Correcting the bias would more than double the number of black patients flagged as at risk” https://www.washingtonpost.com/ ... by @Caroly…
@jhunursing
Hopkins Nursing on x
Suchi Saria, a machine learning and health-care expert at @JohnsHopkins, said the study was fascinating because it showed how, once a bias is detected, it can be corrected. https://www.washingtonpost.com/ ...
@maliciarogue
@maliciarogue on x
Black patients were less likely than white ones to get extra medical help, despite being sicker, when an algorithm used by a large hospital chose who got the attention, acc to a new study underscoring the risks as tech gains a foothold in medicine https://www.wsj.com/...
@nnimaa
Nima Aghaeepour on x
Seems like every generation of scientists has to re-learn this lesson the hard way! This mouse research mentality of small and homogeneous datasets is inconsistent with both #AI and #epidemiology. https://twitter.com/...
@hardmaru
@hardmaru on x
An algorithm that a major medical center used to identify patients for extra care has been shown to be racially biased, according to a paper published in Science. The IEEE Spectrum article has a nuanced discussion. paper https://science.sciencemag.org/ ... article https://spectru…
@chrissyfarr
Christina Farr on x
I also wonder whether women are going to be (potentially fatally) misdirected when triaging/symptom-checking chatbots become more pervasive. For instance, women can have different symptoms when experiencing heart attacks. https://www.wsj.com/...
@andrhia
Andrea Phillips on x
Gosh, it's almost like if you train algorithms with racist data, they turn out to make racist decisions! https://twitter.com/...
@blakersdozen
Blake Montgomery on x
Frightening: “Black patients who were ranked by the algorithm as equally as in need of extra care as white patients were much sicker.” https://www.washingtonpost.com/ ...
@brianrahmer
Brian Rahmer on x
“...the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts.” Dissecting racial bias in an algorithm used to manage the health of populations. https://twitter.com/... | https://science.sciencemag.org/ …
@atulbutte
Atul Butte on x
Would you get into a self-driving car trained only in Mountain View? Probably not. Then we better not tolerate medical #AI algorithms trained w/ small populations or only certain race/ethnicities! https://www.washingtonpost.com/ ...
@dermhag
Howard Green on x
What a surprise; For-profit industry coded, engineered and manufactured software and algorithms are biased to increase profits and earnings for the industry coding the software at the expense of patients and their doctors. Who woulda thought? https://twitter.com/...
@naacp_ldf
@naacp_ldf on x
“When the company replicated the analysis on a national data set...they found that black patients who were ranked by the algorithm as equally as in need of extra care as white patients were much sicker” https://www.washingtonpost.com/ ...
@motherboard
@motherboard on x
A series of studies argue that by focusing on costs as a proxy for health, risk algorithms are ignoring the racial inequalities in healthcare access. https://www.vice.com/...
@ravindranize
Mohana Ravindranath on x
Here's the study in Science. Researchers note that the algorithm predicts costs, not illness....and “unequal access to care means that we spend less money caring for Black patients than for White patients.” https://science.sciencemag.org/ ... https://twitter.com/...
@jfbonnefon
Jean-Franois Bonnefon on x
❝Dissecting racial bias in an algorithm used to manage the health of populations❞ If the algorithm learns to predict health needs from examples of health costs + less money spent on black patients, then algorithm underpredicts needs of black patients https://science.sciencemag.or…