Investigation: software used for targeting medical services to US patients is infusing racial bias into decision-making about who should receive stepped-up care
Casey Ross / STAT : Tweets: @frankpasquale , @rsvassellmd , @ravindranize , @docjeffd , @uche_blackstock , @erbrod , @statnews , and @rebeccadrobbins Tweets: Frank Pasquale / @frankpasquale : “None of the top products for analyzing patient populations explicitly warn users that racial differences in access to care could skew their referral decisions. Instead, their online brochures promise to accurately identify” high-need patients. https://www.statnews.com/... Rashida S. Vassell / @rsvassellmd : My God. This is what we mean when we decry structural racism. Sometimes we can't quite explain it but, we can feel it in our bones. Why does this patient meet criteria for admission and this one doesn't? Why is this procedure/treatment approved for this patient and not that one? https://twitter.com/... Mohana Ravindranath / @ravindranize : “None of the developers of the most widely used software systems warns users about the risk of racial disparities.” https://twitter.com/... Jeffrey Duchin, MD / @docjeffd : Bias from algorithms used in hospitals across the country reinforces deeply rooted inequities in the American health care system, walling off low-income Black & Hispanic patients from services that less sick white patients routinely receive. https://www.statnews.com/... Uch Blackstock, Md / @uche_blackstock : A STAT investigation found that a common method of using analytics software to target medical services to patients who need them most is infusing racial bias into decision-making about who should receive stepped-up care. https://www.statnews.com/... Erin Brodwin / @erbrod : Insanely proud to work with @caseymross, who wrote this fantastic look into how software infuses racism into health care. https://www.statnews.com/... “The hard lines of segregation have faded in Ahoskie. But in health care, a new force is redrawing those barriers.” Stat / @statnews : The racial bias can produce huge differences in assessing patients' need for special care to manage conditions such as hypertension, diabetes, or depression, a STAT investigation found. https://www.statnews.com/... Rebecca Robbins / @rebeccadrobbins : Remember that study last year that documented racial bias in the use of an algorithm in one health system? A @statnews investigation by @caseymross found that the problems arise from multiple algorithms used in hospitals across the country https://www.statnews.com/...
Context & Ripple Effects
This investigation extends an arc STAT and others have been building since 2018: healthcare AI inherits the biases of its inputs, as researchers warned when they found healthcare AI carrying the racial and gender biases of its training data. Last year's study of a widely used needs-assessment algorithm biased against black patients showed the problem inside a single deployed system; today's reporting widens the lens to the vendor market itself.
First-order effects
- Hospitals and clinics using these population-health products are making referral decisions for stepped-up care with tools whose brochures promise accuracy but carry no warning that racial differences in access to care can skew their outputs.
- Vendors named in the investigation now face direct pressure to disclose how their scoring handles patients who historically received less care — the same exposure that followed last year's algorithm-bias study.
Second-order effects
- Buyers who adopted these tools amid the quiet rollout STAT documented earlier this year — where most patients were unaware hospitals were introducing AI-powered decision support — must now weigh auditing or replacing systems sold as turnkey.
- The precedent echoes the Arkansas case, where algorithms automating health assessments cut essential state-sponsored home care for disabled patients; state programs and payers become the likeliest first movers toward contractual audit rights.
Third-order effects
- If undisclosed access-to-care bias proves systemic across top products rather than idiosyncratic to one algorithm, procurement shifts from trusting vendor claims to requiring bias disclosures and validation data — a structural change in how health systems buy analytics software.
- Regulators and litigators gain a clearer target: the gap between what marketing materials promise ('accurately identify high-need patients') and what the tools actually encode points toward disclosure mandates for clinical decision software.
The trend: Healthcare AI accountability is moving from isolated studies of single biased algorithms to scrutiny of the entire commercial pipeline that ships them into clinical decisions.