Experts raise concerns about the use of a predictive child welfare algorithm in Allegheny County, PA, and its potential to harden racial disparity in the system
A weekly newsletter from the Ira A. Lipman Center For Journalism … Tweets: Jon Schiefer / @brandxjon : When a society trains an AI, it's going to be trained on aspects of that society and will reflect the biases therein. So this outcome is predictable. https://twitter.com/... @ap : As child welfare agencies nationwide weigh using algorithms to help decide which families are investigated, an @AP investigation finds concerns about the technology, including questions about reliability and the potential to harden racial disparities. https://apnews.com/... Lynnette KhalfaniCox / @themoneycoach : Per @AP, an algorithm in PA that screens 4 child neglect is raising concerns. It flags a disproportionate # of Black kids for “mandatory” neglect investigations. Social workers disagree w/ the algorithm's risk scores in 1/3rd of cases. #AI #BigData https://apnews.com/...
Context & Ripple Effects
This AP investigation lands mid-wave: predictive scoring had already spread from police patrols and sentencing to pretrial detention — where New Jersey's PSA tool cut jail populations but left racial disparities intact — and into benefits administration, where Arkansas's automated health assessments began cutting disabled patients' home care. Child welfare was the next frontier, and New York City had been running a family-risk scorer since 2018.
The story matters because it names the mechanism before the enforcement caught up: experts warned the Allegheny County model could harden existing racial disparities, and the follow-on coverage shows that warning aging into federal action — a DOJ discrimination probe — while New York City's own screener drew parallel bias concerns.
First-order effects
- Allegheny County families — disproportionately Black ones, per the experts cited — face screening decisions shaped by a model trained on historical agency data that already reflects the system's disparities.
- The county's child welfare agency now has to defend the tool's reliability publicly, since the AP reporting puts its methodology and error rates under national scrutiny.
Second-order effects
- Other agencies weighing adoption — New York City among them — inherit a harder sell: every new deployment now gets audited against the Allegheny example, and NYC's 2018-era screener is already facing the same racial-bias questions.
- Vendors and county governments building these tools take on civil-rights legal exposure, a risk the DOJ's later disability-discrimination investigation into the same county made concrete.
Third-order effects
- If the pattern holds across pretrial scoring, benefits automation, and child welfare, government risk-scoring shifts from a procurement decision to an accountability regime — with audits, disparate-impact testing, and litigation as standard conditions of deployment.
- Agencies may retreat to human-first screening or demand documented fairness validation before purchase, restructuring a vendor market that grew on the assumption that historical data was an acceptable training baseline.
The trend: Public-sector risk-scoring algorithms are entering an enforcement-and-audit phase, where civil-rights scrutiny — not procurement enthusiasm — determines which deployments survive.