New York City's AI tool, launched in 2018 to score families at risk for child abuse using factors like their neighborhood, is raising racial bias concerns
“They target you from the minute you step out of foster care,” Hamblin said. “They have this microscope on you, and for them to build a case based off your prior records of you being in care—it's not right. — themarkup.org/investigatio... … @themarkup.org : The algorithm doesn't explicitly use race to score families, but uses “variables that may act as partial proxies for race (e.g., geography)” to make its decisions, according to a report obtained by The Markup. themarkup.org/investigatio... @themarkup.org : Despite Black people being only about 23 percent of the population in New York City, Black children make up 52 percent of children removed from their home without a court order, an ACLU report found. themarkup.org/investigatio... Khari Johnson / @khari : “The baby's not even here yet and you're already talking about removing her?” themarkup.org/investigatio... @themarkup.org : A 2020 draft report by NYC's child services found that caseworkers believed the agency was unfairly targeting Black families, contributing to a “predatory system” that treats families of color as inherently suspicious. themarkup.org/investigatio... Brian Dusablon / @drifterlife.com : “While the tool is new, both the data it's built on and the factors it considers raise the concern that artificial intelligence will re-inforce or even amplify how racial discrimination taints child protection investigations in New York City and beyond...” — themarkup.org/investigatio...
Context & Ripple Effects
New York City’s child-welfare scoring system extends a broader use of predictive tools in high-consequence public decisions. Earlier coverage documented predictive algorithms spreading into policing, sentencing and probation, where model outputs can shape who receives scrutiny.
The child-welfare debate already has a close precedent: experts questioned an Allegheny County risk-scoring system’s potential to deepen disparities, followed by reporting of a DOJ inquiry into whether that system discriminated against parents with disabilities. NYC’s reported use of geographic and historical-record proxies makes that prior scrutiny directly relevant.
First-order effects
- Families flagged by the tool may face intensified child-protection scrutiny even though race is not an explicit input; geography and prior-system records can carry unequal historical exposure into the score.
- NYC child-services leaders face pressure to explain how the model is used alongside caseworker judgment and whether its inputs or outcomes disproportionately burden Black families.
Second-order effects
- Advocates and oversight bodies gain a concrete basis to seek testing of proxy variables, disclosure of decision rules, and review of removals or investigations associated with risk scores.
- Other agencies using predictive child-welfare systems will face stronger comparisons with the federal scrutiny of Allegheny County’s tool, raising the operational cost of deploying models without demonstrable fairness controls.
Third-order effects
- If agencies continue to use administrative data as a proxy-rich input to consequential decisions, AI governance will increasingly turn on measurable outcomes and contestability rather than whether a model directly includes protected traits.
- The pattern could shift public-sector procurement toward systems with auditable inputs, documented human review, and mechanisms for affected families to challenge automated risk signals; the extent of that shift depends on enforcement and disclosure requirements.
The trend: Predictive systems in public services are moving from experimental decision support toward a governance test over whether proxy data can be used fairly in coercive or high-stakes interventions.