A game made with a real world dataset of defendants shows the shortcomings of COMPAS, an AI-powered risk assessment tool used in the US criminal legal system
The US criminal legal system uses predictive algorithms to try to make the judicial process less biased. But there's a deeper problem. Tweets: @_karenhao , @marylgray , @ledataminer , @varoonmathur , @ainowinstitute , @jason_pontin , and @hiphination Tweets: Karen Hao / @_karenhao : IT'S HERE!!! The biggest story I've ever worked on. @techreview's very first interactive ever, which walks through a concrete example of AI bias, and why it's so much more complicated than initially meets the eye. https://www.technologyreview.com/ ... 1/ Mary L. Gray / @marylgray : Original, critical, thought-provoking journalism—idata-driven reporting at its best. It doesn't let data sit self-evident. Instead, @_KarenHao and @techreview colleagues ask us to consider how we impose tech on each other, particularly the most disenfranchised 👇🏼👇🏼 👇🏼 https://twitter.com/... Franois Petitjean / @ledataminer : Such a fantastic, didactic tool to explain the issues of fairness and AI to non-specialists. Hard recommend. Via @sarahbmyers. In case you didn'tsee this @westylesty @rgibli @kim_weatherall @Lizzie_OShea @parismarx https://twitter.com/... Varoon Mathur / @varoonmathur : This is really good - and highlights well the divide between algorithmic “fairness” and actual justice. https://twitter.com/... @ainowinstitute : Great article by @_KarenHao that shows why predictive algorithms don't make the judicial process more fair - and why we need impact assessments 👍 (nice to see shout outs to @ruha9 and our policy director Rashida Richardson) https://twitter.com/... Jason Pontin / @jason_pontin : This interactive exploration of the criminal risk assessment algorithm COMPAS by @_KarenHao and @jonathanstray is one of the best things published by @techreview. It unpacks the conflicts inherent in any such tool in a way a linear narrative could not. https://www.technologyreview.com/ ... Hi-Phi Nation / @hiphination : MIT has made a visual to show the exact problems with algorithmic risk assessment to date, and essentially why there is no solution. https://www.technologyreview.com/ ...
Context & Ripple Effects
MIT Technology Review turned its first interactive feature into a teaching instrument: a built by Karen Hao's team around a real-world defendant dataset, it walks readers through the COMPAS risk-score trade-offs that make 'less biased' far harder to define than it looks. The timing matters because the same tooling was spreading — reporting months later documented predictive algorithms setting police patrols, prison sentences, and probation rules across the US and Europe (New York Times).
First-order effects
- Courts and probation systems relying on COMPAS face a public that can now play the fairness trade-off themselves rather than read an abstract critique, raising pressure on vendors to explain score construction.
- Journalism shifts from exposing single incidents to demonstrating structural bias interactively, changing what 'proof' of algorithmic unfairness looks like for judges and policymakers.
Second-order effects
- Defense counsel gain a reusable argument template: the civil-litigation strategies against automated systems and the defense challenges to Cybercheck's accuracy both follow the path this interactive opened — attacking the tool, not just one case outcome.
- Vendors of risk-assessment tools are pushed toward disclosure of error rates and demographic performance, since adversarial scrutiny now comes from journalists, academics, and lawyers simultaneously.
Third-order effects
- If every deployed risk tool attracts its own audit-and-litigation cycle, criminal-justice AI procurement moves toward contestable systems whose methods must survive courtroom and press examination by default.
- The deeper lesson of the game — that competing definitions of fairness force unavoidable value judgments — points toward formal governance standards for public-safety algorithms rather than case-by-case scandal.
The trend: Criminal-justice AI is entering an accountability phase where interactive audits and defense-bar litigation, not vendor claims, define whether risk-assessment tools stay deployed.