As employers turn to algorithms to sift through job applications, the poor are more likely to lose out
Cathy O'Neil / Guardian : Tweets: @boonerang , @mkelley , @ilparone , @ilparone , @digiphile and @digiphile Tweets: Boon Sheridan / @boonerang : “Most of these algorithmic applications were created with good intentions” ... and so begins another terrible story. http://www.theguardian.com/... Mike K / @mkelley : “the Kronos test can be considered a medical exam, the use of which in hiring is illegal under the ADA of 1990.” http://twitter.com/... Jarno M. Koponen / @ilparone : “We're not told what the tests are looking for. The process is entirely opaque.” http://www.theguardian.com/... by @mathbabedotorg #ml #privacy Jarno M. Koponen / @ilparone : “We're not told what the tests are looking for. The process is entirely opaque.” https://www.theguardian.com/ ... -@mathbabedotorg v @Blakei #ml #ux Alex Howard / @digiphile : “these algorithms have the potential to create an underclass...inexplicably shut out from normal life"-@mathbabedotorg http://www.theguardian.com/... Alex Howard / @digiphile : “the algorithms that power the data economy are based on choices made by fallible human beings"-@mathbabedotorg http://www.theguardian.com/...
Context & Ripple Effects
Cathy O'Neil's Guardian piece lands at the start of an arc the related coverage traces forward: automated scoring moving from hiring into every gatekeeper role. Commenters on the piece flag that the Kronos pre-hire test may qualify as a medical exam under the ADA of 1990, making its use in hiring potentially illegal — and that applicants are never told what the tests measure.
The later coverage shows the pattern generalizing: predictive algorithms setting police patrols, prison sentences, and probation rules in the US and Europe, and services like Predictim generating AI character scores for babysitters from years of social media activity. By late 2020, civil lawyers are building litigation strategies specifically to challenge automated systems that deny the poor basic services.
First-order effects
- Low-income job applicants are filtered out by opaque pre-employment tests whose criteria are never disclosed to them, while employers like Kronos clients gain a cheaper screening layer they don't have to explain.
- The ADA question raised about the Kronos test puts vendors of personality and aptitude assessments directly in legal exposure if the tests are deemed medical exams.
Second-order effects
- Civil lawyers develop a practice area around challenging automated denials, turning individual screening decisions into class-action-sized targets for assessment vendors and their employer customers.
- Screening tools expand into adjacent markets — criminal justice risk scores, consumer-facing services like babysitter vetting — because the same opacity that shields hiring algorithms from scrutiny works elsewhere.
Third-order effects
- If the pattern holds, access to jobs, justice, and services gets mediated by proprietary scoring systems whose logic is undisclosed, with courts rather than regulators becoming the primary accountability mechanism.
- Vendors face a structural choice between explainability (which invites legal challenge but survives it) and opacity (which scales until one adverse ruling exposes the whole product line).
The trend: Automated decision systems are expanding from hiring into policing, sentencing, and everyday vetting, with litigation emerging as the main counterweight to their opacity.