How algorithmic systems to score humans like FICO and China's Sesame Credit can lead to social control, and some possible remedies to this problem
The risks — and benefits — of letting algorithms judge us — Bruce Schneier is a security technologist and chief technology officer …
Context & Ripple Effects
Writing before most of this coverage existed, Bruce Schneier framed a question that the following years kept answering: what happens when institutions adopt the FICO model — a single number summarizing a person — for domains beyond credit? His warning pointed at China's Sesame Credit as the state-controlled version of the pattern.
The corpus confirms the spread he predicted. Months after this piece, [[a:878441|China began piloting a system combining citizen records into a social credit score governing jobs, loans, and travel]]; by 2019, private-sector equivalents had arrived with [[a:940377|fraud-detection services like Sift and SecureAuth generating user-trustworthiness scores from thousands of opaque signals]], and predictive tools were reaching police patrols, prison sentences, and probation decisions across the US and Europe.
First-order effects
- Individuals are already being judged by scores they cannot inspect or contest: Chinese citizens under the social credit pilot, US defendants facing predictive sentencing inputs, and job applicants screened by algorithms shown to disadvantage the poor.
Second-order effects
- Score providers expand their footprint — fraud-scoring vendors push into adjacent trust decisions while governments adapt the same technique for policing — so each new domain normalizes scoring in the next, widening the market Schneier warned about.
Third-order effects
- If scoring keeps migrating from credit into employment, justice, and civic eligibility, access to basic opportunities becomes mediated by proprietary or state-run numbers, making remedies like transparency requirements and contestability a governance question rather than a consumer-protection one.
The trend: Person-scoring is expanding from its financial origin in credit bureaus like FICO into commerce, criminal justice, and state administration, with China's social credit system as the most explicit case.