As governments adopt third-party automated fraud detection algorithms, researchers say the companies running the systems are often overpaid and under-supervised
Morgan Meaker / Wired : Tweets: @glichfield Tweets: Gideon Lichfield / @glichfield : The final story in the series takes a broader look at the algorithmic fraud-detection industry and shows the flaw at its heart: Lacking the expertise to build these systems in-house, governments typically subcontract them to consulting firms. https://www.wired.com/...
Context & Ripple Effects
This closes a Wired series on algorithmic fraud detection, and it lands two days after the outlet's reconstruction of Rotterdam's Accenture-built welfare fraud algorithm showed ethnicity and gender discrimination baked into its training data. The through-line: governments lack the expertise to build these systems in-house, so they subcontract to consulting firms — and then struggle to supervise what they bought.
The failure mode isn't hypothetical. Michigan's flawed automated unemployment-fraud system falsely charged thousands and collected millions in fines, while defense lawyers have spent years challenging Cybercheck, an AI tool used across thousands of US cases. As researchers argue, audits alone don't fix it because there's no market incentive for vendors to submit to real oversight.
First-order effects
- Consulting firms running government fraud-detection systems now face direct questions about their pricing and supervision arrangements — the article names them as often overpaid by clients who can't evaluate the work they're buying.
- The governments deploying these systems carry the immediate reputational and legal exposure when algorithms wrongly flag citizens, as Michigan's fines and Rotterdam's discriminatory model already demonstrated.
Second-order effects
- Vendors' pricing power grows precisely where state capacity is weakest: without in-house expertise, agencies can't benchmark contracts or verify outputs, reinforcing dependence on the same consultants they'd need to hire to check them.
- Liability fights sharpen around who eats fraud losses — as with the unresolved dispute among governments, banks, and tech companies over who should cover AI-driven payment fraud, under-supervised third-party systems blur who is answerable when detection goes wrong.
Third-order effects
- If the pattern holds, public-sector AI procurement becomes structurally dependent on a small set of consultancies whose incentives run against transparency — the oversight-inversion problem where the regulated party controls the evidence of its own performance.
- The likely corrective is regulatory: mandated disclosure of training data, independent audit rights written into contracts, and loss-allocation rules — turning what was a purchasing decision into a governance regime for state-deployed algorithms.
The trend: Government adoption of third-party fraud-detection AI is outpacing the state's capacity to supervise it, pushing algorithmic accountability from voluntary audits toward contractually enforced public oversight.