Interviews and documents show that One Concern, which wants to use AI to help emergency responders in disasters, dangerously exaggerated its tools' abilities
Sheri Fink / New York Times : Tweets: @random_walker , @nytimes , @jimmyjwu , @robreich , @jswatz , @kellyhereid , @kellyhereid , and @kellyhereid Tweets: Arvind Narayanan / @random_walker : The fake-it-till-you-make-it culture of Silicon Valley has no place in domains where people's lives are at stake. How many repeats of Theranos are we going to have? https://www.nytimes.com/... HT @Aaroth. https://twitter.com/... @nytimes : Cities are turning to an A.I. start-up to help save lives during natural disasters. But some fear its promise has been dangerously exaggerated. https://www.nytimes.com/... Jimmy Wu / @jimmyjwu : When you try to do ethical machine learning under capitalism but end up funneling data to insurance companies for a living: https://twitter.com/... Rob Reich / @robreich : And the standard of “don't be Theranos” is not a high bar. https://twitter.com/... John Schwartz / @jswatz : “The usual start-up attitude of ‘fail fast and try something new’ was inappropriate to apply to disaster response because, ‘If you fail fast, people die.’” Eye opening story from @sherifink https://www.nytimes.com/... Kelly Hereid / @kellyhereid : The insurance industry uses cat models to make realistic estimates of how damaging a whole suite of natural perils will be - hurricanes, floods, wildfires, earthquakes, and so forth. Thousands of years of hypothetical events, and huge catalogs of historical “as-if” scenarios 3/n Kelly Hereid / @kellyhereid : So to wade in... What One Concern has built is a catastrophe model. We can quibble about how it's done it, whether it uses public data/AI/magical fairy dust/whatever, but that's what it is - a model that predicts damage from a disaster 2/n Kelly Hereid / @kellyhereid : So, I imagine @oneconcerninc is currently formulating a response to this @nytimes piece, but here are some personal thoughts, from yours truly, sitting in a space that intersects with its field fairly closely. https://www.nytimes.com/... 1/n
Context & Ripple Effects
Sheri Fink's investigation lands at the intersection of two threads this coverage has been following: civic AI sold on promise rather than proof, and the researchers who catalog that gap. Arvind Narayanan's immediate framing — that Silicon Valley's fake-it-till-you-make-it culture has no place where lives are at stake, and that Theranos-style repeats keep recurring — turns One Concern from a startup story into a case study in unverified high-stakes AI.
The piece also predates the institutional response the corpus documents: the safety-first culture later profiled at Anthropic, Open Philanthropy's funding of AI-risk work, and Wired's warning that clinical AI like Google's mammography system can amplify existing harms. One Concern is an early data point for why that skepticism hardened.
First-order effects
- Cities and emergency responders that bought One Concern's disaster-planning tools were making life-safety decisions on capabilities the company had overstated — the direct cost falls on their planning assumptions, not just their budgets.
- One Concern's sales pipeline and credibility with government buyers take the immediate hit, since the reporting undermines the exact claim — that its models can predict disaster impact — it was selling on.
Second-order effects
- Other vendors selling predictive AI to governments now face procurement-level demands for independent validation, because One Concern showed municipal buyers they cannot trust vendor self-assessment in life-safety categories.
- The episode feeds the researcher-led critique Narayanan and Sayash Kapoor later formalized — the same authors behind the Princeton pair's 'AI Snake Oil' work — giving journalists and buyers a template for auditing capability claims across adjacent high-stakes domains like [[a:949550|clinical AI, where Wired found Google's mammography system could worsen overtesting and overtreatment]].
Third-order effects
- If the pattern holds, life-safety AI markets split between unaudited startups and tools that survive third-party evaluation, with public agencies — the buyers least equipped to verify models themselves — driving demand for external audit as a condition of purchase.
- The deeper structural question the corpus leaves open is whether verification becomes regulation or stays reputational; the security-researcher interviews noting that global regulatory frameworks lag suggest the enforcement layer arrives slowly, leaving exposure in the interim.
The trend: High-stakes 'AI for good' vendors are being forced from claimed capability toward independently verified capability, as researchers and public-sector buyers treat fake-it-till-you-make-it as a safety failure rather than a growth strategy.