A for-profit spinoff of Crisis Text Line uses anonymized data collected by the AI-driven suicide chat service to create and market customer service software
a non-profit named Crisis Text Line that offers outreach to people worried about suicide, self-harm etc, has been harvesting conversations to use as training data for an AI customer service startup https://www.politico.com/... David Carroll / @profcarroll : We need a GDPR for the USA and we need it now because non-profits should not be exempt from data protection and anonymization is a reckless myth that only a horizontal data privacy mechanism can reconcile. The key idea is we need a regulation that applies GENERALLY to everything. https://twitter.com/... Kim Zetter / @kimzetter : When you think there's no new way your privacy and information can be exploited, someone comes up with one https://twitter.com/... Eric Geller / @ericgeller : Holy cow, this @Ali_Lev story... https://www.politico.com/... https://twitter.com/...
Context & Ripple Effects
Crisis Text Line's arrangement with its for-profit spinoff Loris.ai did not emerge in a vacuum: reporting on therapy apps like Better Help had already documented how suicide-related signals inside mental-health apps get piped to third parties. What is new here is the direction of flow — a crisis service itself harvesting conversations as training data for a commercial product.
The scrutiny was immediate. Within days of publication, Crisis Text Line ended the data-sharing relationship and asked Loris.ai to delete existing data, while privacy advocates like David Carroll seized on the case to argue that anonymization is not a substitute for a general-purpose US privacy law.
First-order effects
- People who texted Crisis Text Line at moments of suicidal crisis learned their conversations had been repurposed as training data for Loris.ai's customer-service software, putting both organizations under immediate reputational fire.
- Loris.ai loses its founding dataset claim: the software it markets as informed by real crisis conversations now carries a provenance problem it must explain to enterprise buyers.
Second-order effects
- Every nonprofit operating a helpline or chat service now faces donor and press questions about whether its data flows to commercial entities, forcing audits of spinoff and licensing arrangements across the sector.
- The episode hands ammunition to advocates pressing for US legislation modeled on GDPR, since the case shows a nonprofit exploiting the gap left by the absence of a horizontal federal privacy law.
Third-order effects
- If the pattern holds, 'anonymized' will stop functioning as a legal safe harbor for sensitive conversational data, pushing crisis and mental-health platforms toward explicit consent models rather than blanket terms-of-service harvesting.
- Nonprofit-to-for-profit spinoffs may face structural separation on data governance: the charitable arm and the commercial arm can no longer assume shared access to user data without independent oversight.
The trend: Crisis and mental-health platforms are being forced to treat user conversations as governed personal data rather than a free training corpus, with anonymization claims losing credibility as a defense.