As US schools adopt AI, a look at the growing use of AI-powered monitoring tools like GoGuardian to scan students' chatbot conversations for signs of self-harm
As US educators embrace AI in the classroom, firms are selling software to flag mentions of self-harm, raising concerns over privacy and control. Bluesky: @cdt.org , @hypervisible.blacksky.app , and @charleswlogan . Mastodon: @remixtures@tldr.nettime.org . Forums: r/artificial Bluesky: @cdt.org : “In a recent survey by CDT, a nonprofit civil-liberties body, 29% of responding teachers whose schools participate in monitoring said students' personal devices were also tracked by their schools through apps, browser extensions or network filters.” @hypervisible.blacksky.app : Student surveillance companies cash in on the ai grift by promising schools and parents their tech will keep students safe from the harms of ai. Charles Logan / @charleswlogan : And this: article cites a recent @cdt.org survey, where, “17% of teachers reported that student information such as grades, attendance and discipline information was shared with immigration enforcement, and some school staff were reporting members of the school community to ICE of their own accord.” … Mastodon: Miguel Afonso Caetano / @remixtures@tldr.nettime.org : “Ahead of the new school year, a handful of tech companies issued grim warnings to American educators embracing artificial intelligence in the classroom: Chatbots, they said, could endanger students and lead to self-harm. Vigilance was paramount. ” The risks of students using AI can literally be deadly," one company cautioned. … Forums: r/artificial : Chatbots Are Sparking a New Era of Student Surveillance
Context & Ripple Effects
US schools' classroom AI uptake has been accompanied by vendor and union-backed education initiatives, as covered in the recent expansion of AI adoption across schools and colleges. This reporting shows that the same shift is also widening the data schools can observe.
The monitoring market is not new: GoGuardian use grew sharply during lockdown-era remote learning, while schools had already been adopting social-media monitoring with unresolved effectiveness and privacy questions. Chatbot conversations add a more intimate category of student activity to that established surveillance infrastructure.
First-order effects
- Students using school-managed devices, accounts, extensions or networks may have chatbot conversations scanned for self-harm signals, extending monitoring into AI-assisted interactions.
- GoGuardian and comparable vendors gain a safety-focused sales case as schools introduce chatbots, while educators and administrators must handle alerts alongside privacy and access-control concerns.
Second-order effects
- Schools may face pressure to clarify which devices, accounts and conversations are covered, especially because CDT's survey found some teachers reporting monitoring of personal devices.
- AI classroom adoption becomes more closely tied to surveillance procurement: chatbot providers and school IT teams may be judged not only on instructional use, but on how readily activity can be monitored and escalated.
Third-order effects
- If safety claims continue to drive deployments, education AI could normalize an enforcement layer around student use rather than treating classroom AI solely as a learning tool; whether that improves safety remains a central unresolved question.
- The overlap of sensitive student data, automated flags and school authority increases the case for durable governance over access, retention and human review, particularly where monitoring can reach beyond school-owned devices.
The trend: AI's entry into schools is creating a broader AI enforcement surface, in which tools adopted for learning also become channels for safety monitoring and institutional control.