Asking ChatGPT to repeat words “forever” is now against OpenAI's terms of service, after Google researchers showed that doing so led it to reveal training data
Join the newsletter to get the latest updates. — Success — Great! Check your inbox and click the link. — Error
Context & Ripple Effects
The policy change follows researchers' demonstration of a divergence attack that elicited training-data sequences through repetitive prompting. It turns a model-behavior finding into an explicit user-policy issue for OpenAI.
Later coverage of ChatGPT has also centered on tightening exposure pathways, including the removal of a conversation-discovery feature that exposed chats to search engines. Together, these episodes show privacy and data handling becoming product-governance concerns, not only model-design questions.
First-order effects
- ChatGPT users are now contractually barred from using the repetitive-prompt technique, giving OpenAI a basis to limit or act on attempts to induce training-data disclosure.
- OpenAI shifts its immediate response from merely addressing an observed model failure to setting a usage boundary around the triggering behavior.
Second-order effects
- Security researchers and developers testing model memorization face a narrower line between legitimate evaluation and prohibited prompting, increasing the importance of coordinated disclosure and provider-approved testing channels.
- Other model providers have an incentive to assess whether similarly simple prompt patterns can expose memorized material and whether their policies and safeguards address them.
Third-order effects
- If such restrictions become routine, AI safety assurance will increasingly combine technical mitigations with terms enforcement—a governance model that can deter misuse but does not by itself establish that the underlying leakage risk is eliminated.
- The episode reinforces pressure for clearer boundaries around training-data provenance, memorization testing, and accountability when consumer-facing models reproduce source material.
The trend: Generative-AI providers are moving from broad content rules toward more specific governance of prompt techniques that can expose data or create model-safety risks.