Researchers: some Reddit usernames and other keywords cause ChatGPT to give bizarre responses, likely due to the web data OpenAI scraped to train its model
Chloe Xiang / VICE :
Context & Ripple Effects
This report is the earliest entry in a running line of research showing that ChatGPT's behavior is entangled with the specific web corpus OpenAI trained on: certain Reddit usernames and keywords reliably produce bizarre outputs, implying individual scraped identities are baked into the model. Later findings sharpened the point — a [[a:846850|divergence attack that makes the model emit sequences copied verbatim from its training data]] showed extraction is systematic, not anecdotal.
The story also sits on the Reddit side of a feedback loop: by April, moderators were reporting AI-generated spam hitting Reddit in higher volume and at faster post speeds, meaning the platform that contaminated the training data was itself being flooded with model output. And OpenAI's eventual handling of the "David Mayer" name-refusal episode suggests the company ended up patching name-by-name behavior rather than fixing the corpus.
First-order effects
- Reddit users whose usernames act as triggers have their identities functionally wired into a mass-market product's failure modes, without consent or a removal mechanism — the affected names surface in bizarre responses for anyone who types them.
- OpenAI faces a data-quality problem it cannot patch at the source: the scraped web data is already in the weights, so remediation has to happen in model behavior, not in the dataset.
Second-order effects
- Researchers gain a reproducible probe for training-data extraction — the keyword-trigger finding and the divergence attack together give the field a method to map what specific scraped content a model absorbed.
- Reddit's corpus becomes both more contested and more consequential: moderators battling AI spam are managing the same platform whose data quality now demonstrably shapes frontier-model behavior, raising the stakes of any future licensing or blocking decisions.
Third-order effects
- If scraped-web provenance keeps surfacing in outputs, training-data hygiene and individual privacy shift from academic critique to operational liability — OpenAI's name-specific refusals hint at a future of per-entity patches layered over a corpus that cannot be edited.
- The Reddit-to-model-to-Reddit loop points toward a structural problem for the next training generation: models trained on web data increasingly generate the web data itself, degrading provenance for whoever trains on it later.
The trend: Frontier-model behavior is increasingly traceable to specific scraped web sources, pushing OpenAI toward name-by-name behavioral patches while the platforms it scraped become both its contamination source and its output dumping ground.