Researchers suggest that using “synthetic” data, created by AI systems to train LLMs, could lead to the rapid degradation of AI models and a collapse over time
In April this year, OpenAI CEO Sam Altman was asked during … Lindsay Clark / The Register : AI models face collapse if they overdose on their own output Tech Xplore : Using AI to train AI: Model collapse could be coming for LLMs, say researchers Scott J Mulligan / MIT Technology Review : AI trained on AI garbage spits out AI garbage Devin Coldewey / TechCrunch : ‘Model collapse’: Scientists warn against letting AI eat its own tail M.G. Siegler / Spyglass : AI's Potential Ouroboros Problem Threads: Mark Zuckerberg / @zuck : Distilling models into smaller models that are almost as smart but only a fraction as expensive to run clearly works and is a lot of what I expect people to use Llama 3.1 405B to do. There's also evidence that you can further train these smaller models to surpass the intelligence of the teacher model, although that's not necessarily using synthetic data from the teacher. Sam Lessin / @lessin : The whole story of using AI to generate training data to keep training AI has always been a head scratcher to me. It never made sense to me why that would work, vs. just drift rapidly into nonsense — but for a long time, lots of people seemed to believe it. I am glad to see that the research is coming back to in-line with what intuitive expectations would be... X: Bill Gurley / @bgurley : Worth a read. Results seem interestingly similar to inbreeding w species. 🥸 Forums: Hacker News : The problem of ‘model collapse’: how a lack of human data limits AI progress
Context & Ripple Effects
The warning arrives as the quality and availability of training corpora are becoming a constraint. The removal of access to LAION-5B after researchers identified abusive material in it illustrates why simply relying on large web-scale datasets is not frictionless.
It also sharpens a tension in model-development strategy: Mark Zuckerberg has promoted distilling larger models into cheaper ones, while researchers caution that repeated reliance on model-produced material can erode the information a successor model learns from.
First-order effects
- Teams using AI-generated material in training pipelines face a more immediate need to distinguish controlled synthetic-data uses from feedback loops that repeatedly recycle model outputs.
- The findings put added scrutiny on distillation and other model-to-model training approaches, including the cheaper-model strategy Zuckerberg described, because output quality and source diversity become central controls.
Second-order effects
- Demand rises for curated human-created, licensed, and carefully validated datasets as developers seek to preserve diversity and accuracy in training inputs.
- Model providers may need to document data provenance and filtering more clearly for customers and partners, turning dataset quality assurance into a competitive differentiator rather than a back-office task.
Third-order effects
- If high-quality non-synthetic data remains scarce, access to trusted data sources could become a durable advantage for a smaller set of frontier-model developers—a dynamic reflected in the growing shortage of conventional web training data.
- The field may shift from treating synthetic data as a broadly scalable substitute to using it in bounded, audited workflows; whether that limits consolidation depends on how effectively developers can measure and prevent feedback-loop degradation.
The trend: AI development is moving from a race to collect more training material toward a race to secure, verify, and govern the quality of that material.