AI companies, running out of conventional training datasets from the web, may be forced to shift from big, all-purpose LLMs to smaller, more specialized models
why human-sourced data can help prevent AI model collapse Matthias Bastian / The Decoder : OpenAI co-founder says AI is reaching “peak data” as it hits the limits of the internet Kylie Robison / The Verge : During his NeurIPS talk, Ilya Sutskever says “Pre-training as we know it will end”, as “we've achieved peak data and there'll be no more” Bluesky: James McDermott / @jmmcd : I refuse to read the article but I wonder what scenario people like this imagine, when they hear we're running out of data. Do they think it's used up, or deleted? [embedded post] Nicolai B. Hansen / @nbhansen.dk : Honestly much more interested in small specialized models than big all purpose models. They so far have not really been that useful. [embedded post] Casey Newton / @caseynewton : Seems to me that one answer to “we have run out of data to steal” could be to pay people to make stuff and use it with their consent [embedded post] @photogenealogy : Using AI generated data to train AI! — 😳 That can't be good — www.nature.com/articles/d41... Scott McGrath / @smcgrath.phd : 🧪 AI faces a significant data bottleneck: by 2028, training data may equal the total stock of public online text. — Solutions like synthetic data, smaller models, and specialized datasets are key to future advances.🩺💻 #MLSky Katherine Stiles / @katherinestiles.org : The Internet is a vast ocean of human knowledge, but it isn't infinite. And artificial intelligence (AI) researchers have nearly sucked it dry. Christian Frezza / @frezzalab : The sad realistion that someone will have to do experiments at some point....damn it — “The AI revolution is running out of data. What can researchers do?” — www.nature.com/articles/d41... X: Eric Topol / @erictopol : What happens when LLMs run out of data to ingest? https://www.nature.com/... @nature feature by @nicolakimjones [image] @bermaninstitute : The AI revolution is running out of data. What can researchers do? AI developers are rapidly picking the Internet clean to train large language models such as those behind ChatGPT. Here's how they are trying to get around the problem. https://www.nature.com/... Steven Ashley / @steveashleyplus : At current growth rates, the AI industry runs out of readily accessed HQ data in four yrs... Workarounds incl using synthetic data and less-easily-accessed data. (N) https://www.nature.com/... Nicola Jones / @nicolakimjones : “Compute is growing but the data is not growing... data is the fossil fuel of AI” - Ilya Sutskever #NeurIPS2024 Read my story in Nature about the data shortage: https://www.nature.com/...
Context & Ripple Effects
Earlier coverage framed synthetic training data as a risky substitute: researchers warned that recursively using AI-generated material could degrade models over time and risk a model-collapse feedback loop. That makes the supply of reliable human-created data a strategic constraint rather than merely an input to scale.
The story also follows Ilya Sutskever's shift toward finding the field's “next thing” beyond familiar pre-training approaches as the limits of conventional pre-training became more salient. It matters because a data ceiling would change which model-development paths remain practical.
First-order effects
- AI model builders face pressure to prioritize consented human data and narrower, domain-specific training over simply expanding broad web corpora.
- Teams pursuing general-purpose frontier models must reassess pre-training roadmaps, with synthetic data remaining an uncertain workaround rather than a straightforward replacement.
Second-order effects
- Competition for proprietary or paid-for human datasets is likely to intensify, favoring organizations that can secure reliable data rights and quality controls.
- Customers and developers may place more value on specialized models where focused datasets can produce useful performance without relying on ever-larger general training runs.
Third-order effects
- If the constraint persists, data provenance and access could become a more durable source of AI advantage than raw public-web scale, alongside compute.
- The sector may shift from a single scaling playbook toward a portfolio of specialized models and post-training or inference improvements; whether that outperforms continued general-model scaling remains unresolved.
The trend: AI development is moving from public-web data abundance toward competition for high-quality, rights-cleared data and methods that extract more value from it.