Ilya Sutskever says “we're back in the age of wonder and discovery” as AI companies focus on pre-training, inference improvements, and finding the “next thing”
There's been a lot of chatter about the “end of LLMs” or “LLMs starting to fail.” Jon Keegan / Sherwood News : OpenAI models may be starting to plateau Siddharth Jindal / Analytics India Magazine : OpenAI is So Doomed if Inference Time Scaling for o1 Fails Bluesky: Jan Jęcz / @jeczjan.bsky.social : Investors who poured 1 bln $ into his startup in the seed round must be thrilled to hear that they financed “wonder and discovery” [embedded post] X: @modestproposal1 : “This shift will move us from a world of massive pre-training clusters toward inference clouds, which are distributed, cloud-based servers for inference,” Sonya Huang, a partner at Sequoia Capital, told Reuters. https://www.reuters.com/... @modestproposal1 : “Ilya Sutskever, co-founder of AI labs SSI and OpenAI, told Reuters results from scaling up pre-training - the phase of training an AI model that uses a vast amount of unlabeled data to understand language patterns and structures - have plateaued” https://www.reuters.com/... Krystal Hu / @readkrystalhu : As we heard more about the failed training runs in major AI labs, @annatonger & I wrote about the question on everyone's mind—It's been two years since GPT4. What's after the pre-training scaling law? Spoiler: Ilya has something to say about it Link: https://www.reuters.com/... [image] Forums: r/aiwars : OpenAI and others seek new path to smarter AI as current methods hit limitations r/mlscaling : OpenAI and others seek new path to smarter AI as current methods hit limitations Msmash / Slashdot : OpenAI and Others Seek New Path To Smarter AI as Current Methods Hit Limitations
Context & Ripple Effects
The reported plateau in pre-training returns lands amid mounting constraints on the input side: related coverage says AI companies may have to move beyond broad general-purpose models as conventional web data becomes scarcer the supply of conventional web training data tightens.
The significance is not that model development stops, but that the locus of progress may shift. The article frames inference optimization and new research directions as increasingly important alongside the large training runs that defined the prior cycle.
First-order effects
- AI labs must place more weight on improving inference and on research bets beyond simply enlarging pre-training runs, after failed runs and weaker scaling gains raise the cost of repeating the old approach.
- Sutskever’s comments reinforce investor and lab scrutiny of whether current training spend produces enough incremental capability to justify it.
Second-order effects
- Compute planning shifts toward distributed inference capacity as well as training clusters, making serving efficiency a more central competitive variable.
- If broad-model gains slow while usable training data is constrained, teams have stronger incentives to pursue specialized models and alternative data strategies, though synthetic-data quality remains an open risk synthetic-data degradation concerns.
Third-order effects
- AI competition could become less defined by the largest single pre-training run and more by a combination of research breakthroughs, data access, and the economics of operating models at inference.
- That would deepen the industry’s shift toward inference as durable infrastructure: lower unit prices need not lower customer costs when more capable reasoning systems consume more tokens reasoning-model token use raises developer costs.
The trend: This is one data point in AI’s transition from pre-training scale as the primary frontier to a more balanced race across data, post-training research, and inference economics.