Sources: OpenAI discussed training GPT-5 on public YouTube video transcripts; AI industry's need for high-quality text data may outstrip supply within two years

Firms such as OpenAI and Anthropic are working to find enough information to train next-generation artificial-intelligence models

Wall Street Journal 2024-04-01 Deepa Seetharaman

Context & Ripple Effects

Coverage had already tied GPT-5 to a prospective near-term release, with some enterprise customers reportedly seeing demos of the model. This report makes training-data availability—not only model design—a visible constraint on that roadmap.

The discussion of YouTube transcripts sits within a broader search by OpenAI and Anthropic for usable training material. It also foreshadows later reporting that OpenAI had used Whisper to transcribe more than a million hours of YouTube video for GPT-4 training.

First-order effects

OpenAI’s reported consideration of public YouTube transcripts expands the set of text sources under review for GPT-5 training, while underscoring that high-quality text is becoming a practical input constraint.
Anthropic and other frontier-model developers face the same immediate procurement problem: finding sufficiently useful data as they prepare successive model generations.

Second-order effects

Large platforms and content owners gain greater leverage over valuable text and transcript archives as model developers look beyond conventional web datasets.
Data acquisition, transcription, filtering and rights assessment become more important competitive capabilities alongside compute; the later report of YouTube-video transcription for GPT-4 training illustrates that pipeline’s strategic value.

Third-order effects

If high-quality public text becomes scarce, frontier AI development is likely to depend more on controlled data access, licensing arrangements and proprietary data pipelines, concentrating advantage among firms that can secure them.
The trade-off between enlarging training corpora and preserving data quality will intensify interest in synthetic data, though its usefulness depends on avoiding degradation from training repeatedly on model-generated material.

The trend: Frontier AI is turning high-quality training data from an abundant web resource into a strategic infrastructure bottleneck.

Discussion

@robtyrie Rob on threads
I don't know why this keeps repeating.. this linear math and assumption that the amount of data we have recorded digitally is fixed somehow. Some of the machines that we're building just create data extremely large amounts.. it's highly organized data based on the natural world …
@dseetharaman Deepa Seetharaman on threads
Taking a quick break from a family trip to share my latest: it's generally understood that more data = better AI models. But what happens when you've exhausted the internet? We're rapidly getting to that point of data drought. …
@alex @alex on x
set up a simple way for me to sell you access to my writings and i will contribute to the great AI maw
@davidclinchnews David Clinch on x
This is throwing another “s” word against the wall and hoping it will stick. There is only one path forward and that is proper licensing of authoritative content and data. #MediaRevenue
@carnage4life Dare Obasanjo on x
A limiting factor of LLM-based artificial intelligence is training data. There's a real challenge that there simply isn't enough openly available data in certain fields to effectively train a generative AI. Generating games using LLMs is a simple example of where it's a problem. …
@jeffjarvis @jeffjarvis on x
As journalism retreats from AI behind walls, the real question we need to address is what is missing from the web: the voices of people who did not have the power to publish in history... For Data-Guzzling AI Companies, the Internet Is Too Small https://www.wsj.com/...

Chronicles