OpenAI may have tested GPT-4 on its training data, violating the cardinal rule of ML, and GPT-4's exam performance says little about its real-world usefulness
OpenAI may have tested on the training data. Besides, human benchmarks are meaningless for bots.
AI Snake Oil
Context & Ripple Effects
At launch, OpenAI leaned heavily on GPT-4's bar-exam and standardized-test scores as proof of capability — the precision and image-input gains it announced alongside them set the public narrative. But the same week, experts were already faulting the company for refusing to disclose its training data or methods, which is exactly what makes a possible train-on-the-test-set violation hard for outsiders to rule out.
This critique lands in an unusual window: OpenAI simultaneously commissioned an external risk assessment by the Alignment Research Center, showing it accepts outside scrutiny selectively while keeping the evaluation-relevant details closed. The claim that exam scores say little about real-world usefulness also foreshadows the benchmark-skepticism thread that resurfaces in later coverage of GPT-4.5's coding-benchmark performance.
First-order effects
Buyers and researchers relying on OpenAI's published benchmark tables lose their main yardstick: if test items leaked into training data, GPT-4's exam numbers overstate capability, and only OpenAI can check because it withheld dataset documentation.
Second-order effects
Rival labs face pressure to publish cleaner data-provenance disclosures to differentiate themselves from OpenAI's opacity, and independent evaluators like the Alignment Research Center become more valuable as the credible alternative to vendor-scored tests.
Third-order effects
If vendor-reported benchmarks keep proving unreliable, procurement and regulation will drift toward third-party evaluation regimes — operational AI assurance rather than marketing scorecards becomes the basis for trusting frontier models.
The trend: Frontier-model credibility is shifting from self-reported exam benchmarks toward independent assurance, as disclosure gaps at OpenAI erode trust in vendor-scored evaluations.
OpenAI may have tested GPT-4 on the training data: we found slam-dunk evidence that it memorizes coding problems that it's seen. Besides, exams don't tell us about real-world utility: It's not like a lawyer's job is to answer bar exam questions all day. https://aisnakeoil.substac…
GPT-4 memorizes coding problems in its training set. How do we know? @random_walker and I prompted it with a Codeforces problem title. It outputs the exact URL for the competition, which strongly suggests memorization. https://twitter.com/...
Interesting thread. I agree that one of the challenges we face is distinguishing actual intelligence from learned knowledge and “skills”. I think one of the more intelligent things I've observed in LLMs is in-context learning, where arguably the learned skill *is* learning. https…
I so appreciate this piece from @random_walker and @sayashk. A lot of things have been buzzing 🐝 around in my head since the launch of GPT-4: There were signs of data contamination. This covers a ton of it. What contamination is, how to find it, why it matters. https://twitter.co…
great article, highlights many flaws in GPT-4 (and other LLM) evals and their automation fears “OpenAI's lack of transparency, we can't answer the contamination question with certainty. But what's certain is that OpenAI's method to detect contamination is superficial and sloppy” …
To be honest, i think it's only valid to properly evaluate GPT-4 on material that was created after the training was finished. Anything that existed before, should just be assumed to be part of the training data, because no matter how hard you try, it's going to leak somehow. htt…
Phenomenal piece by @random_walker, with an extremely interesting anecdote about automation: Just one of the 270 jobs in the 1950 census has been eliminated by automation: elevator operator. https://twitter.com/...
Perhaps unsurprising — at least if you're ever made a serious attempt at probing the system with *novel* questions The distinction between intelligence and knowledge will keep getting more and more relevant over time https://twitter.com/...