While OpenAI has become more upfront about ChatGPT's limitations, the company should do more to make clear the bot can't reliably distinguish fact from fiction
“May occasionally generate incorrect information.” — This is the warning OpenAI pins to the homepage of its AI chatbot ChatGPT …
The VergeJames Vincent
Context & Ripple Effects
When OpenAI first showed off ChatGPT in its December 2022 demo, the pitch leaned on the bot admitting mistakes and challenging incorrect premises; within two months, researchers publishing in Nature were already flagging apprehension about chatbot errors on technical topics. Today's piece lands on the gap between those two facts: OpenAI's homepage disclaimer that ChatGPT "may occasionally generate incorrect information" frames fabrication as an edge case, when the deeper problem is that the model cannot reliably tell fact from fiction in the first place.
First-order effects
Users who read the one-line warning as license to trust ChatGPT elsewhere are left without an accurate mental model of when outputs are fabricated versus verified — precisely the confusion researchers flagged on technical subjects.
Second-order effects
OpenAI's own follow-on work concedes the scale of the problem internally: it built CriticGPT, a separate GPT-4 fine-tune whose job is helping human trainers spot errors in ChatGPT's output — an admission that catching falsehoods requires dedicated tooling, not user vigilance.
The critique also cuts against OpenAI's direction of travel: in February 2025 the company removed warning messages from ChatGPT to cut down on denials, showing the product organization optimizing away exactly the kind of friction critics say is too thin already.
Third-order effects
If a one-line homepage disclaimer remains the industry standard for accuracy labeling, the burden of verification shifts structurally onto downstream users and publishers — pushing regulators and platform reviewers toward demanding explicit confidence or provenance signals rather than blanket caveats.
The trend: Consumer chatbots are drifting toward frictionless presentation of unverified output, widening the authority gap between what these agents confidently state and what they can actually verify.
here's my take on last week's lawyer-cites-fake-lawsuits-generated- by-ChatGPT debacle: OpenAI needs to do more to inform users about ChatGPT's limitations https://www.theverge.com/... [image]
I posted a mockup of a design change for ChatGPT that I think could help address the risk of people being lead astray by its incredible ability to invent faleshoods: ChatGPT should include inline tips https://simonwillison.net/... [image]
Because people want to either disrupt the Google's or see the search interface as the UX users spend most time in, we're creating a dangerous blurring of intent and outcome ... https://twitter.com/... [image]
I've been making a lot of TikTok/Instagram videos about the limitations of ChatGPT. When I've talked about misinformation, the number of people who INSIST that “everyone knows that!” would make you roll your eyes out of your head. https://twitter.com/...
“Why, for example, can ChatGPT simply not recognize when it's being asked to generate factual citations and caution the user to “check my sources”?” https://www.theverge.com/...
Another good from @jjvincent. 2 important points: (1/2ish) - These systems are being used as search, whether or not it's what OpenAI intended. By recognizing the *use* (both intended & unintended but foreseeable), companies can do much more to situate their products responsibly. …
This is understandably less juicy than the apocalypse warnings, but I think it's a more realistic assessment of the problem right now https://www.theverge.com/...
Using ChatGPT and other AI bots without understanding what they are and are not capable of will get your fired (or maybe disbarred, idk https://www.theverge.com/... ) https://twitter.com/...