A study claims GPT-4's outputs became worse between March and June 2023, but some experts are not convinced and urge OpenAI to be less opaque with its AI models
Either way, experts think OpenAI should be less opaque about its AI model architecture. — On Tuesday, researchers …
Ars TechnicaBenj Edwards
Context & Ripple Effects
The disputed finding arrives soon after criticism that OpenAI withheld GPT-4’s training-data and methodology details, limiting outsiders’ ability to explain or reproduce observed behavior changes. That opacity sits alongside OpenAI’s decision to permit an external GPT-4 risk assessment by the Alignment Research Center.
The immediate significance is not that degradation has been proven—experts contest the study—but that users lack a clear way to distinguish model drift, evaluation variance, or deliberate updates. Later feedback about GPT-4 appearing “lazier” to users underscores how recurring behavior concerns can persist without clear change documentation.
First-order effects
GPT-4 users and developers must treat the study as a contested signal, not a definitive benchmark result; reproducibility and version-specific testing become more important for deployments relying on consistent outputs.
OpenAI faces renewed pressure to explain model architecture and changes more clearly, because outside researchers cannot readily audit why outputs differ over time.
Second-order effects
Enterprise customers and evaluators are pushed toward maintaining their own regression tests and acceptance thresholds rather than assuming a model name guarantees stable behavior.
Competing model providers can differentiate through clearer versioning, evaluation disclosures, and change notices as opacity becomes a procurement concern.
Third-order effects
If behavior-changing updates remain difficult to inspect, operational AI assurance will shift from one-time model selection toward continuous monitoring, documentation, and rollback-ready deployment practices.
The episode points toward AI governance norms in which providers are judged not only on headline capability but also on whether customers can trace, test, and manage model changes.
The trend: Foundation-model competition is evolving from static benchmark claims toward operational accountability for how models change in production.
Great reporting from @benjedwards on where we're at now with understanding GPT-4 perfomance “How are we meant to build dependable software on top of a platform that changes in completely undocumented and mysterious ways every few months?- simon willison (1/2) — https://arstechn…
No, we haven't made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one. Current hypothesis: When you use it more heavily, you start noticing issues you didn't see before.
GPT-4 is getting worse over time, not better. Many people have reported noticing a significant degradation in the quality of the model responses, but so far, it was all anecdotal. But now we know. At least one study shows how the June version of GPT-4 is objectively worse than...…
We evaluated #ChatGPT's behavior over time and found substantial diffs in its responses to the *same questions* between the June version of GPT4 and GPT3.5 and the March versions. The newer versions got worse on some tasks. w/ Lingjiao Chen @matei_zaharia https://arxiv.org/... [i…
Code generation: the change they report is that the newer GPT-4 adds non-code text to its output. They don't evaluate the correctness of the code (strange). They merely check if the code is directly executable. So the newer model's attempt to be more helpful counted against it.
We dug into a paper that's been misinterpreted as saying GPT-4 has gotten worse. The paper shows behavior change, not capability decrease. And there's a problem with the evaluation—on 1 task, we think the authors mistook mimicry for reasoning. w/ @sayashk https://www.aisnakeoil.c…
I encourage the folks at @OpenAI to publish better release notes between their versions. Whether #ChatGPT is getting better or worse, developers need dependable services to build their applications on https://arstechnica.com/...
Is ChatGPT's Behavior Changing over Time? https://arxiv.org/... // Arxiv strikes again. These fake papers need to go away. Enough already with pseudo-academic work.
Great reporting from @benjedwards on where we're at now with understanding GPT-4 perfomance “How are we meant to build dependable software on top of a platform that changes in completely undocumented and mysterious ways every few months?”-@simonw (1/2) https://arstechnica.com/...
We do think the paper is a valuable reminder of the unintentional and unexpected side effects of fine tuning. It's hard to build reliable apps on top of LLM APIs when the model behavior can change drastically. This seems like a big unsolved MLOps challenge.
Is ChatGPT really getting worse? Nah, I say that we're just super bad at evaluating and comparing LLMs. And when it comes to closed-source LLMs, we might as well be comparing 🦝 and 🐿️. Great reporting by @benjedwards: https://arstechnica.com/... [image]
My hypothesis is that more RLHF and fine-tuning destroys the calibration of pretrained models. It is a shallow fix that encourages the models to convince the user that they are following instructions, rather than actually following them.
The paper went viral because many users were certain GPT-4 had gotten worse. They viewed OpenAI's denials as gaslighting. Others thought these people were imagining it. We suggest a 3rd possibility: performance did degrade—w.r.t those users' carefully honed prompting strategies. …
I've also noticed that Chat GPT is getting worse, and Bard is getting better. Moreover, Bard shows the sources of figures when asked, which is very helpful to determine whether it's a reliable response
Demonstration that ChatGPT has indeed gotten worse in some tasks over time. For example the success rate of GPT-4 in determining whether 17077 is a prime number went from 97.6% (March) to 2.4% (June). This is not a comprehensive study, but a proof that a decline exists.