OpenAI built the gpt2-chatbot, renamed to “im-also-a-good-gpt-chatbot”, per the gpt2-chatbot's 429 rate limit error message, which appeared in the LMSYS arena
gpt2-chatbot confirmed as OpenAI (via) The mysterious gpt2-chatbot model that showed up in the LMSYS arena a few days ago …
Simon Willison's WeblogSimon Willison
Context & Ripple Effects
The model first drew attention as an unidentified entrant in the LMSYS Arena, where its capabilities invited comparisons with leading systems. This error-message attribution turns that mysterious Arena entrant into an identifiable OpenAI test.
The setting matters because Chatbot Arena had recently registered a leadership change when Claude 3 Opus moved ahead of GPT-4. Anonymous evaluation offers OpenAI a way to gather comparative signals before attaching its brand to a model.
First-order effects
OpenAI is now directly associated with gpt2-chatbot through the rate-limit message, ending the central uncertainty around its operator for Arena participants.
The renamed endpoint signals that access to the model is being actively managed, with users encountering OpenAI-style rate limiting rather than a stable, fully announced product interface.
Second-order effects
Arena comparisons involving gpt2-chatbot can now be interpreted as feedback on an OpenAI model, sharpening scrutiny of how it performs against GPT-4 and rival systems.
Competitors using public leaderboards face greater pressure to decide whether anonymous or preview-model entries are useful for collecting unbiased comparative feedback before launch.
Third-order effects
If major model providers increasingly use semi-anonymous benchmark deployments, public leaderboards may become both evaluation venues and pre-release testing channels, complicating clean model-to-model comparisons.
The pattern reinforces a shift toward staged model launches: controlled access and external feedback can precede formal naming, product packaging, and broad availability.
The trend: Frontier AI vendors are increasingly separating model evaluation from formal product launches, using limited-access public testing to collect market and benchmark signals.
Get the Reddit app — Scan this QR code to download the app now — Go to singularity
@sama
Sam Altman on x
im-a-good-gpt2-chatbot
@nanulled
Nano on x
@corbtt ... [image]
@minchoi
Min Choi on x
Whoa the new gpt2-chatbot just created Flappy Bird clone in one-shot 🤯 And it was a dead simple prompt. 🧵👇 [video]
@abacaj
Anton on x
So “gpt2” seems to be coming from OpenAI lol. They should probably not return exact errors from providers
@dimitrispapail
Dimitris Papailiopoulos on x
im-a-good is slightly better than gpt-4-turbo, but im-also-a-good is not that great
@dimitrispapail
Dimitris Papailiopoulos on x
i've seen enough, im-also-a-good-gpt2-chatbot is the sonnet version of im-a-good-gpt2-chatbot
@bindureddy
Bindu Reddy on x
The gpt-2 chatbots are back and all indications are these are indeed the latest GPT-5 versions They seem to be superior to Opus and GPT-4, but still quite underwhelming compared to the insane hype about GPT-5 It will be hilarious if Gemini 2.0 and Llama-3 400b beat it!
@emollick
Ethan Mollick on x
For those of you not following the drama, gpt2-chatbot, which mysteriously appeared on an AI leaderboard site, then disappeared, has now reappeared, is definitely from OpenAI, and seems to be quite good, not sure how good Whether this is a preview release of GPT-4.5/5 is unknown
@dimitrispapail
Dimitris Papailiopoulos on x
and for the grand finale gpt4-turbo vs gpt2-chatbot, tikz draw yourself. [image]
@emollick
Ethan Mollick on x
I was randomly assigned the mysterious OpenAI im-a-good-gpt2-chatbot in the LLM Arena, so I naturally asked it how a superhero would make fudge and to make an ASCII control panel for a time machine. Then it timed out before I could do anything serious. Very good answers, though. …
@rowancheung
Rowan Cheung on x
gpt2-chatbot is back. Capabilities seem to exceed GPT-4, Gemini 1.5, Claude, and anything else currently available. The only way to access it is by testing a prompt in battle mode on the Chatbot Arena and getting a lucky draw. A quick tutorial: 1. Go to chat. lmsys. org 2.... [vi…
@literallydenis
Denis Shiryaev on x
I was skeptical about the GPT2 chatbot, but it is undoubtedly more capable than opensource models and, in some cases, better than GPT4-turbo But it is not better than Opus in my experience - curious to know what is behind it. Also, about the gpt2-chatbot: It does not have a... [v…
@burkov
Andriy Burkov on x
If GPT-5 was any good they would not release it cowardly hiding behind gpt2-chatbot. This A/B testing using @lmsysorg is pathetic.
@emollick
Ethan Mollick on x
Even after only a dozen uses, it is clear im-a-good-gpt2-chatbot is full of ghosts. I mean this in the same way that GPT-4 and Claude 3 Opus and Gemini 1.5 are full of ghosts/sparks/whatever - they are occasionally uncanny. Seems to be an emergent feature of frontier LLMs. [image…
@skirano
Pietro Schirano on x
im-a-good-gpt2-chatbot it's so good that it created a code interpreter that uses Claude Opus for me. Excuse me as I faint in ontological shock. [video]
@abacaj
Anton on x
what are these models? they don't seem like the next iteration of gpt-n
@abacaj
Anton on x
If I had to take a wild guess it's just the phi-3 variants that aren't released yet
@abacaj
Anton on x
im-a-good-gpt2-chatbot will *gladly* hallucinate information where the current gpt-4-turbo *does* not, left “gpt2” right gpt-4-turbo [image]