Meta's Unavailable Maverick Ranked Second on LMArena

In April 2025, a variant of Meta's Llama 4 Maverick reached second place on LMArena's model leaderboard. Meta disclosed that the ranked version was not publicly available.

A perfect public score has a narrow boundary

The ARC Prize Foundation introduced ARC-AGI-3 in March 2026 to test reasoning through novel, video-game-like problems rather than recall of familiar answers. In August, Nvidia said its AVO agent system completed all 183 levels across the benchmark’s 25 public environments. Nvidia reported a result for a complete agent system on the public set—not a general score for an underlying model on unseen environments.

Completing every level does not show that AVO memorized them; a system can make genuine progress on a public test. Nor does a full clear establish how it performs outside those environments. Once AVO has completed every public level, completion rate cannot distinguish it from a stronger system. Evaluators need fresh tasks or another measure, while buyers need to know they are looking at AVO on the public set, on that version of the test.

Launch teams and investors have made rankings valuable

LMArena began as a research project at UC Berkeley and said in April 2025 that it was forming Arena Intelligence to support continued development. The next month, sources reported a $100 million seed round at an approximately $600 million valuation. The reported financing says nothing about revenue or buyer reliance, but it shows what investors thought a model-ranking platform could be worth.

Model vendors give the ranking its distribution. Google presented a 24-point LMArena Elo gain for an upgraded Gemini 2.5 Pro preview alongside coding and other benchmark claims. In November 2025, it cited Gemini 3 Pro’s 1,501 LMArena score in another flagship launch. A public preference test earns that place for a reason: it offers a common comparison of interactive outputs that a fixed answer-key exam may miss.

None of this establishes that a purchasing team selects a model by Elo. Vendors have an incentive to seek a legible rank, and prospective users will encounter that rank before running their own tests. A shared comparison is useful; the identity of the system it compares matters just as much.

Meta’s Maverick episode put model identity into the policy

LMArena said it was updating its leaderboard policies after the Maverick result. The dispute centered on access to the ranked version, not a demonstrated flaw in the score calculation.

A buyer trying to reproduce such a result would need the model identifier and availability status, plus the system prompt, serving configuration, sampling settings, evaluation data, rubric, and test version. Chip Huyen’s AI Engineering makes experiment tracking central to a reliable evaluation pipeline: when data, prompts, rubrics, or sampling change without a record, a later score cannot reliably guide an application decision. The Maverick case adds a procurement question to that log: can the buyer run the ranked system at all?

Meta’s generative-AI chief Ahmad Al-Dahle denied a separate rumor that Llama 4 Maverick and Scout had been trained on test sets. Nothing in the version mismatch substantiates that allegation. The leaderboard-policy question is narrower: how to identify public, preview, and unavailable configurations.

Protected tasks buy freshness at the expense of scrutiny

Evaluators have been separating broad preference from specific work. Scale AI launched private-dataset SEAL rankings for coding, instruction following, and math in 2024. Google introduced a distinct factual-grounding benchmark that year. In October 2026, Artificial Analysis said 60% of its Intelligence Index v5 rested on private test sets or held-out answers; it also added coding and science measures. Its cited FrontierCode 2.0 benchmark uses 132 tasks in real open-source repositories and asks whether a maintainer would merge the change.

Those designs answer different questions. Chat preference can help find a candidate; a repository task can test a consequential coding outcome. Keeping answers private also makes direct optimization against them harder. But a buyer who cannot inspect a private test faces a different uncertainty: whether its tasks resemble the buyer’s work, and whether a changed score reflects a changed model or a changed index. Versioned methods and disclosed task coverage can narrow that uncertainty without publishing every answer.

Specialization costs smaller buyers, too. Coding, grounding, preference, latency, and price do not collapse neatly into one cross-model rank. A common public leaderboard remains a useful starting point; additional tests earn their place when each addresses a decision the common score cannot make.

Bedrock makes evaluation part of the buying workflow

AWS was already moving evaluation toward the point of selection in November 2023, when it introduced Model Evaluation on Bedrock in preview to help companies assess models before use, including with human testers. A company can put its own tasks and human judgments in front of candidate models before choosing one. It must then track the configuration it chose and check whether production error rates, latency, and cost remain acceptable.

People can decide whether an output meets their institution’s standard, not merely whether it matches an answer key. A buyer can use a public rank to find candidates, then test them on protected tasks and its own work. As the evaluation-to-procurement gap makes clear, a model change or prompt revision means rerunning those tests rather than inheriting the old score.

Maverick's second-place rank told buyers where to look. A version they can run still has to earn that place on their own tasks.