Artificial Analysis put 60% of Intelligence Index v5 on private test sets or held-out answers. Model makers can cite its rankings in public; buyers cannot independently rerun much of the test.

Key takeaways

  • Artificial Analysis launched Intelligence Index v5 on October 28, 2026, adding coding and science benchmarks.
  • FrontierCode 2.0 uses 132 private tasks in real open-source repositories and gives a submission zero when a blocking check fails.
  • Mistral said Large 4 used 82,000 output tokens per coding task, 39% fewer than its preview version.
  • Artificial Analysis said its evaluation of OpenAI’s o1 cost $2,767.05.
  • The Cyber Index Alliance launched with Collinear, IBM, Nvidia and Vercel on September 29, 2026.

OpenAI released Evals in 2023 so users could report shortcomings in its models. Later that year, AWS introduced Bedrock Model Evaluation in preview, with human testers to help companies assess models before using them. A score helped people narrow a crowded field; testing a prospective deployment remained a separate job.

A rank now helps define the market it measures

Artificial Analysis built its Intelligence Index as a composite of capabilities across tasks meant to reflect real-world use. Its fifth version added coding and science benchmarks. Choosing those tasks does more than sort models: it determines which improvements can move a vendor up the table, and therefore which improvements a vendor has reason to emphasize in a release.

Mistral, for example, positioned Large 4 against other open models using Artificial Analysis’s separate Coding Agent Index. Its case also included token efficiency: 82,000 output tokens per task, 39% fewer than the preview version. An agent that completes a task with fewer tokens can occupy a different place in a buyer’s budget even when its capability score looks similar. The coding index and the general Intelligence Index answer related, but not identical, questions.

Artificial Analysis made that distinction visible in its Gemini 4 Hydrogen comparison. Hydrogen nearly matched Claude Sonnet 5.5 and GPT-6.1 Sol on the Intelligence Index, while costing about one-third as much as Sonnet and roughly 2.5 times as much as Sol. A buyer choosing between the three would still need to know which tasks mattered, but the comparison made the price of a near-tie impossible to ignore.

A model vendor benefits when an outside evaluator supplies the ranking; the claim carries more weight than the vendor’s own test. The evaluator, in turn, decides which tasks enter the composite and how capability sits beside price and token use. The measuring tape has acquired a say in what the market calls progress.

Public tests lost discrimination before private tests gained authority

In 2024, Meta, OpenAI, Microsoft, and other firms developed internal benchmarks as models approached or exceeded 90% accuracy on established public tests. Once many contenders pass the same exam, another point tells a buyer less about the work a model can actually do. Repeated exposure also gives developers opportunities to train against a test, whether directly or through the broader circulation of its questions and answers.

Artificial Analysis responded in Intelligence Index v5 by basing 60% of the composite on private test sets or held-out answers—the first time a majority had that status. Its FrontierCode 2.0 component uses 132 private tasks in real open-source repositories. It asks whether a maintainer would merge an agent’s change and assigns zero if a blocking check fails. That rule tests the submitted change against an operational threshold, not merely whether the agent produced plausible code.

Other evaluators have made related choices without arriving at one standard design. Scale AI used private datasets for its SEAL Leaderboards. Google piloted double-blind external evaluations in a cryptographic box, seeking to protect tests from contamination while protecting model intellectual property. Both approaches restrict what participants can see, but they assign custody and access differently.

Secrecy has a cost. An outsider cannot fully reproduce a result built on withheld tasks or answers; that outsider must instead assess how the evaluator designed, protected, ran, and scored the test. The Index placed the leader three points above the leading unreleased open-weights model in v4.3 and 12 points above it in v5. A revised measurement regime can change a gap as well as reveal one. Buyers lack comparable v4.3 component weights in the cited coverage to separate those effects.

The hosting address belongs beside the model name

Artificial Analysis found inconsistent performance across hosting providers for gpt-oss-120b and other open-weight models. A buyer cannot carry a result from one provider to another merely because both offer the same weights. The deployed service also has an API version, inference settings, latency, and a bill. Artificial Analysis’s Gemini comparison shows why cost belongs in the same discussion as capability; the hosting result shows why even a carefully measured capability number needs an address.

For enterprise AI adoption, buyers have to record the conditions under which they call a model and test the work they expect it to perform. They should track cost per useful task, rather than token price alone: retries, long outputs, and failed work can change the comparison. AWS’s human-assisted Bedrock evaluation addressed part of this problem at the point of use. It did not make a general leaderboard unnecessary; it gave the buyer a different test to run after consulting one.

Artificial Analysis took another step toward task-specific evaluation with the Cyber Index Alliance. Collinear, IBM, Nvidia, and Vercel joined it to assess whether AI agents could find and remediate vulnerabilities. Unlike a general intelligence ranking, the Alliance must test whether an agent repairs software: a convincing vulnerability description does not, and a proposed fix must survive checks in the system it changes. The partners bring relevant operating knowledge. Their participation also makes it important to know who designed the tasks, who supplied access, and who could influence publication of the results.

The evaluator has to make confidential evidence interrogable

Artificial Analysis said evaluating OpenAI’s o1 cost $2,767.05. More expensive reasoning runs make it harder for independent groups to repeat every comparison, especially when a score depends on many tasks and settings. A survey of 400 algorithms presented at major conferences found that only 6% included code and 30% included test data. That survey concerns research replication, not a particular benchmark provider, but it shows how readily a result can circulate without the materials needed to check it.

Stanford’s Center for Research on Foundation Models treated disclosure as something to measure in its Foundation Model Transparency Index. It assessed 10 developers against 100 indicators; Llama 2 scored 54%, GPT-4 48%, and PaLM 2 40%. Stanford evaluated model developers rather than benchmark custodians. A developer can disclose little about a model while an evaluator discloses little about the test, leaving a reader with two separate gaps to bridge.

OpenAI said in September 2026 that it plans to permit third-party technical safety evaluations during training, evaluation, and deployment. That proposal concerns safety testing, not a guarantee that outsiders can audit every capability index. It nevertheless puts access terms on the table. An evaluator using private tasks can disclose the task-selection procedure, scoring rubric, model version, access conditions, sampling settings, costs, and conflicts without publishing the answer key. An independent reviewer can then examine a controlled evidence trail rather than rely on the rank alone. The narrower question of who may inspect that trail—and what they may say about it—is the problem of evaluation provenance.

No single exam can stand in for a deployment decision

Scale AI’s SEAL Showdown asks contributors in more than 100 countries to vote on everyday model usability. Their judgments expose experiences a private coding task may miss, though popularity and task completion measure different things. MLCommons’ AILuminate uses more than 12,000 prompts to assess LLM safety, giving a specialized standardized test a role that a general capability index cannot fill. OpenAI has also introduced 13 evaluations for chain-of-thought monitorability, a property neither usability votes nor a coding score is designed to capture.

A buyer can use a broad index to select candidates, domain tests to examine consequential failures, and trials on its own provider and settings to measure errors, latency, and cost. Governance documents then establish who ran each test and what could be inspected. No named enterprise buyer in the cited reports rejected a model because it could not audit an evaluator’s methods; calling provenance a universal procurement rule would outrun the record. The narrower evaluation-to-procurement gap is already concrete: none of these tests, alone, specifies how a model will behave in a particular workflow.

Frequently asked questions

Which Intelligence Index v5 components make up the 60% based on private tests or held-out answers?

The piece identifies FrontierCode 2.0 as one such component, but it does not list every v5 component included in the 60% or each component’s individual weight.

Can a reader isolate how much of the v4.3-to-v5 leaderboard-gap change came from model performance rather than revised methodology?

No. The cited coverage gives the three-point v4.3 gap and the 12-point v5 gap, but not comparable v4.3 component weights needed to separate methodology effects from performance effects.

Which hosting providers or inference settings caused the reported differences for gpt-oss-120b?

The piece says Artificial Analysis found inconsistent performance across providers, but it does not name the providers or supply the API versions, sampling settings, latency conditions, or result-by-result differences.

Will OpenAI’s September 2026 third-party safety-evaluation plan give outsiders access to private capability-benchmark answer keys?

Not on the record cited. OpenAI said it plans to permit third-party technical safety evaluations during training, evaluation and deployment; that is not a commitment to make every capability index or its withheld answers auditable.

Quantified signals behind AI evaluation

MeasureReported figureWhat it covers
Intelligence Index v560%Share of the composite based on private test sets or held-out answers
FrontierCode 2.0132 private tasksTasks in real open-source repositories
Research-replication survey400 algorithms; 6% included code; 30% included test dataAlgorithms presented at major conferences
Foundation Model Transparency Index10 developers; 100 indicatorsStanford CRFM’s developer-disclosure assessment
AILuminateMore than 12,000 promptsMLCommons’ LLM safety assessment

The model gets the first-place badge. For 60% of Index v5, Artificial Analysis must make the result credible without showing the answer key.