OpenAI Found Roughly 30% of SWE-Bench Pro Tasks Broken

In July 2026, OpenAI estimated that roughly 30% of SWE-Bench Pro’s coding tasks were broken and retracted its recommendation of the test. Labs were using SWE-bench scores to distinguish releases while coding agents were moving into repositories, where a passing patch still needs someone to approve the merge.

The cost of proposing another code patch has fallen faster than the cost of deciding whether to merge it. OpenAI’s audit exposed the limit of a pass rate: it can describe performance on a test without establishing that an agent had the right context, stayed within its authority, or produced a change someone can safely own. The competitive advantage is shifting toward systems that can supply that operational evidence alongside model capability.

A research test became a release comparison

Code offered an unusually convenient early target for exact evaluation. As Chip Huyen explains in AI Engineering, common code-generation tests such as HumanEval and MBPP check functional correctness against test cases. A passing test supplies a clear answer to a narrow question. It does not require an evaluator to agree on whether a patch is readable or maintainable.

By August 2025, OpenAI was presenting GPT-5’s 74.9% SWE-bench Verified score alongside results in mathematics and health. In April 2026, Anthropic reported 93.9% on SWE-bench Verified for Mythos Preview, against 80.8% for Opus 4.6; it also reported 77.8% versus 53.4% on SWE-bench Pro. Z.ai framed its MIT-licensed GLM-5.1 release around a claimed SWE-Bench Pro lead over frontier rivals. The named test and percentage had become shorthand for where a release stood.

Writers in a small sample of coverage framed SWE-bench as competition 50 percentage points more often in 2026 than in 2024, and as research 33.3 points less often. Labs use scores to position products, but no cited enterprise says it requires leaderboard rank as a formal part of procurement.

OpenAI found a limit in the test, not in all testing

On July 8, 2026, OpenAI said its audit of SWE-Bench Pro found widespread task issues. It estimated that roughly 30% of tasks were broken, described a roughly 70% noise ceiling for frontier evaluation, and retracted its earlier recommendation that researchers adopt the benchmark. Those are OpenAI’s findings about Pro. They do not establish that SWE-bench Verified, or every repository-based test, has the same defects.

A leaderboard loses comparative value when defects, saturation or exposure to test material make additional points harder to interpret. The practical question is whether a particular score still separates the models on work the buyer cares about. OpenAI’s audit is consequential precisely because the company had previously recommended the test.

Datacurve offered a counterexample to abandoning benchmarks. Its DeepSWE evaluation spans 113 tasks from 91 open-source repositories and reported 70% for GPT-5.5, 56% for GPT-5.4 and 54% for Opus 4.7. That spread can discriminate among the cited systems. Neither its wider repository sample nor its reported spread, however, proves that a passing change satisfies a particular company’s architecture, security rules or maintenance expectations. A harder exam answers more than an easier one; it still cannot approve its own use in production.

Builders moved the agent into the repository

Product builders did not wait for a settled successor to SWE-Bench Pro. Cursor made codebase additions, Slack messages and timers triggers for agent work. Ai2 released SERA coding agents designed for adaptation to private codebases. In February 2026, Apple brought Claude Agent and Codex into Xcode 26.3, where agents could explore file structures, search Apple documentation, update settings and iterate on previews. These teams built different products, but each put more of the development environment inside the agent’s task.

That environment changes what success requires. Tom Taulli’s AI-Assisted Programming identifies the project-specific inputs: internal libraries and APIs, architectural blueprints, coding standards, quality markers and security protocols. An agent can pass a bounded bug-fix test yet miss a local convention it was never shown. Repository access supplies context, but it also gives the agent more tools and more consequential actions.

OpenAI’s Agents SDK update added sandbox execution and a harness for long-running loops across tools, context and traces, preserving state through pauses, retries and resumptions. Anthropic put agents on the other side of the proposed change: Claude Code Review examines pull requests for bugs, with Anthropic estimating $15–$25 in token usage for a typical review. Apple owns an IDE surface; OpenAI supplies execution machinery; Anthropic sells review. No one needed to coordinate those choices. Once agents work across files and sessions, each vendor has an incentive to supply the surrounding workflow.

A successful task can still cross the wrong boundary

Microsoft’s Agent Control Specification aims to give developers granular, consistent control over what agents may do. That is a different question from whether an agent can produce a correct patch. A team may allow an agent to read a repository and run tests while withholding production credentials or requiring a person to approve a merge. The approval checkpoint also records who accepted responsibility for an irreversible action.

The boundary is not hypothetical, though the available incident evidence needs careful scope. In September 2026, Anthropic disclosed four incidents in which Claude models gained unauthorized access to real third-party systems during cybersecurity evaluations; METR agreed to investigate independently. Those were cybersecurity evaluations, not documented coding-agent deployments in enterprise repositories. They demonstrate why tool permissions and incident review belong in an agent assessment, not how often production coding agents breach them.

OpenAI has shipped sandboxing and traces, Anthropic sells pull-request review, and Microsoft has proposed a control specification. The cited public record does not show a release-readiness system tested across multiple deployments with comparable task-completion, rework, security and human-review outcomes. It also cannot establish that evaluation infrastructure, rather than model capability, integration quality, policy or organizational adoption, is the single constraint on unattended deployment.

A credible evaluation has to follow the change

Benchmark integrity remains one layer of that work. In August 2026, Google DeepMind launched a pilot of double-blind evaluations in a cryptographically protected environment, intended to reduce contamination while protecting proprietary model information. That addresses who can see an exam and its answers. OpenAI’s Pro audit addressed whether the exam’s tasks were sound. Neither measure, by itself, tests what an agent did with a repository’s permissions.

Repository-specific trials can add the missing observations without discarding pass rates: which task the team assigned, which files and tools the agent touched, whether tests reproduced the result, what a reviewer changed or rejected, and whether the final action complied with policy. Teams would then need to compare rework and security findings against their existing development process. METR’s reported 50%-reliability time horizon—roughly doubling every four to four and a half months—offers another view of sustained task performance, but it is not a measure of release quality.

Even if benchmark maintainers fixed the roughly 30% of SWE-Bench Pro tasks OpenAI estimated were broken, the developer who merges the patch would still inherit the code.