OpenAI says using its Responses API harness with GPT-5.6 Sol tripled its ARC-AGI-3 score with fewer output tokens, after Sol with the official harness got 7.8%
A sped-up video of GPT-5.6 Sol attempting to solve puzzles in the ARC-AGI-3 benchmark, with the official harness (left) …
OpenAI
Context & Ripple Effects
ARC-style evaluation has remained a difficult test in this coverage: the Arc Prize Foundation said ARC-AGI-2 stumped most leading models, while OpenAI has increasingly emphasized benchmarks alongside practical reasoning work.
This result extends OpenAI’s recent focus on longer-context, agentic tasks—areas highlighted in its GPT-5.5 capability update—by making the runtime harness, not just the base model, central to the reported outcome.
First-order effects
OpenAI’s Responses API harness becomes a materially stronger evaluation path for GPT-5.6 Sol on ARC-AGI-3 than the official harness, whose reported score was 7.8%.
The reported reduction in output tokens means the higher score is paired with lower inference usage, improving the result’s efficiency as well as its headline performance.
Second-order effects
Benchmark comparisons around GPT-5.6 Sol will need to distinguish model capability from harness design; competing developers face pressure to disclose comparable orchestration and token-use details.
For API customers building multi-step reasoning systems, the result strengthens the case for using OpenAI’s managed runtime rather than treating the model endpoint as the full product surface.
Third-order effects
If similar gaps persist, frontier-model benchmarking will increasingly measure model-and-runtime systems rather than standalone models, making benchmark leadership less portable across APIs and toolchains.
Reasoning competition may shift toward efficiency-adjusted outcomes—capability per token and per orchestration setup—rather than raw scores alone, though this single vendor-reported comparison cannot establish that shift by itself.
The trend: AI reasoning is becoming a systems-level contest in which the orchestration layer can improve both benchmark performance and the cost of reaching it.
Turns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason and work over multiple context windows with the help of our canonical compaction implementation. https://openai.com/...
GPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what it had learned. We found that enabling two API settings tripled our scores with …
We implemented the harness with the Responses API and turned on: → Retained reasoning → Context compaction On the public set, GPT-5.6 Sol's score rose 188% while using 6x fewer output tokens. [image]
Model + harness. We have barely begun to understand the best ways to do harness engineering. A huge amount of untapped potential even without models getting better (but models are getting better)
Terrific work by @ilanbigio and @sandersted on the investigation and post. Seems like it was a lot of fun. Take care of your harnesses out there, your model will thank you.
That's pretty interesting ARC-AGI-3 performance is bottlenecked by context management rather than raw reasoning so letting the model retain its reasoning across context windows tripled the score Not sure if it's “actually SoTA” since other models were measured by different rules …
I followed this work internally and found it super interesting. The TL;DR is that if you allow GPT-5.6 Sol to retain its reasoning across turns + perform compaction, it scores ~3x higher on the ARC-AGI-3 public set. https://openai.com/... [image]