The Arc Prize Foundation says its new ARC-AGI-2 test stumps most AI models; humans get 60% of the questions right but GPT-4.5 and Claude 3.7 Sonnet score ~1%
[image] François Chollet / @fchollet : Unlike ARC-AGI-1, this new version is not easily brute-forced. Current top AI approaches score 0-4%. All base LLMs (GPT-4.5, Claude 3.7 Sonnet, Gemini 2, etc.) score 0%. Single-CoT reasoning models (Claude Thinking, R1, o3-mini...) score 0-1%. So you can't solve these tasks via memorization alone. You need the ability to recombine concepts on the fly - you need test-time adaptation... Peter Wildeford / @peterwildeford : Very interesting - the new ARC prize finds tasks where even advanced reasoning models don't do anywhere close to human level. [image]
Context & Ripple Effects
ARC-AGI-2 arrives after ARC-AGI-1 produced sharply different results: o3 reached 87.5% on its semi-private evaluation after training on the public set, while GPT-4o scored 5%, highlighting how benchmark exposure and compute configuration can shape apparent progress. The new test is positioned to make that route less effective by requiring novel concept recombination.
The result also sharpens the distinction between broad model comparisons and capability-specific evaluation. Later coverage framed the ARC exam as a measure of the gap between memorized answers and fluid intelligence, and the foundation subsequently extended the approach with ARC-AGI-3's interactive, game-like tasks.
First-order effects
- ARC-AGI-2 gives GPT-4.5, Claude 3.7 Sonnet and Gemini 2 a conspicuously weak result on a benchmark designed around on-the-fly adaptation: the foundation reports near-zero to low-single-digit scores versus roughly 60% for humans.
- Single-chain-of-thought approaches also score around 0–1%, limiting the case that simply adding an explicit reasoning trace closes this particular capability gap.
Second-order effects
- Labs and evaluators gain a stronger reason to separate performance on knowledge-heavy or familiar test formats from performance on novel task adaptation; a prior high-compute o3 ARC-AGI-1 result makes that distinction especially salient.
- Benchmark competition may shift toward methods that adapt at inference time rather than models optimized chiefly for static training-data recall, while vendors will need to explain why weak ARC-AGI-2 scores do or do not map to their practical products.
Third-order effects
- If successor tests continue to resist memorization and brute-force search, frontier-model progress will be judged increasingly on the efficiency and reliability of test-time adaptation, not just aggregate benchmark rankings.
- That would make evaluation design a more consequential competitive input: benchmarks that distinguish generalization from exposure can reshape which research claims customers and investors treat as evidence of reasoning gains.
The trend: AI evaluation is moving from broad scoreboards toward tests intended to isolate novel reasoning and adaptation from memorization and benchmark-specific optimization.