OpenAI introduces a framework to evaluate chain-of-thought monitorability and a suite of 13 evaluations designed to measure the monitorability of an AI system
OpenAI
Context & Ripple Effects
OpenAI has been building a more formal evaluation stack since it open-sourced its Evals framework, then added a public safety-evaluations hub for model behavior such as jailbreak resistance and hallucinations. This release extends that work from output testing toward whether a model’s reasoning process can be meaningfully inspected.
The focus is consequential because researchers had already urged more work on chain-of-thought monitorability, while separate reporting found cases where stated reasoning and chatbot answers diverged. The new evaluations create a common way to test that gap rather than treating visible reasoning as self-validating.
First-order effects
OpenAI gains a defined framework and 13 tests for assessing whether an AI system’s chain of thought can be monitored, making this property an explicit evaluation target.
Developers and safety teams evaluating OpenAI systems have a more structured basis for distinguishing readable reasoning from reasoning that is actually useful for oversight.
Second-order effects
Competing model developers face pressure to show comparable evidence that their reasoning traces support monitoring, not merely user-facing explanation.
Safety evaluators can begin treating monitorability as a separate dimension alongside harmful-output, jailbreak, and hallucination testing, building on OpenAI’s earlier public safety scorecards.
Third-order effects
If adopted across the field, monitorability benchmarks could shift AI assurance from judging outputs alone toward testing whether oversight mechanisms retain visibility into model behavior.
The approach also makes a key limitation easier to surface: chain-of-thought transparency is not equivalent to faithfulness, so deployment governance may increasingly require multiple, independently validated safety signals.
The trend:AI safety evaluation is moving from broad model-performance tests toward auditable, standardized evidence about whether increasingly capable systems can be supervised.
We introduce 3 eval archetypes, a metric, and a broad suite of 13 evals. Example: Can we detect solely from the CoT whether a model: - Reward hacks by changing unit tests? - Acts sycophantic when we give personalized memory? - Uses a particular math theorem? [image]
One thing I really like about this CoT monitorability is the reframing that CoT isn't a truth oracle, but more like a control interface. The point isn't that CoT faithfully reflects how the model really reasons, but that it gives us signals we can observe, poke at, and use to
progress on monitoring AI agents by looking at their chains of thought was bottlenecked by lack of measures of monitorability we could *really* trust. this paper fills this hole and might be one of the most important pieces of AI safety research in 2025. [image]
Incredible paper. Quick prompt engineering findings: Models are becoming more capable and autonomous, so we're losing the ability to directly supervise every decision. Prompt engineering is not just about the initial instruction, but about the iterative extraction of a model's [i…
New @OpenAI research: How can we scale supervision of increasingly capable models? Can we rely on monitoring GPT-7's chain-of-thought? We develop a new metric for monitorability and study its scaling trends, coming away with cautious optimism. 🧵: [image]
We evaluate frontier models and find monitorability scales well with more thinking tokens. GPT-5 is the most monitorable model we studied. And monitoring the CoT is much better than just actions! [image]
it sort of feels like a research dead end to rely on a training procedure + monitoring scheme that's based on a lie ("your CoT is a safe space and you won't be punished for it"). there is a limit to how smart and aware a model can be without seeing the obvious contradiction