Google retires the Octane JavaScript benchmark, since developers are making optimizations to boost scores at the expense of real world performance
JavaScript engines have focused too much on the benchmark, to the detriment of the real world. — Google has announced …
Context & Ripple Effects
Google's retirement of Octane is a rare public admission that a benchmark stopped measuring what it was built for: JavaScript engine teams had learned to optimize for the score itself, so gains on Octane no longer translated into faster pages for users. It is a textbook case of benchmark saturation — once an instrument becomes a target, it stops being an instrument.
The decision also fits a longer arc in which Google acts as curator of how web performance is defined and measured. A year later, developers were criticized for optimizing for standards that first ship in Chrome, and by 2021 Google was justifying V8 work through real-world claims like the 23% Chrome 91 speedup from Sparkplug rather than synthetic scores.
First-order effects
- Engine teams at Google and rival browser vendors lose their incentive to chase Octane-specific micro-optimizations, redirecting engineering effort toward workloads that mirror actual sites and apps.
- Browser vendors can no longer use Octane numbers as a marketing yardstick, forcing whoever wants to claim performance leadership to adopt newer, harder-to-game measurements.
Second-order effects
- Benchmark maintainers face pressure to rotate or retire suites before they saturate, since any long-lived public score eventually attracts optimization-for-the-score behavior from engine teams.
- Web developers lose a familiar comparison point across browsers, shifting competitive signaling toward vendor-published real-world results — the mode Google itself adopted with later V8 releases.
Third-order effects
- If the pattern holds, benchmarks become managed instruments with planned lifecycles rather than permanent fixtures, and the organization that controls the measurement — here Google, with its dominant browser — gains outsized influence over what 'fast' means on the web.
The trend: Browser makers are moving from fixed public benchmarks to curated, rotating performance measures, with Google's control of both V8 and Chrome letting it define real-world speed for the web.