However, benchmarks have distinct life cycles, as technology advances and browser developers work around the benchmarks’ limitations. SunSpider was one of the first JavaScript benchmarks, and as its usefulness waned, Google’s Octane was introduced to take its place in 2012. Now Octane, as well, has reached the end of its usefulness and it, too, is being retired.
The reasons for Octane’s demise are fairly complex, and you can check out Google’s announcement at the V8 project blog for all of the details. In simple terms, starting around 2015, most JavaScript engines had optimized their compilers to score well on Octane tests. As developers worked to achieve ever-higher benchmark results, the actual benefits to real-life web page performance became increasingly marginal.
In addition, sites like Facebook, Twitter, and Wikipedia have demonstrated that the Octane benchmark wasn’t accurately reflecting how Google’s V8 JavaScript engine actually worked on real sites. Therefore, Octane wasn’t actually capturing important information on how V8 and other JavaScript engines perform in the modern web environment.
Finally, it became apparent that efforts to gain higher Octane benchmark results were actually having a deleterious effect on performance in real-world scenarios. The combination of Octane’s increasing disconnect with how web sites actually work with developers’ efforts to achieve higher and higher Octane scores meant an increasingly negative impact on how JavaScript engines were designed to perform when it really matters. Even bugs in Octane have been leveraged by developers to gain higher benchmark results.
All benchmarks suffer from similar problems, according to Google, meaning that the very process of making benchmarks to demonstrate performance eventually leads to performance decreases as developers write code that’s optimized for the benchmarks and not the real world. Google’s efforts going forward will be focused on improving performance measurements of browser performance on real web pages as opposed to merely running static test suites.
