;

Google Calls Out Rivals’ Web Benchmark Tools, Rebuilds Them To Better Gauge Chrome

When it comes to Chrome, Google has long been addicted to speed. And for many tasks on the web today, that speed is related to how fast your JavaScript engine is. Google has long held that their’s is the fastest. But it’s hard to know for sure because there are a few different benchmark suites to test such speeds — and the most popular ones are made by companies with stakes in the game: Apple, Mozilla, and yes, Google.

In a post yesterday on their Chromium, it’s pretty clear that Google feels their V8 benchmark suite is the best. In fact, they directly call our their rivals’ suites, noting bugs and saying that they must evolve. And then they go one step further: providing links to versions of the rivals’ suites supposedly perfected by Google!

Wow. In the extremely nerdy (and fairly incestuous) browser world, this is hardcore.

Specifically, Google says that SunSpider, first developed by Apple in 2007, contains tests that are “less relevant to the web in 2011″. Here’s the best part:

Even for the more relevant tests, JavaScript has gotten so fast that many finish in only a few milliseconds. This just isn’t long enough to figure out which engine is faster–a golf cart and a Tesla will finish a 10-foot race in nearly the same time.

Ouch.

To fix this issue, Google made its own modified version of SunSpider which essentially runs tests 50 times consecutively to better gauge speed. When the tests are run this way, Google says that “the results begin to reflect Chrome’s true performance.” Naturally. According to Google, Chrome is more than 30 percent faster (in the test results) when measured this way.

Meanwhile, they say that Kraken, the new benchmark suite Mozilla just created, is “in better shape” — but buggy. “As a result, the benchmark is less useful and has even (mis)led us to spend time making some irrelevant optimizations in Chrome,” they note.

Again, ouch!

To get around this, Google is now hosting a new version of Kraken “built directly from Mozilla’s source code repository”.

Are these claims about rivals’ suites legit? It’s hard to say for sure, but I have a feeling that the rivals themselves would dispute that. It is a bit odd that Google is reworking the suites, and that the end result is Chrome performing much better in the tests.

Of course, my eyes don’t lie. I’m a Chrome guy all the way because in daily usage I find it to be much faster than either Safari or Firefox (on a Mac, at least). Until that changes, I’m trusting Google on this one.