Android browser tops iPhone on 84% of real-world website loading tests, 52 percent faster overall: report

iOS 4.3 introduced the new Nitro Javascript engine which was supposed to double JavaScript execution, although real-life testing by various sites hasn't necessarily shown that to be the case. Now, with all the flak about Web apps launched directly from the iOS home screen running two to two and 1/2 times slower than if launched via the Safari browser, it's the perfect time for an Android vs. mobile Safari browser test.

A series of some 45,000 load tests run by Blaze Softwareshowed a relatively recent Android smartphone (Samsung Nexus S) loading Web pages 52 percent faster than an iPhone 4 running iOS 4.3. The Nexus S is a good candidate for this sort of test, as it's running Android 2.3, the latest smartphone version of Android, and also runs stock Android, meaning no Sense UI, TouchWiz or MotoBlur UI layer.

On the other hand, the Nexus S is hardly the speed leader among recent smartphones, so this sort of result is even worse for the iPhone than it might be. The Atrix 4G has been said to be too powerful to be called just a smartphone, and the HTC Thunderbolt, Verizon's first LTE smartphone just released today has been labeled "freakin' fast" by LAPTOP magazine. Those, however, only run Android 2.2 (Froyo).

In addition to testing out 52 percent faster, Android’s Chrome browser bested the iPhone’s Safari by 84 percent of the websites tested faster. Also, the firm found that while both firms said their Javascript engines have improved from prior versions, it seems that those improvements are isolated to benchmarks and high-complexity Web apps, not typical browsing.

Here's what Blaze said about benchmarks vs. real-life browsing:

Past comparisons usually focused on custom-created benchmarks, such as the SunSpider JavaScript benchmark. While useful, these benchmarks are very different than real world sites, and don’t reflect the actual user experience. This study measured the load time of 1000 real web sites, mimicking the experience users would get when browsing on their smartphones.

In terms of their methodology, Blaze said:

The study was done primarily on iPhone 4 and Google Nexus S. The websites used were those of the Fortune 1000 companies. Each page was loaded multiple times and on different days, measured primarily over WiFi. For each device, we used the median load time for the comparison. The total number of tests was over 45,000.

This study was made possible through custom apps we developed, used to measure page load time on mobile devices. These apps run on the actual devices, load a page on demand, and measure how long it took. These agents are available as a free service to measure your own site on Mobitest.

Blaze also used a number of methods to minimize anomalies in the testing, such as tossing out ed out results higher than 40 seconds or lower than 400 milliseconds, as those usually indicated server or network issues, and even rerunning the tests on different days. The full methodology is detailed in an appendix in their report.

The key conclusion is that benchmarks are fine, but real-world results will differ. What end users really care about, in general, are real-world results. It is, however, true that if more app vendors move toward Web apps on iOS because of Apple's App Store policies, Javascript performance may become more significant to end users.