Hardware, Software Programs

Qualcomm Benchmarking Day – Benchmark Results

As a follow up to our previous article, attending the Qualcomm Benchmarking Day at the W Hotel in San Francisco we are going to present our benchmark results as well as our analysis.

The Tablet
The MDP APQ8064 is a 10.1" tablet featuring a 1366×768 resolution, packing Qualcomm’s latest quad-core APQ8064 applications processor with Adreno 320 graphics. The tablet itself has a ton of connectivity and features that enable developers to test different aspects of the processor’s abilities. The tablet packs 2GB of LPDDR2 RAM, 32GB of storage, a 13 MP main camera and a 2MP front-facing camera all running on Android 4.0 ICS. As far as connectors go, it has a Micro USB type AB, 3.5mm audio jack, DC-in barrel charger and a micro SD card slot. When it comes to sensors, it has a 3D accelerometer, a 3-axis gyroscope, a 3-axis compass, an ambient light and proximity sensor as well as a temperature and pressure sensor. As if all of that weren’t enough, it also has a fingerprint reader for biometric uses.

So, when it comes to the internals of the tablet, the thing that Qualcomm brought us to San Francisco to test was the Krait 1.5 GHz quad-core and the Adreno 320 GPU. Since this is a development device, there are no OEM software overlays or any optimizations that we have been made aware of outside of the usual driver optimizations which would get rolled over to an OEM anyways.

Getting down to the benchmarks themselves we decided to do an array of different benchmarks, with some focusing on the GPU and others on overall system performance, while others on browser performance. In the following graphs, we will include Mozilla’s Kraken Benchmark, Qualcomm’s Vellamo, GLBenchmark, Quadrant, Rightware’s Browsermark and AnTuTu. Some of these benchmarks were already pre-installed on the tablets, however we discovered that some of them were already outdated (like AnTuTu) and we took it upon ourselves to update everything to the latest version.

In our testing, we will be comparing the APQ8064 powered MDP against the Qualcomm MSM8960 powered HTC One S and the NVIDIA Tegra 3 powered HTC One X. The nice thing is that the MSM8960 actually has a lot of similarities with the APQ8064 since both of their CPUs are based off of Qualcomm’s Krait architecture. The main difference is that the APQ8064 is a quad-core versus the MSM8960’s dual-core and the fact that the APQ8064 packs the long awaited Adreno 320, which claims on paper to deliver 3-4x the graphical performance of the Adreno 225, which is Qualcomm’s predecessor to the 320.

In Browsermark, we got similar results as to what we saw in Mozilla’s Kraken benchmark, however, the margins were much narrower.

The Highest scoring device was the APQ8064, which came in at 123457 compared to the MSM8960 powered One S which scored 88693 and the Tegra 3 powered One X at 78944. Both Browsermark and Kraken in our eyes are the most objective benchmarks that we’ve run into and as such we consider them to be the two best to measure by. We tried using browsermark in Chrome, however, it would not run and we resorted to using the default browsers on the devices.

Vellamo, as many already know is Qualcomm’s benchmark mostly for HTML5 browsing. This benchmark combines a series of different widely accepted benchmarks (like Sunspider) to generate a score.

In Vellamo, the APQ8064 scored 2520, the highest score we’ve ever seen. The MSM8960 in the One S, scored 2391, indicating to us that the Vellamo benchmark as we already know is not necessarily threaded for performance. The Tegra 3 in the HTC One X ended up coming in with 1528, which is much lower than the other two devices with Qualcomm’s chips in them. While we do believe that Vellamo is definitely a good benchmark, we simply cannot escape the stigma associated with it being a Qualcomm benchmark.

Linpack for Android
In this benchmark, we are purely measuring the raw CPU power of the processor in a multi-threaded manner. Linpack for Android is CPU-only and does not test the GPU nor the whole system’s performance. We are purely running this test in order to see the performance improvement from the MSM8960 to the APQ8064 as well as where NVIDIA’s Tegra 3 fits in.

In our benchmarks, the APQ8064 effectively clobbers everything in sight. The APQ8064 delivers a peak performance of 537.155 MFLOPS which is more than double the MSM8960’s 214.867 MFLOPS. Unfortunately for NVIDIA, the Tegra 3 only scored 160.788 MFLOPS, meaning that the APQ8064’s quad-core processor is 3.3x faster than NVIDIA’s quad-core. We believe that this is going to be one of the places where NVIDIA has the most room for improvement and where Qualcomm really shows the strength of their Krait architecture.

GLBench 2.1 Egypt and Pro Offscreen
Everyone is very familiar with GLBench and their GPU benchmarks, since we wanted to be independent of screen resolution in this benchmark, we went with the offscreen option for both the Egypt and Pro benchmarks to purely see the differences in GPU performance.

In GLB Egypt, we got some pretty interesting results with the APQ8064 scoring 131 FPS compared to the Tegra 3 which scored 64 FPS and the MSM8960 which reported 51 FPS. This shows an improvement of about 2.6x in terms of GPU performance going from the Adreno 225 to the Adreno 320. Similarly, with the Tegra 3’s GPU, the APQ8064 more than doubles it’s performance.

In GLB Pro, We expected to see more of the same from all three chips and for the most part, did. The performance difference between all three chips narrowed, however, the difference was still pretty astonishing. The APQ8064 scored 183 FPS while the Tegra 3 notched in at 91 FPS and the MSM8960 at 87 FPS. Once again, the APQ8064 wins by a broad margin, but this time by 2x as opposed to 2.6x. Since the Tegra 3 and MSM8960 were so close in performance, it’s safe to say that the
Adreno 320 in this case is a little more than 2x faster than the Adreno 225.

We also decided that we would run Quadrant just because so many different people use it as a benchmark, even though we don’t necessarily think it’s a good benchmark for real world performance or CPU/GPU performance. We simply included it because it has been used so widely and we would not consider it a completely accurate measure of system performance. The real difference was that the margin of performance had been significantly narrowed compared to all the other tests that we ran.

The APQ8064 reported a score of 7630, which beat the Tegra 3 with 4819 and the MSM8960 with 4341, so no surprises there. However, the performance difference between the APQ8064 and the MSM8960 was significantly narrowed to an increase of ‘only’ 75 percent. Additionally, the APQ8064 beat the Tegra 3 by a margin of 37%, the narrowest margin we’ve seen so far.

Last, but not least , we have AnTuTu which is similar to Quadrant in popularity but not necessarily as bad in terms of being a bad benchmark. They do, however, have specific values that they attribute to different benchmarks that give them different weight over others. As such, we’re mostly running AnTuTu for people that have run AnTuTu on their phones before purely based upon what they’ve known about and used in the past. Interestingly, we actually had to re-run our AnTuTu benches because the version on Qualcomm’s tablet was actually older than the version on our phones, so we had to re-run the Qualcomm tablet after installing the newest version. This did not necessarily affect performance from our own observations.

In this test, we see that the APQ8064 more than doubles the score of the MSM8960 (since it does have double the cores and a significantly more powerful GPU) and bests the Tegra 3 by a significantly lower 20 percent. This 20 percent performance difference is the narrowest we’ve seen between the Tegra 3 and APQ8064.

There is no doubt that the APQ8064 is the fastest applications processor we’ve seen to date. The only thing that we really have to say beyond the commentary in our benchmarks is that the APQ8064 was tested on a basically untouched manufacturer tablet, while the other two chips were in full-fledged consumer devices. There may be a chance that once this processor is given to OEMs it could suffer a few performance decreases, but it would not necessarily be to a degree where it would change the overall performance deltas we’ve seen here today.

We also have to consider that the APQ8064 is still not technically shipping in any devices outside of Qualcomm’s own mobile developer platform, so one must also consider that it isn’t quite finished yet and could potentially improve further. Additionally, one must also consider that its main competitor, the Tegra 3, is nearing almost a year in age already and is shortly due for an update with the Tegra 4, which all signs point to very soon. The APQ8064 definitely is the processor that Qualcomm has needed to beat NVIDIA in performance. It also shows that even though they talked quite a bit about how little importance a quad-core processor had on the experience, they too had to come out with a quad-core processor in order to deliver the utmost highest level of performance.

Now, the balls is in NVIDIA’s court and we are really interested to see what they have to answer against the APQ8064. Since the Tegra 3’s release we had been talking to people internally inside of NVIDIA and expressing our disappointment with NVIDIA’s GPU performance on the Tegra 3 considering that NVIDIA is a GPU company. We are hopeful that Tegra 4 will deliver everything that NVIDIA needs in order to beat Qualcomm’s APQ8064 to keep pushing the envelope further and continually making better processors available to consumers. Clearly, with the Adreno 320, Qualcomm is recognizing the importance of GPGPU, which we believe NVIDIA will also do with the Tegra 4. The real question will then become, which company will harness GPGPU the best and what benchmarks will be out there to measure that?

Edit 7/28/2012: Upon being tipped by some of our readers to some confusing results, we have decided to throw out our Kraken benchmark results until we can get more reliable results. Currently, we have the MSM8960 with 23981, APQ8064 with 26157 and the Tegra 3 at 31399. These results are almost an entire flip of what we got when testing at the Qualcomm event, as such we are going to scratch everything and revisit this benchmark at another date. We apologize for any inaccuracies or confusion.