Yesterday, upon rolling out an update to their popular 3D benchmark, 3DMark, Futuremark decided to announce that they would be de-listing a few devices from their public ranking/database of smartphones. While the update to the 3DMark 3D gaming benchmark rolled out, Futuremark also released a press release stating that they had de-listed a few devices from their scoreboard (put to the bottom and removed from comparison). This de-listing came as a result of a multitude of reviewers and manufacturers questioning the validity of certain devices’ scores.
In addition to reviewers and competitors becoming suspicious, Futuremark themselves also became suspicious of these devices’ scores and decided to investigate. Upon completing a long investigation of the devices’ benchmark behavior and code, Futuremark decided to de-list four Samsung devices and two HTC devices from their benchmark database and essentially nullify their scores. The reason that Futuremark gave for de-listing the devices was because they did not adhere to the rules that they set for hardware manufacturers and software developers when running 3DMark benchmark tests. The full statement can be read here, "People rely on Futuremark benchmarks to produce accurate and unbiased results. That’s why we have clear rules for hardware manufacturers and software developers that specify how a platform can interact with our benchmark software. In simple terms, a device must run our benchmarks without modification as if they were any other application.
When a device is suspected of breaking our rules it is delisted. 3DMark scores from delisted devices should not be used to compare devices. Delisted devices appear unranked, and without scores, at the bottom of the 3DMark Device Channel and the Best Mobile Devices list on our website."
If you look at the de-listed devices above, you can see that they do not all use the same SoC nor do they share the same manufacturer. The clear similarities between these devices are the fact that they are manufactured by only two companies and that they are both devices that the companies have touted as their latest and greatest. Even though HTC has released the HTC One Max, it still has the same SoC as the HTC One which is curious that it doesn’t cheat on 3DMark while the HTC One and HTC One Mini do.
Samsung comes as no surprise when it comes to smartphone benchmark cheating, as they had been caught earlier by the guys at Rightware trying to cheat on Basemark’s 3D benchmarks. The truth is that many of these smartphone manufacturers understand the importance of doing well in benchmarks and as a result are trying to optimize for them when they shouldn’t be. The sad truth is that when companies try to compete on paper alone, benchmarks end up becoming a vastly differentiating factor and some companies will go to any length to be the best on paper. Even if they are cheating to do it.
I applaud the guys at Futuremark for going out and de-listing these devices and making a statement that they will not allow these companies to pull the wool over our eyes when it comes to their devices’ performance. They should adhere to the rules set out by Futuremark as well as their benchmark development program and appreciate any scores that they get and do their best to improve them through engineering, not software optimizations.