AnandTech shows how deep and wide benchmark cheating extends in Android

BY

Published 2 Oct 2013

NSFW AI Why trust Greenbot

We maintain a strict editorial policy dedicated to factual accuracy, relevance, and impartiality. Our content is written and edited by top industry professionals with first-hand experience. The content undergoes thorough review by experienced editors to guarantee and adherence to the highest standards of reporting and publishing.

Disclosure

Android-43

Earlier this week, AnandTech confirmed Samsung was using hardware and software trickery to cheat on its benchmarks. Now, the online publication is  back with a followup report that suggests Samsung in not alone in this ruse.

AnandTech’s initial report on the Galaxy S4 and the Note 3 show that Samsung was boosting both max GPU and CPU performance in its devices by modifying the thermal limits and CPU voltage/frequency of the SoC in their devices. AnandTech writes,

1) On the Exynos 5410, Samsung was detecting the presence of certain benchmarks and raising thermal limits (and thus max GPU frequency) in order to gain an edge on those benchmarks, and

2) On both Snapdragon 600 and Exynos 5410 SGS4 platforms, Samsung was detecting the presence of certain benchmarks and automatically driving CPU voltage/frequency to their highest state right away. Also on Snapdragon platforms, all cores are plugged in immediately upon benchmark detect.

AnandTech then looked beyond Samsung’s hijinks and discovered that every handset manufacturer except for Motorola has at least one device that uses some form of CPU optimization to game benchmark results. It’s a messy affair with manufacturers targeting a variety of benchmark apps on a long list of devices going back more than a year (maybe even two).

anandtechbenchmarks

With the way things are playing out, there’s a grim the future for Android benchmarking. Rather than halting these optimizations, AnandTech believes that OEMS will just hide these hacks, making them more difficult to uncover. Thankfully, the authors believe there are still ways of detecting these OEM tweaks no matter how carefully they are hidden.

[Via AnandTech]