Why Access to Devices is Still a Concern (and What to Do About It)

, Dayana is a QA engineer turned technology writer living in Milan, Italy. She's always down for a smoothie.

Android application testing is notoriously challenging. There are at least 24,000 different Android devices in the world made available by over 1,300 brands. Even for developers well-versed in the conundrum of Android fragmentation, those stats are still shocking.

In this post, I talked about how QA teams can strategically tackle this issue. Rather than allow Android testing to either swallow the entire QA budget or ignore the issue (and risk lack of support for large amounts of users), teams need a strategy that’s realistic to achieve.

To narrow down the devices to test on and which test cases to run in which devices, QA teams need to incorporate in-app analytics, market data, and unique hardware concerns during the decision-making process.

And yet, even when narrowed, access to those devices is still a huge concern. Today I’m talking about why this issue persists and what the best solution is.

Why Android fragmentation will always exist

According to a survey conducted by Sencha and Forrester, developing and deploying for multiple devices is the biggest barrier to development. This survey pertained to app development of any kind, from an Amazon tablet to a desktop browser. Of all the other concerns—including security, cost, and short cycles—device fragmentation topped the list as the biggest blockade to app development as a whole.

Apple does have multiple products, product versions, and OS versions, but the combinations created by iOS are nothing when compared to those created by Android.

There are three main factors involved:

  • OEM custom features
  • Internal hardware
  • Android version being run

Not only do consumers often NOT update to the current version of Android, but manufacturers will develop and sell devices running on previous versions.

Those 1,300 brands all have various products and product versions, not to mention vastly different hardware features, such as screen size, resolution, and shape.

And more OEMs continue to enter the mobile space. Since manufacturers don’t have to create their own OSes, it’s much easier for them to get products to market.

That means there will only be more devices with more particularities.

The number of IT professionals reporting that they don’t have the right devices available for testing almost doubled in one year, from 26% to 44%. This jump is partly attributable to the need for IoT testing, which has sharply increased. The volume of devices that connect to the internet, mobile phone or otherwise, is clearly only going to grow.

The failings of device emulators

When QA teams manage to strategically decide which of the thousands of devices to support, they must then choose how they will support them. Will they emulate the device or test it manually?

For lower priority devices that still need coverage, emulators are a great option. This technology allows QA engineers to test unique devices on their PCs in one platform by working on various device profiles. Emulated devices are a smart choice during initial stages of testing for all identified important devices so that teams can test quickly and improve quality early on.

But there are some clear disadvantages to relying on device emulators:

  • They have increased processing power that does not match mobile processing speeds
  • Screen resolutions and image rendering aren’t guaranteed to be accurate
  • External conditions (like the effect of loud noise on sensors or slow network connections on task completion) can’t be tested
  • Internal hardware differences (like CPUs and GPUs) aren’t tested

Emulators lack so many of the differences that make strategic device coverage important in the first place.

The most reliable way to get Android device coverage

Humans! Mobile testing is synonymous with manual testing because of complex nature of mobile apps and how intimately they’re used. Testing on actual devices is a must for deploying for supported devices with absolute confidence.

Every app is different, so the needs of every team are different. Some apps (particularly games) require extensive manual testing to ensure that the product is interfacing with sensors properly and the graphics are on-point.

For most apps, responsiveness can’t be sacrificed, and the multitude of available screen sizes are reason enough to require manual testing.

But it really doesn’t make sense to collect hundreds or thousands of devices. The cost of devices is high, and they will need to be connected to a network to test outside of WiFi. Most QA teams don’t have the budget for maintaining a large collection of devices, or the time to actually test on them.

Plus, there are hundreds of mobile network operators in the world. Network technologies, standards, and infrastructures can all potentially affect the flow of information from your servers to a device, meaning that location also matters.

Testlio has expert testers around the world who love collecting devices so you don’t have to. QA managers keep track of which testers own which devices, so they can assemble the right team. This testing strategy mimics your user base and allows QA teams to breathe easy.

Why mobile testing should be left to the experts

In the World Quality Report 2016 – 2017, IT teams identify the lack of “mobile testing experts” as a struggle equal to that of the lack of devices. That’s because for many enterprises, mobile testing is still considered a “relatively new” skill.

The temptation to NOT cover all necessary devices means that the user experience goes unsupported. Similarly, the temptation to leave a certain extent of mobile testing to the users equals poor app ratings and low adoption.

Expert mobile testing isn’t just about finding bugs. It’s about verifying the overall quality of an app, whether it achieves its purpose, how seamless the experience is, and detailing the ways it can improve.

To access a global community of expert mobile testers, get in touch with us for the best demo ever.