Real device testing options and best practices

Real device testing is a critical ongoing requirement for providing a quality experience in a mobile app. But it’s hard to do well, as there are thousands of devices your users might own.

So how can mobile app companies test the right devices in a scalable way? In this article, we showcase real device testing options and best practices.

What is real device testing?

Also, simulated devices aren’t as accurate when it comes to testing mobile environments, which are just as important as the app’s functionality.

The mobile environment includes:

  • Device battery life
  • Device storage capacity
  • Connectivity (WiFi, 4G, 5G)
  • Internet quality and speed
  • Bluetooth connectivity

Real device testing options

Employees’ own devices

One way to cover important device types is for employees to test the app using their own mobile phones and tablets. This is great for a quick unit test of a new feature or a recently updated feature. Engineering teams might collaborate with each other to test what they’re working on.

However, this isn’t scalable.

For one thing, your internal development and QA team likely won’t possess every device that matters to your user base. For another, your internal teams are busy with other large-scale quality endeavors, like automated regression testing or test cycle strategy.

Purchase and maintain a big device library

Some companies opt to purchase devices and keep them in the office so that employees can test with more devices than just what they own personally.

While your company might want to purchase some of the most popular, current devices, it’s not a smart strategy to try and purchase every device that matters around the world. Why? You’ll amass devices, but you still won’t have the testing capacity in the form of additional human testers. You can end up with lots of devices and no one to test them.

Also, when the drastic rise in remote work, this strategy is becoming increasingly outdated.

Freelance testers

Another option for mobile app companies is to hire freelance testers on Upwork or from crowdsourcing companies.

While this can help solve capacity and device coverage issues, it creates other problems. Namely, management.

Freelance testers will need to be managed by your team. It can also be hard to vet the skill of freelance testers. If they’re paid per bug (common with crowdsourcing), then you’ll end up with lots of low quality issues for your engineers to sift through.

Networked testing company

Networked testing allows software companies to hire skilled freelancers in burstable teams, meaning they might have 6 freelance testers testing the app every weekend, but during an important feature release, there are 30 testers around the globe working during an 8-day testing cycle.

Digital device farms

The AWS Device Farm offers the ability to test real devices hosted in the AWS cloud. Of course, these aren’t real devices being held in a tester’s hand. Rather, these are simulated devices.

Using device simulators can be great for regression testing and maintaining quality in existing functionality. This strategy is effective for testing base quality and functionality of core features. However, it can’t help you cover the full extent of the mobile environment.

  • Device farms have more processing power than mobile devices
  • Screen resolutions are always accurate
  • It’s not possible to test external conditions such as network speed and device battery

Real device testing best practices

Choose a mobile testing company that helps you strategize and execute on device coverage. Look for these important features in any solution to real device testing:

  • Vetted testers
  • Available in the global markets you need
  • Ability to shrink and expand testing resources as needed
  • Can cover popular devices from the past 4 years
  • Test run management
  • Collaboration between freelancers to reduce duplicate issues
  • Ability to utilize in-house QA teams alongside freelance testers in one platform

Device coverage

We live in a world of increasingly diverse and personalized digital experiences. Users access mobile and web apps through devices that offer thousands of combinations of screen sizes and operating systems, leaving QA managers to decide how many and which devices to prioritize. With employees working from home, in-house device labs are inaccessible to teams accustomed to testing on real-world devices before a release.

Based on a comprehensive analysis of client release data and device-OS configuration tests, Testlio found leading mobile app companies in most industry segments follow a best practice of testing 24 or more unique device OS combinations before each update. In the Commerce & Retail category, a Testlio client that tests 41 unique device/OS combinations per update ties with three other shopping apps for the highest rating (4.9) within the top-30 shopping apps. Check out The State of App Testing 2020 report for in-depth insights and benchmarks for commerce & retail app engineering and QA teams.

Jumbotron image