Real device testing options and best practices

Real device testing is a critical ongoing requirement for providing a quality experience in a mobile app. But it’s hard to do well, as there are thousands of devices your users might own.

The diversity of Android devices make complete coverage a challenge for many
app producers. In fact, there are well over 24,000 Android devices, and while iOS devices remain popular in the US, globally Android makes up 73% of the device market share. This means that Android devices can’t be glossed over. But, testing thousands of different devices isn’t possible or even necessary.

So how can mobile app companies test the right devices in a scalable way? In this article, we showcase real device testing options and best practices.

What is real device testing?

Real device testing typically refers to manual QA testing of mobile apps and mobile-optimized websites in real devices, but many companies use device simulators for device coverage as well. Depending on the complexity of gestures that your mobile app requires, simulated devices may not be sufficient.

Also, simulated devices aren’t as accurate when it comes to testing mobile environments, which are just as important as the app’s functionality.

The mobile environment includes:

• Device battery life
• Device storage capacity
• Connectivity (WiFi, 4G, 5G)
• Internet quality and speed
• Bluetooth connectivity

In addition to these device factors and app functionality, QA testers using real devices might also be testing for localization and cultural adherence.

Android and iOS own a combined 99% of the market when it comes to cell phone usage. So while most app companies only test for these operating systems and their devices, it’s important to know what your users are using, so you can cover their devices.

Real device testing options

Tech companies have options when it comes to real device testing. Most companies will use a combination of these options. For example, a large online marketplace might test on employees’ own devices, hire a networked testing company for skilled manual testing, and also utilize a digital device farm during development sprints.

Employees’ own devices

One way to cover important device types is for employees to test the app using their own mobile phones and tablets. This is great for a quick unit test of a new feature or a recently updated feature. Engineering teams might collaborate with each other to test what they’re working on.

However, this isn’t scalable.

For one thing, your internal development and QA team likely won’t possess every device that matters to your user base. For another, your internal teams are busy with other large-scale quality endeavors, like automated regression testing or test cycle strategy.

Purchase and maintain a big device library

Some companies opt to purchase devices and keep them in the office so that employees can test with more devices than just what they own personally.

While your company might want to purchase some of the most popular, current devices, it’s not a smart strategy to try and purchase every device that matters around the world. Why? You’ll amass devices, but you still won’t have the testing capacity in the form of additional human testers. You can end up with lots of devices and no one to test them.

Also, when the drastic rise in remote work, this strategy is becoming increasingly outdated.

Freelance testers

Another option for mobile app companies is to hire freelance testers on Upwork or from crowdsourcing companies.

While this can help solve capacity and device coverage issues, it creates other problems. Namely, management.

Freelance testers will need to be managed by your team. It can also be hard to vet the skill of freelance testers. If they’re paid per bug (common with crowdsourcing), then you’ll end up with lots of low quality issues for your engineers to sift through.

Networked testing company

Networked testing allows software companies to hire skilled freelancers in burstable teams, meaning they might have 6 freelance testers testing the app every weekend, but during an important feature release, there are 30 testers around the globe working during an 8-day testing cycle.

Testlio’s networked testing approach offers vetted freelancers working with their own devices across distributed locations, so you can choose the markets, languages, and device types that matter to you.

Digital device farms

The AWS Device Farm offers the ability to test real devices hosted in the AWS cloud. Of course, these aren’t real devices being held in a tester’s hand. Rather, these are simulated devices.

Using device simulators can be great for regression testing and maintaining quality in existing functionality. This strategy is effective for testing base quality and functionality of core features. However, it can’t help you cover the full extent of the mobile environment.

Simulated devices aren’t a complete solution for accessing real devices for these key reasons:

• Device farms have more processing power than mobile devices
• Screen resolutions are always accurate
• It’s not possible to test external conditions such as network speed and device battery

Real device testing best practices

Although real device testing does pose a logistical challenge, that logistical challenge doesn’t have to be your company’s responsibility. For the vast majority of enterprise customers, it simply doesn’t make sense to manage manual QA testers all on their own. There are many reasons why, but one of them is the change in required resources. One week you may need 3 mobile testers, and the next you need 50.

Choose a mobile testing company that helps you strategize and execute on device coverage. Look for these important features in any solution to real device testing:

• Vetted testers
• Available in the global markets you need
• Ability to shrink and expand testing resources as needed
• Can cover popular devices from the past 4 years
• Test run management
• Collaboration between freelancers to reduce duplicate issues
• Ability to utilize in-house QA teams alongside freelance testers in one platform

Device coverage

We live in a world of increasingly diverse and personalized digital experiences. Users access mobile and web apps through devices that offer thousands of combinations of screen sizes and operating systems, leaving QA managers to decide how many and which devices to prioritize. With employees working from home, in-house device labs are inaccessible to teams accustomed to testing on real-world devices before a release.

Based on a comprehensive analysis of client release data and device-OS configuration tests, Testlio found leading mobile app companies in most industry segments follow a best practice of testing 24 or more unique device OS combinations before each update. In the Commerce & Retail category, a Testlio client that tests 41 unique device/OS combinations per update ties with three other shopping apps for the highest rating (4.9) within the top-30 shopping apps. Check out The State of App Testing 2020 report for in-depth insights and benchmarks for commerce & retail app engineering and QA teams.

Ultimately, real device testing is essential to making sure that your real users love your app. Testlio offers human in-the-wild software testing with a unique networked approach. Get in touch to learn more.