How Many Mobile Device And OS Combinations Should You Test?

Testlio finds that leading app providers test 24 mobile device-OS combinations on average.

January 22, 2020. Austin, TX. Today, users access mobile apps through devices that offer thousands of combinations of screen sizes and operating systems, leaving app QA managers with a challenging decision of how many and which devices to prioritize.

This week, managed app testing company Testlio released new mobile device testing data from a recent query of its client base, providing a unique look into the number and most common mobile device/OS testing combinations among some of today’s QA testing leaders. The data set includes QA testing data from 31K+ client test records spanning a diverse range of industries (including commerce, education, entertainment, finance, productivity, sports, and travel). Collectively, these Testlio clients have a user base of more than 1.65 billion people.

As a benchmark for other app providers, Testlio identified the top ten most frequently tested mobile device and operating system combinations from their 2019 customer data:

1. Google Nexus 5X Android 8.1
2. Apple iPhone 6s iOS 11.4.1
3. Samsung Galaxy S7 Android 8.0
4. Google Pixel 2 XL Android 9.0
5. Apple iPhone 5 iOS 10.3.3
6. Apple iPhone X iOS 12.3.1
7. Xiaomi Mi A1 Android 9.0
8. Google Nexus 6P Android 8.1
9. Google Pixel Android 9.0
10. Apple iPhone 7 iOS 12.3.1

Based on a comprehensive analysis of client release data and device-OS configuration tests, Testlio also found leading mobile app companies follow a best practice of releasing mobile apps every 1 or 2 weeks, and testing 24 or more unique device-OS combinations before each update.

We live in a world of increasingly diverse and personalized mobile experiences. We’re thrilled to see our clients embrace a human-driven approach to real device/OS testing, and we salute our clients’ collective approach to testing ~24 device/OS combinations per mobile app release.

Testlio CEO, Steve Semelsberger

Cross-platform testing — testing across multiple devices and operating systems — is not only essential to ensure a quality customer experience across the myriad of different mobile environments, but also to keep up with frequent updates to the Google Play and Apple App stores. When prioritizing mobile device testing, we recommend QA managers consider the data above, as well as the top mobile app testing devices and operating systems currently available.

For easy reference to perspectives on mobile device testing, please grab the infographic.