4 Ways to Combat Android Fragmentation in QA

Android fragmentation poses a challenge for developers and for QA teams: How can they deliver quality on all of their users’ devices?

Android device coverage: the conventional wisdom

With upwards of 24,000 devices, Android developers are forced to make tough decisions about testing coverage. Clearly, not every device and OS version can be tested. Even so, not every device that is tested can be tested for every use case.

It’s just too time consuming.

Teams will often test the most popular 80% or 90% of devices owned by their users, that way they get peace of mind without testing everything on the market.

If your team has struggled to identify what those devices should be, or has wondered if a higher or lower percentage is more fitting, I’m going to get into four points to consider for any device coverage strategy. But first, a word of caution…

The problem with ignoring fragmentation

Developers can be tempted to ignore Android fragmentation at any time. QA consultant Joe Schultz details working with a large bank who had chosen to test on just two Android devices when rolling out an update to its mobile app.

The bank was flooded with poor ratings that impacted their original app rating. After upping the test coverage to 200 hardware and OS combinations, the bank was able to identify improvements, roll them out in a subsequent upgrade and eventually get their app store ratings back to original standings (and halt the complaints received via social media).

Teams regularly have to decide how much flack they’re willing to risk in case there are major issues with the devices they’ve left untouched.

Combatting Android fragmentation during QA

1. Analyzing your own data for OS version, device usage, and phone versus tablet

When strategizing device coverage for an existing application, nothing is more important than your own usage analytics.

Here’s the data you should be collecting and analyzing:

  • Operating system: do your users typically upgrade to the newest version of Android, or do you need to support past versions? If so, how far back?
  • Devices and brands: which devices are commonly used? Are your users on the most current versions? If there are multiple versions of one device, how similar or different are those versions and can they be combined?
  • Phone versus tablet: If your app has been designed for use on a phone and on a tablet, what is the breakdown of use? How much of the device coverage should be allocated to phones versus tablets?

When you know exactly how many users you risk by excluding KitKat but covering LolliPop and Marshmallow, you can feel more confident in accepting the risk and narrowing your testing combinations (thus reducing overwhelm).

2. Making choices based on geographical distribution & market data

Teams will also find it useful to analyze general market data to give insight into the growth of their product and the type of user who might be adopting it in the near future.

Depending on the markets they serve, some developers may need to test on low-end devices that are not officially compatible with Android and yet run on it anyways. This could require inside knowledge about devices whose popularity isn’t well tracked.

3. Covering manufacturers of specific hardware

So, depending on the nature of the app, device coverage may include underlying capabilities rather than just brands and their popular offerings.

4. Using combinatorial testing and sampling to narrow test cases

With a better understanding of device, OS version, and hardware needs of their user base and overall market, teams still have to reach a level of confidence in how they combine these requirements together with test cases.

The combinations could result in hundreds of thousands of unique test cases if not kept in check.

That would be bad math. But good mathnamely analysis, sampling, and combinatorial calculationsallows for the reduction of test cases.

Randomization and sampling can also help lessen QA overwhelm when it comes to combining devices, OSes, and test cases.

Every QA team needs to cover the most critical user journeys in the most important devices. Fragmented platforms like Android don’t make this easy, but you can still deliver an amazing customer experience — all it takes is a little strategy.

Jumbotron image