How to build a Cross-platform matrix for Usability Testing

When developing an app, it’s vital that you build in a testing strategy for it. 

Some developers think in terms of Android or iOS only, but there is a myriad of other variables that should be considered and tested for. Especially with apps for mobile devices. 

It isn’t only the OS

When you get your first beta testers, there is a good chance that they have a wide variety of devices at their disposal. 

Developers should use all of them. Not only to see if the user experience works with the desired operating system but also to ensure that you’re getting the same experience on different types of devices.  

Cross-platform testing is when you not only test across various devices but also operating systems. In the case of a web app, you’ll also need to test across multiple browsers to ensure that you’re getting the user experience that you’d like to see. 

How do I test everything?

In a word; don’t. You won’t have enough time or personnel to test everything. When you look at designing a testing model, it’s very important to look at a few factors:

  • Who are your users
  • Which test platforms do you want to use
  • Limits on test cases

It’s extremely important to know your users. 

If you know the vast majority of your users use Android, you may skip the iOS from the start. On the other hand, if your users tend to work between a laptop and a mobile device, you may choose to design testing to cover web-browsers too. Here you’ll want to develop specific use cases to test how your users will be interacting with your app, and on which devices.

Testing platforms

When selecting which platforms to test, it’s important to know that you will not be able to test them all. If it’s a browser app, there are a lot of different browsers on a lot of different machines running a lot of different operating systems. 

The same is true for mobile devices. 

Android and iOS are the big two, but those devices run on a wide variety of devices with a wide variety of screen sizes and that isn’t even taking into account that various versions of the OS that may be running on these devices. So, don’t beat yourself up if a few users end up on a machine running some form of windows from the late 90s that you didn’t test for. 

Limits on testing

When designing your tests, it’s very important to define the scope of the tests. Which devices will be used. Which browsers or Operating Systems should be tested, and which versions of each should be supported. This will keep you from scope-creep and make sure that you can complete testing and roll-out in a timely manner. It’s really easy to get into a test-everything-mode. That’s not only expensive but as I’ve pointed out, nearly impossible from a practical standpoint. 

You know what to do

A few other things that could save you both time and money:

Platform coverage and testing design aren’t much different than any other type of testing work. You need to clearly define who your customers are, and dedicate resources to testing on the devices and platforms that they use in your defined use-cases.