How to maintain complete device coverage when under pressure to release

So many devices, so little time. When you’re on a tight release schedule, it can be challenging to cover the devices that matter to your user base.

But excellent device coverage is essential. The experience that your users actually receive should match what your product and engineering team intends. 

There are 5.11 billion unique mobile users and 1.26 billion tablet users around the world. You want all of your mobile app’s users or mobile website’s users to get an equally high-quality experience, so long as they’re not using hardware that’s drastically outdated. 

Catering your app to different devices and mobile experiences is crucial. The top-reviewed apps are just 28-days young on average. The best apps get tested and updated often. 

If you’re a mobile-first company, device coverage is a critical part of maintaining quality.

Working from home makes device coverage more challenging than ever

More people are working from home than ever before. 88% of organizations have encouraged at least some of their employees to work from home during the shutdowns, and 31% of companies who continue to allow remote work say that COVID restrictions were the trigger.

While working from home might offer more lifestyle freedom and safety, it can be incredibly challenging for any company that uses or builds hardware. This could be companies like Apple, which sent hardware engineers and testers back to the office complex before anyone else. Or it could be mobile app companies that need to use physical devices for accurate testing. 

Device coverage is no less important during trying times, but actually maintaining that device coverage is an enormous challenge. 

Regardless of which of the following ways you choose to implement, one thing is for certain: to cover the devices that matter to your users, you need more resources. 

Ways to maintain device coverage 

Whether you’re dealing with time constraints, or time and location restrictions, there are things you can do to maintain device coverage during release cycles. 

Check out these four unique options. 

1. Work with a networked testing company

Freelancing is growing quickly. 54% of Millennials and Gen Z in developed countries either have already started a business or plan to start one. Because these generations crave freedom, freelancers do tend to skew younger than the general workforce. 

While there is plenty of freelance talent available, the challenge is finding the right people and putting their efforts to good use. 

You may have heard of crowdsourced testing before. QA managers or engineers can crowdsource testers and choose which testers they want to hire and manage. The burden of selecting testers, assigning test cases, and verifying device coverage and app coverage falls squarely with the client. 

But, you may not have heard of networked testing. With a fully managed networked testing approach, the vendor is fully in charge of selecting the right team, coordinating their efforts, and managing the testing cycle end to end. Whereas a co-managed networked testing partnership shares the responsibility: clients write test cases, invite testers, assign test cases, run tests, and monitor test results. This approach is preferred by organizations that want to share testing processes and test any time based on immediate needs. Ultimately, clients find that a networked testing approach (either fully-managed or co-managed) produces a higher quality of test coverage and issue reporting, while testers gain rewarding work and fair pay.

2. Bring in individual freelance testers as needed 

While a fully managed testing vendor is ideal, it’s potentially out of budget for small, early-stage companies. But the need for testing real devices remains. A QA manager can hire contract testers to help out. Finding skilled testers in LinkedIn or Upwork who have testimonials or reviews will likely prove more fruitful than relying on crowdsourced testing sites. 

You’re better off forging an individual relationship because then you can set a fair hourly compensation. Because crowdsourced sites typically pay per bug, they attract very inexperienced testers and the result is often not worth the cost savings. 

3. Split devices among employees

If you have a large library of devices in the office, you can split them among your team. Employees working remotely can take a percentage of the hardware for at-home testing if your organization is still encouraging remote work. 

While this sounds like a fun strategy, it’s typically time-prohibitive. Your QA team is likely lean and focused on test strategy, management, and automation. You likely don’t employ enough manual testers to cover devices. 

Time isn’t the only issue with this strategy. Your in-house team knows how the mobile app or site should operate, and won’t be coming at it with fresh eyes. It’s also challenging to handle localization tests as well as device coverage when testing with this method.

4. Use a device farm

Another option is to use a device farm, which simulates real devices, such as the AWS Device Farm. With a device farm, you can choose a device based on its make, model, and OS version. You’ll then see the device in your web browser, and you can interact with it just like you would a manual exploratory test. You can gesture, swipe, and tap the device, assuming you’re using a touchscreen computer. 

Although these digital device labs do provide easy online access, the simplified cleansing methods they use to present the devices and the centralized location of the tests don’t match real-world environments. 

What you see is the size and responsiveness of the device screen, but you’re not really experiencing the OS and other physical elements such as battery life, device storage, and connectivity. 

There’s nothing that can compare to dozens of hundreds of device/OS combinations using real hardware in the hands of skilled testers. Not only do you get the device’s responsiveness, but also how a real user interacts with that device, with a myriad of other real-world factors at play.

That’s why most engineering and quality teams choose to partner with experienced testers who can provide real device coverage. 

Testlio offers fully managed expert testers. Contact us for a free demo today.