How to Build QA at Scale Through Device Architecture

Let’s start with some absolutes. Maintaining a steady CI/CD pipeline is crucial to modern software development. Testing and quality engineering principles that drive solid deliverables should be flexible and agile. QA teams need hundreds of device and O/S combinations. With those non-negotiables in mind, how would you implement a flexible device architecture to support ongoing and evolving needs to assess mobile and web applications?

In our recent webinar, “Device Farms Reimagined,” experts from eToro, Sauce Labs, and Testlio met to discuss device architectures’ role in building a quality engineering system for scale. 

Dan Ganon, Senior Director of Quality at eToro, Marcus Merrell, VP of Technology Strategy at Sauce Labs, and Summer Weisberg, Chief Client Officer at Testlio, pull from real-world experiences to offer perspectives on the pros/cons of alternative testing architectures.

Quantify coverage in a scalable way

First, quantify coverage as it matters to you. The explosion of variability in today’s software, microservices, technologies, languages, and teams in a modern software stack makes it impossible to gain 100% coverage. 

“I always encourage people to look at risk and risk mitigation instead and quantify that before you go into testing,” said Marcus Merrell, VP of Technology Strategy at Sauce Labs. “[Then] you can say with confidence: ‘I’m about to cover all of the major important risks that matter to us.'”

With the nearly infinite device, O/S, and functionality combinations to test, it’s critical to start here before assessing what the metrics say is most important to your user base. Start by asking: 

• How much can I spend on coverage? 
• Which devices are used the most? 
• What are key flows clients are leaving you for if they fail? 

What do clouds, crowds, and device farms offer?

Device architecture encompasses a wide swath of readily available devices. When you think of testing scenarios, whether local teams or externally sourced, your team is trying to account for all possible scenarios. These scenarios can sometimes give a more comprehensive view of real-world usage. Like web applications running on different browsers under different operating systems. Leaving behind the often archaic and hard-to-use device labs, two leading infrastructures companies are utilizing today are real devices via the cloud and real devices via crowdsourcing. 

Companies utilize real device clouds with or without a vendor to maintain device and O/S coverage during testing runs. Device clouds offer access to a virtual catalog of devices and operating systems that companies can use remotely and on demand. Device architecture allows you to scale testing as needed and can be cost-effective.

In comparison, or sometimes in conjunction with, crowdsourcing real devices requires building out a remote team of testers (through a vendor with a network of testers like Testlio or your teams) that use personal or provided devices in hand. This allows your development team to get a bird’s eye view of how the deliverable performs when in the end user’s hands.

There’s still a human element that’s really important, especially when you’re dealing with consumers who will leave you quickly if they don’t enjoy the experience of your application.

Summer Weisberg, Chief Client Officer at Testlio

Focusing on optimizing for scale will likely require a unique combination of virtual and in-hand testers, utilizing one or more vendors to reach the coverage area that makes you feel most comfortable releasing a new/updated product.   

Quality engineering, automation, and device architecture 

What do automation and quality engineering ask of device architecture? Your organization has likely already implemented a hybridized workflow focusing on manual and automated testing. Extend that to your architecture. Implementing device clouds and scaling with device crowds could readily supplant your manual testing phase while retaining the automation phase as needed. 

You need a device architecture that can support your QE directives. Device clouds bring value in shifting left, speed, time to market on scripts, and a broader coverage area. Device crowds get hyper-specific tests, UX runs, and a focus on localization and location. Having real-world data is invaluable. So how do you marry these two?

“Maybe you have two parallel flows. You’re automating some tests and manually running other tests to get human insights. Then, you get results where you can make really intelligent decisions on if you should move forward with this release, or what are your risks,” Weisberg said.

Your organization could make substantial gains by implementing a hybrid device architecture. You can readily scale the scope of your quality engineering while accounting for various unique scenarios that benefit from real users on real devices.