How to Build QA at Scale Through Device Architecture

Let’s start with some absolutes. Maintaining a steady CI/CD pipeline is crucial to modern software development. Testing and quality engineering principles that drive solid deliverables should be flexible and agile. QA teams need hundreds of device and O/S combinations. With those non-negotiables in mind, how would you implement a flexible device architecture to support ongoing and evolving needs to assess mobile and web applications?

Dan Ganon, Senior Director of Quality at eToro, Marcus Merrell, VP of Technology Strategy at Sauce Labs, and Summer Weisberg, Chief Client Officer at Testlio, pull from real-world experiences to offer perspectives on the pros/cons of alternative testing architectures.

Quantify coverage in a scalable way

“I always encourage people to look at risk and risk mitigation instead and quantify that before you go into testing,” said Marcus Merrell, VP of Technology Strategy at Sauce Labs. “[Then] you can say with confidence: ‘I’m about to cover all of the major important risks that matter to us.'”

Flowchart on defining coverage via device type and test type

With the nearly infinite device, O/S, and functionality combinations to test, it’s critical to start here before assessing what the metrics say is most important to your user base. Start by asking: 

What do clouds, crowds, and device farms offer?

decorative image featuring a real device crowd vs a real device cloud

There’s still a human element that’s really important, especially when you’re dealing with consumers who will leave you quickly if they don’t enjoy the experience of your application.

Summer Weisberg, Chief Client Officer at Testlio

Focusing on optimizing for scale will likely require a unique combination of virtual and in-hand testers, utilizing one or more vendors to reach the coverage area that makes you feel most comfortable releasing a new/updated product.   

Quality engineering, automation, and device architecture 

What do automation and quality engineering ask of device architecture? Your organization has likely already implemented a hybridized workflow focusing on manual and automated testing. Extend that to your architecture. Implementing device clouds and scaling with device crowds could readily supplant your manual testing phase while retaining the automation phase as needed. 

“Maybe you have two parallel flows. You’re automating some tests and manually running other tests to get human insights. Then, you get results where you can make really intelligent decisions on if you should move forward with this release, or what are your risks,” Weisberg said.

Your organization could make substantial gains by implementing a hybrid device architecture. You can readily scale the scope of your quality engineering while accounting for various unique scenarios that benefit from real users on real devices.