• Become a Tester
  • Sign in
  • The Testlio Advantage
    • Why We Are Different

      See what makes Testlio the leading choice for enterprises.

    • Our Solutions

      A breakdown of our core QA services and capabilities.

    • The Testlio Network

      Learn about our curated community of expert testers.

    • Our Platform

      Dive into the technology behind Testlio’s testing engine.

    • Why Crowdsourced Testing?

      Discover how our managed model drives quality at scale.

  • Our Solutions
    • By Capability
      • Manual Testing
      • Test Automation
      • Payments Testing
      • AI Testing
      • Functional Testing
      • Regression Testing
      • Accessibility Testing
      • Localization Testing
      • Customer Journey Testing
      • Usability Testing
    • By Technology
      • Mobile App Testing
      • Web Testing
      • Location Testing
      • Stream Testing
      • Device Testing
      • Voice Testing
    • By Industry
      • Commerce & Retail
      • Finance & Banking
      • Health & Wellness
      • Media & Entertainment
      • Learning & Education
      • Mobility & Travel
      • Software & Services
    • By Job Function
      • Engineering
      • QA Teams
      • Product Teams
  • Resources
    • Blog

      Insights, trends, and expert perspectives on modern software testing.

    • Webinars & Events

      Live and on-demand sessions with QA leaders and product experts.

    • Case Studies

      Real-world examples of how Testlio helps teams deliver quality at scale.

Contact sales
Contact sales

How to Build QA at Scale Through Device Architecture

Arpita Goala , Content Marketing Manager
August 9th, 2023

Let’s start with some absolutes. Maintaining a steady CI/CD pipeline is crucial to modern software development. Testing and quality engineering principles that drive solid deliverables should be flexible and agile. QA teams need hundreds of device and O/S combinations. With those non-negotiables in mind, how would you implement a flexible device architecture to support ongoing and evolving needs to assess mobile and web applications?

In our recent webinar, “Device Farms Reimagined,” experts from eToro, Sauce Labs, and Testlio met to discuss device architectures’ role in building a quality engineering system for scale. 

Dan Ganon, Senior Director of Quality at eToro, Marcus Merrell, VP of Technology Strategy at Sauce Labs, and Summer Weisberg, Chief Client Officer at Testlio, pull from real-world experiences to offer perspectives on the pros/cons of alternative testing architectures.

Quantify coverage in a scalable way

First, quantify coverage as it matters to you. The explosion of variability in today’s software, microservices, technologies, languages, and teams in a modern software stack makes it impossible to gain 100% coverage. 

“I always encourage people to look at risk and risk mitigation instead and quantify that before you go into testing,” said Marcus Merrell, VP of Technology Strategy at Sauce Labs. “[Then] you can say with confidence: ‘I’m about to cover all of the major important risks that matter to us.'”

Flowchart on defining coverage via device type and test type

With the nearly infinite device, O/S, and functionality combinations to test, it’s critical to start here before assessing what the metrics say is most important to your user base. Start by asking: 

  • How much can I spend on coverage? 
  • Which devices are used the most? 
  • What are key flows clients are leaving you for if they fail? 

What do clouds, crowds, and device farms offer?

Device architecture encompasses a wide swath of readily available devices. When you think of testing scenarios, whether local teams or externally sourced, your team is trying to account for all possible scenarios. These scenarios can sometimes give a more comprehensive view of real-world usage. Like web applications running on different browsers under different operating systems. Leaving behind the often archaic and hard-to-use device labs, two leading infrastructures companies are utilizing today are real devices via the cloud and real devices via crowdsourcing. 

decorative image featuring a real device crowd vs a real device cloud

Companies utilize real device clouds with or without a vendor to maintain device and O/S coverage during testing runs. Device clouds offer access to a virtual catalog of devices and operating systems that companies can use remotely and on demand. Device architecture allows you to scale testing as needed and can be cost-effective.

In comparison, or sometimes in conjunction with, crowdsourcing real devices requires building out a remote team of testers (through a vendor with a network of testers like Testlio or your teams) that use personal or provided devices in hand. This allows your development team to get a bird’s eye view of how the deliverable performs when in the end user’s hands.

There’s still a human element that’s really important, especially when you’re dealing with consumers who will leave you quickly if they don’t enjoy the experience of your application.

Summer Weisberg, Chief Client Officer at Testlio

Focusing on optimizing for scale will likely require a unique combination of virtual and in-hand testers, utilizing one or more vendors to reach the coverage area that makes you feel most comfortable releasing a new/updated product.   

Quality engineering, automation, and device architecture 

What do automation and quality engineering ask of device architecture? Your organization has likely already implemented a hybridized workflow focusing on manual and automated testing. Extend that to your architecture. Implementing device clouds and scaling with device crowds could readily supplant your manual testing phase while retaining the automation phase as needed. 

You need a device architecture that can support your QE directives. Device clouds bring value in shifting left, speed, time to market on scripts, and a broader coverage area. Device crowds get hyper-specific tests, UX runs, and a focus on localization and location. Having real-world data is invaluable. So how do you marry these two?

“Maybe you have two parallel flows. You’re automating some tests and manually running other tests to get human insights. Then, you get results where you can make really intelligent decisions on if you should move forward with this release, or what are your risks,” Weisberg said.

Your organization could make substantial gains by implementing a hybrid device architecture. You can readily scale the scope of your quality engineering while accounting for various unique scenarios that benefit from real users on real devices. 

Want more? Watch the entire webinar on demand here.

You may also like

  • Advice Top 7 Outsourced QA Companies in 2025
  • Advice 8 Test Automation Best Practices for Serious QA Teams
  • Advice Top Managed Software Testing Services in 2025
  • Advice How to Test AI Applications: Frameworks, Metrics, and Methods
  • Advice Automated Testing Strategy: How to Develop Yours [ +Template]
  • LinkedIn
Company
  • About Testlio
  • Leadership Team
  • News
  • Partnerships
  • Careers
  • Become a Tester
  • Platform Login
  • Contact Us
Resources
  • Blog
  • Webinars & Events
  • Case Studies
Legal
  • Notices
  • Privacy Policy
  • Terms of Use
  • Modern Slavery Policy

Subscribe
to our newsletter