Request info

Networked Testing: Economic considerations and financial impacts

Should you build an in-house QA team, or should you rely on software testing companies?

The conversation around insourcing versus outsourcing looks very different for quality assurance than for other teams like engineering or product management. These teams offer very clear ROI from insourcing unless we’re talking about some side project that keeps getting deprioritized in the backlog, like a new API.

But when it comes to software testing and QA, enterprises can’t immediately declare that having a fully in-house team makes sense. The common benefits from insourcing — retaining company knowledge, owning processes — don’t outweigh the losses of attempting to take on full responsibility for app coverage.

How networked testing is unlike traditional crowdsourced testing

Both crowdsourced testing and networked testing are designed to give your company access to large groups of testers—and their personal devices.

Real devices are essential for app quality, whether we’re talking mobile apps, mobile-optimized websites, web-based apps, or desktop applications. For any type of app, there are device and environment factors at play.

Here’s how networked testing and crowdsourcing are similar:

  • Burstable teams: when you have an important release or want to make a better effort at device or localization testing, you can ramp up coverage with dozens or even hundreds of testers for a short period of time.
  • Global distribution: you can work with testers all over the world. This offers the benefits of weekend testing, nighttime testing, affordable testing, real device coverage, and of course localization.

But the two methods of outsourcing differ in important ways:

  • Expertise: much like gig sites, traditional crowdsourcing platforms offer anyone with interest in a service, the ability to offer this service. These freelancers aren’t usually trained, experienced, or skilled. However, Testlio’s networked testing service only includes vetted and experienced testers. Today, a good portion of new testers in our network come from referrals from testers who have been with us for years.
  • Consistency: with crowdsourced testing, the consistency of the bug reports can vary wildly. One day you might get a handful of high-quality bugs. The next day, you’re getting duplicates and bug reports that lack important reproduction details. The reason for this is that most crowdsourcing platforms don’t vet testers and pay per bug. On the other hand, networked testing offers very consistent bug reports because testers are trained in bug reporting and in working together. Testers are also paid hourly, not per bug. This is an important distinction because testers become more thoughtful and identify higher-quality issues. This, in turn, saves engineering teams from wasted time sifting through low priority bugs. What’s more, companies using Testlio instrumentation SDKs benefit from automated screenshots and session logs with exact reproduction details and UI IDs.

Furthermore, Testlio’s networked testing service offers different options for QA and testing management, which we’ll explore later on in this article.

Economic considerations of networked testing vs. in-house QA

Now that we’re clear on the quality and value of networked testing, we can have a better conversation about how it compares to in-house QA.

Let’s take a look at a small and simple example. There are 20 test run scenarios. It would take an in-house QA 10 hours to test all scenarios. However, 20 Testlio testers could get this work done in half an hour.

This speeds up the time for your release. Especially, when nights and weekends are at play, using networked testing can help you release much, much faster.

If you were to spend $60,000 to $100,000 employing an in-house QA tester for the year, you would be constrained by their time availability. However, the same investment spent on networked testing would provide constant coverage by multiple testers around the world, allowing you to ship updates to customers as quickly as possible.

I like the flexibility. For example, we’re about to enter a heavy month, and I like that Testlio has a network of testers, and I don’t need to recruit a QA person. Or, worry about any surprises. My goal is to worry less about having to recruit QA people or worry about how long the QA team needs to get someone else ramped up on the project.

Technology Consultant, Health, Wellness & Fitness app

Methods for combining in-house resources with networked testers

Having an entire in-house QA team simply doesn’t make sense for companies that value speed. However, that doesn’t mean that having an internal QA leader isn’t a bad idea. An in-house QA manager can coordinate testing efforts with engineering, lead the charge with automated testing, strategize test coverage, and more.

Here are some ways that you can work with networked testing. When you maintain some QA leadership in-house, you can get those benefits of process and knowledge ownership at a high level, without taking on the burden of app and device coverage.

Fully-managed QA services

  • What this looks like: An engineering lead partners with a networked testing vendor for all QA services.
  • Best for: Small apps, new startups, small user bases. Companies that don’t have a budget for an in-house QA manager as well as app, device, and location coverage.
  • Financial implications: For small user bases and apps, you’ll save money outsourcing QA management, strategy, and test case writing in addition to manual test execution.

QA management and test execution

  • What this looks like: A QA manager or engineer partners with a networked testing vendor for QA management and test execution, but not automated testing.
  • Best for: Enterprises that want to insource automated testing, but don’t have the need for a full-time role handling QA strategy and management.
  • Financial implications: If you have an engineering person who can manage automated testing in addition to their core work and your test coverage strategizing doesn’t require a full-time job, this can help you save money. Your engineering team or in-house QA can handle automated testing, and you can work with a networked testing company to strategize test coverage and create and assign test cases to manual testers.

Test execution only

  • What this looks like: QA manager partners with a networked testing vendor for test execution only.
  • Best for: Large enterprises with two or more QA managers on staff.
  • Financial implications: If you have a large app that requires extensive automated testing and test cycle strategy, you’ll save money by bringing QA management and automated testing in-house, but outsourcing the execution of manual testing.

Final considerations when building your perfect QA team

Ultimately, there are a few things you need to consider when it comes to choosing the right allocation of insourcing and outsourcing:

  • Needs and desires for speed of testing and release cycles
  • Time requirement for strategizing testing cycles and writing test cases
  • Time requirement for executing on standard testing cycles
  • Regression testing needs
  • Automation testing needs, and whether they constitute a full-time role
  • Needs for global device coverage and localization

When you consider all of the above, you’ll be able to come to a conclusion on what roles should be handled internally (because there truly is a full-time job requirement), what roles are important but don’t require 40 hours per week, and what roles would slow your company down if you did manage them in house.

Jumbotron image