Networked Testing: a gentle unveiling

Today, we announced Testlio 3.0. Years in the making, Testlio 3.0 is a massive leap forward for our company. With new modules, a new UI/UX, updated workflows, new software engines, and a new integration framework, Testlio 3.0 powers multiple possibilities for our clients, our services professionals, and our global network of testers.

We bring Testlio 3.0 to the market today fully aware of the magnitude of the global health crisis surrounding us. As you might imagine, we even considered delaying our launch. Yet given how Testlio 3.0 works, how it supports testers around the world, and how it’s designed to test digital apps (on which nearly all of us are now even more dependent), we decided to humbly proceed with the launch. Our hope is that Testlio 3.0 will create value for our clients and their users.

As a company, Testlio was born out of triumphs, possibilities, and frustrations with crowdsourced testing (“crowdtesting”) models. We love some elements of crowdtesting. Scale. Flexibility. Distribution. Software. Others—like payment models, individual working approaches, and reliance on inexperienced testers—we think are flawed. 

We have resisted calling ourselves a crowdtesting company. Our model is different. We believe that Testlio is an evolution of crowdtesting. 

At the same time, we haven’t done a very good job of saying what we are (versus describing what we evolved from). Today, we are gently putting a stake in the ground. We are a managed app testing company. And our approach is called networked testing.

We hear what may be your first reaction. “This sounds kinda like they are testing networks.” Sorry about that. We debated this linguistic challenge. We landed on the hope that the “-ed” (networked) would come through. It might not be clear enough. But we’re giving it a try nonetheless.

If you’re still with us, let’s unpack what we mean by networked testing. Underneath the concept are four tenets. Whatever we (you) call what we do, these are strategic and philosophical and, well, existential for Testlio as a company. 

The four tenets are:  

  • Expert human testers: we hold that the best functional testing results come from professionals. All members of the Testlio network are vetted, validated, screened, project-specific, proven. Only 3% of tester applicants are accepted to projects. Testers are paid by-the-hour at better-than-market rates. We pledge to attract, retain and reward the best testers on the planet.
  • Burstable swarm teams: a core element of networked testing is the use of on-demand teams of 10-50+ testers per major run. Selected and consistent testers burst into action quickly, swarm the testing surface, and then intentionally go dormant until the next major cycle. Different from individually-oriented bug bounty testing models, networked testing manages and incentivizes teams to cooperate for work distribution, collaborative problem solving, and rapid reproducibility checks. 
  • Compressed testing windows: for networked testing to serve the demand of modern, agile engineering teams, testing must happen quickly. Three core concepts are important: first, the work must be bite-sized, generally broken into multi-minute or small-hour chunks. Second, the testing window must be short. Third, major testing should occur in the crevices of engineering teams’ work. For many companies, this means testing primarily happens overnight and/or on weekends—in relation to primary engineering team time zones. To optimize this approach, a multi-time zone team of testers is often optimal.
  • Connected real systems: finally, the entire testing ecosystem must be tethered and software-powered. Builds need to be ingested. Dozens of real hardware devices in target geographies must be used to represent true user experiences. Issues need to be structured, passed to third-party tools like JIRA, and highly actionable. Everything must be traceable for reporting and insights.

We’ve found that networked testing is valuable for multiple forms of software QA testing. It is especially powerful, impactful, and economical for native consumer apps (which often release every 1 or 2 weeks). It is also generally faster, easier, and less expensive than automated functional testing. It can be applied to the simplest of smoke tests and to the most robust of regression passes.

As we write about frequently, we are fans of automated testing—especially at the unit test level. We hold that automated unit testing should be a core function of engineering.

But often we’ve seen clients struggle with automated functional and UI tests. Scripts sometimes lag agile teams and can be flaky. Teams can be burdened by too many false positives or negatives. The overall costs of quality efforts may go up compared to manual testing options.

The world of testing continues to evolve. As you consider your testing strategy going forward, might the concepts of networked testing resonate? Might they be worth trying and/or leaning in to further? 

We’d love to hear what you think. Please contact us with any questions, ideas, challenges, or additional thoughts. Happy to hear an alternative phrase to networked testing if you have one too!