Request info

Networked Testing: a gentle unveiling

We bring Testlio 3.0 to the market today fully aware of the magnitude of the global health crisis surrounding us. As you might imagine, we even considered delaying our launch. Yet given how Testlio 3.0 works, how it supports testers around the world, and how it’s designed to test digital apps (on which nearly all of us are now even more dependent), we decided to humbly proceed with the launch. Our hope is that Testlio 3.0 will create value for our clients and their users.

As a company, Testlio was born out of triumphs, possibilities, and frustrations with crowdsourced testing (“crowdtesting”) models. We love some elements of crowdtesting. Scale. Flexibility. Distribution. Software. Others—like payment models, individual working approaches, and reliance on inexperienced testers—we think are flawed. 

We have resisted calling ourselves a crowdtesting company. Our model is different. We believe that Testlio is an evolution of crowdtesting. 

We hear what may be your first reaction. “This sounds kinda like they are testing networks.” Sorry about that. We debated this linguistic challenge. We landed on the hope that the “-ed” (networked) would come through. It might not be clear enough. But we’re giving it a try nonetheless.

If you’re still with us, let’s unpack what we mean by networked testing. Underneath the concept are four tenets. Whatever we (you) call what we do, these are strategic and philosophical and, well, existential for Testlio as a company. 

The four tenets are:  

  • Expert human testers: we hold that the best functional testing results come from professionals. All members of the Testlio network are vetted, validated, screened, project-specific, proven. Only 3% of tester applicants are accepted to projects. Testers are paid by-the-hour at better-than-market rates. We pledge to attract, retain and reward the best testers on the planet.
  • Burstable swarm teams: a core element of networked testing is the use of on-demand teams of 10-50+ testers per major run. Selected and consistent testers burst into action quickly, swarm the testing surface, and then intentionally go dormant until the next major cycle. Different from individually-oriented bug bounty testing models, networked testing manages and incentivizes teams to cooperate for work distribution, collaborative problem solving, and rapid reproducibility checks. 
  • Compressed testing windows: for networked testing to serve the demand of modern, agile engineering teams, testing must happen quickly. Three core concepts are important: first, the work must be bite-sized, generally broken into multi-minute or small-hour chunks. Second, the testing window must be short. Third, major testing should occur in the crevices of engineering teams’ work. For many companies, this means testing primarily happens overnight and/or on weekends—in relation to primary engineering team time zones. To optimize this approach, a multi-time zone team of testers is often optimal.
  • Connected real systems: finally, the entire testing ecosystem must be tethered and software-powered. Builds need to be ingested. Dozens of real hardware devices in target geographies must be used to represent true user experiences. Issues need to be structured, passed to third-party tools like JIRA, and highly actionable. Everything must be traceable for reporting and insights.

But often we’ve seen clients struggle with automated functional and UI tests. Scripts sometimes lag agile teams and can be flaky. Teams can be burdened by too many false positives or negatives. The overall costs of quality efforts may go up compared to manual testing options.