Networked Testing: A new approach for the modern QA world
Testlio is the originator of, and leader in, networked testing.
What is Networked Testing?
Networked testing is both a testing approach and a mindset.
Regardless of where you are today, networked testing can help you leap forward. Fresh for 2020—yet seven years in the making—networked testing is an evolution of in-house, outsourcing, and crowdtesting best practices. It works for companies early in their quality journey and for firms advanced in their utilization of testing approaches.
Fundamentally, networked testing is a philosophy and set of practices that are tuned to deliver exceptional quality assurance. Speed. Coverage. Economics. It crosses capabilities like functional testing, usability testing, localization testing, and location testing.
Networked testing houses multiple connected concepts. Four major elements are team attributes, testing philosophies, engineering alignment, and technology utilization. Below, we unpack elements of networked testing and compare them to crowdsourced, outsourced, and in-house testing approaches.
Tester skills, experience, selection, consistency, flexibility, and location are important attributes of bespoke teams that are built to tackle testing challenges.
The best testing results often come from professionals. Unlike crowdtesting, participants in networked testing are vetted, validated, screened, proven. For the Testlio network, only 3% of applicants are accepted to client projects.
Vetted and trained testers are highly efficient. Unlike most crowdtesting approaches, networked testing uses a high percentage of consistent testers run-to-run…while also rotating new testers for fresh insights and perspectives.
Networked testing uses on-demand teams of 10-50+ testers per major run. Testers leap into action quickly, swarm the testing surface, and then intentionally go dormant until the next run. Capacity is available when you need it—and you don’t pay for idle personnel.
Networked testing relies upon remote, local personnel. Different from offshore outsourcing, networked testing can provide testers exactly where you need them: at stores, in stadiums, in metros, in countries, etc. These distributed testers can flex into assignments too, helping you with launches, market updates, major events, and more.
Networked testing uses collective methods, tuned management, shared tasks, and short assignments to optimize testing efficiency, and effectiveness.
With networked testing, members of the testing squad are managed. Networked testing can be a “do-it-for-me” approach that covers strategy, best practices, measurement, implementation, oversight, coordination, reporting and more.
With networked testing, the work must be bite-sized, generally broken into 1-2 hour chunks. With short assignments, large teams can swarm testing assignments quickly.
Networked testing encourages teams to cooperate for work distribution, collaborative problem solving, and rapid reproducibility checks. Most crowdtesting encourages isolated, individual work.
Different from crowdtesting bug bounty models, networked testers are paid by-the-hour, equipping them to do the hardest, most thoughtful, deepest quality work. At Testlio, we also pledge to pay testers better-than-market rates.
With networked testing, client and provider teams can collaborate for increased speed, shared responsibility, quick knowledge transfer, and exceptional economics. Co-managed experiences are unique to networked testing approaches like Testlio.
As part of a networked testing approach, Testlio’s collective testing reimagines how to organize multiple test teams with options for in-house, hybrid, or Testlio testers only models. Client employees, Testlio Services personnel and Testlio freelance Network can all work together on a single platform.
Testing and quality assurance initiatives must be tightly aligned with product and engineering teams. Speed, agility, and continuity are critical.
A set of SDKs and new platform capabilities automatically capture and convey testing session information—directly from mobile devices. We call this instrumented testing and it’s a natural extension of the networked testing approach. Enhanced visibility into tester actions and device telemetry delivers rapid bug triage and issue resolution for engineering teams.
With networked testing, run windows can be narrow. Unlike outsourced and in-house approaches with consistent, limited resources, voluminous testing can be delivered by humans at machine-like speeds.
With agile teams operating on 1 or 2-week sprints, networked testing can occur in the crevices of engineering teams’ work. For many teams, networked testing primarily happens overnight and/or on weekends. To optimize this approach, a multi-time zone team of testers is often optimal.
For CI/CD approaches, networked testing can hook into DevOps processes. Both manual and automated testing can be networked into code as it is checked in—ensuring that unit, visual, functional, and other tests are completed. Testing can also incorporate design critiques, early build feedback, and more—acting as a true extension to product development teams.
Networked testing scales through the smart use of real devices, testing management software, system integration, and traceable actions.
Networked testing uses in-the-wild devices. With Testlio, our network is committed to testing on any hardware device, in any location. Today, we can test on over 1,200 hardware device types in over 150 countries around the world.
Networked testing is software-powered—and uses a singular system to manage builds, tests, plans, runs, issues, teams and reports. Unlike in-house and outsourced testing approaches, which often use off-the-shelf tools, networked testing leverages a single bespoke, optimized, broad, and flexible software testing platform.
With networked testing, the primary test management platform connects with DevOps, issue management, business intelligence, monitoring, and other tools. All are tethered into a holistic ecosystem and managed by a primary service provider.
Networked testing provides unlimited licenses to the software platform. Client personnel can see everything happening and understand all moving pieces at play. Testlio’s iOS and Android app instrumentation SDKs automatically record and report nuanced testing insights, empowering engineering teams to diagnose and fix issues faster. Unlike other forms of testing, networked testing is highly transparent and holistically traceable.