Instrumented testing: the answer to faster bug reproduction and issue resolution

Dayana Mayfield , Dayana is a QA engineer turned technology writer living in Milan, Italy. She's always down for a smoothie.

February 4th, 2021

With product analytics, you collect data about how your users are interacting with features and how frequently they use your app.

With website analytics, you know your top pages, devices, and locations, and you can drill down into activity per visitor.

But what about manual app testing analytics?

In this post, we explore instrumented testing, the newest feature in the Testlio platform, and how testing data collection helps engineering teams get even more value from skilled, manual testers’ activity.

Why you need to collect data from manual app testing

Most product and engineering teams aren’t collecting data from their manual testing because historically, testing management systems have been disconnected from targeted software experiences. While there is software that adds behind-the-scenes data to bug reports, there hasn’t previously been a testing analytics solution that records entire testing sessions.

There are so many reasons to collect data from manual testing, and certain ones will stand out more to you than others based on your company’s initiatives for utilizing data-led insights.

The top reasons are:

• More detailed bug reports – data from manual testing can help you understand the behavioral and contextual factors leading up to an issue.
• Transparent app activity – with any testing process, consistency is key. With automatically-collected testing data, you always have access to every app interaction, including UI IDs, login steps, etc.
• Potential for deeper analysis – having this data sets the stage for further analysis. How you’ll use it depends on your manual testing processes and goals. Below, we explore the potential for usability studies and automated script writing.

How app instrumentation collects manual testing data

Through easy-to-deploy iOS and Android SDKs, instrumented testing automatically captures and conveys testing session information directly from mobile devices—enabling software development teams to gather contextual information quickly and efficiently.

As you might already know, via our collective testing feature, our testing management platform can include both our freelance testers and your own in-house team, so data is collected no matter who’s doing the testing.

Data is collected in two key ways:

• Bug submissions – when testers log a bug in Testlio Platform for apps that have set up our instrumentation SDK, the bug report will automatically load up important device data and tester activity.

• Session recordings – Apart from more valuable bug reports, data can be collected at any time. Testers can hit the record button to generate a logline of everything that happens in that testing session. This ensures that all of the information needed to reproduce bugs is available and also sets the company up for further analysis of trends and issues.

Enhanced visibility into tester actions and device telemetry deliver rapid bug triage and issue resolution for engineering teams. And just as you would examine product data, the possibilities for analyzing testing sessions are endless.

How to get the most out of networked testing with app instrumentation

With networked testing, you gain access to a global, vetted network of skilled testers, who are paid fairly for their work (not per bug like traditional crowdtesting models, which prompt low-quality issue reports for your team to sort through).

Networked testing offers several significant advantages, including the ability to increase your testing team’s size during important releases.

Now with instrumentation, product teams get not only high-quality bug reports but also automated data collection that helps them get the most out of every minute of testing activity.

Here are the top ways to use the data from instrumentation:

Debug faster with comprehensive testing data

First and foremost, your engineers will be able to debug faster. A defect tracking tool is only as good as the data entered into it, and fortunately, our instrumentation SDK kit allows for automated data collection for every bug.

Here’s just some of the information that instrumentation automatically collects:

• Device screenshots
• Network connectivity
• Tester location
• Device orientation (portrait or landscape)
• Memory
• CPU
• Actions leading up to the bug

Developers can access this information straight away, without having to follow up with testers to ask additional questions. Developers can also access the same level of detail as they would if running their code through a quick debugging scan, meaning that manual testing can meet their expectations for the information provided.

Analyze product usability

When collecting data on testing sessions, it’s also possible to analyze product usability. If it’s taking testers longer than expected to achieve certain tasks, then that might signal an issue for users as well.

Convert manual test loglines into automated testing scripts

App instrumentation yields a logline of everything that happened in a format that will enable future scriptless automated testing capabilities. Engineering teams can take the loglines from individual test cases and quickly turn them into automated test scripts.

For features that are stable, it’s important to handle a fair amount of regression testing via automation, in order to allocate more manual testing time towards new features, breakage-prone features, and user-centric exploratory tests.

Takeaways: why instrumentation is key to maximizing networked testing

Networked testing allows you to ramp up your testing team for big releases and have smaller levels of manual testing running around the clock.

It’s exciting to think of the potential for collecting data from everything your testers are doing. Immediately, the quick win is faster debugging, which offers a huge incentive to set this up.

And then there’s the future potential to turn that data into meaningful insights that ensure your mobile app is of the highest quality possible.

If you’re already a Testlio customer, talk with your account manager about setting up instrumentation with our SDK.