Instrumented testing: the answer to faster bug reproduction and issue resolution

With product analytics, you collect data about how your users are interacting with features and how frequently they use your app.

With website analytics, you know your top pages, devices, and locations, and you can drill down into activity per visitor.

But what about manual app testing analytics?

Why you need to collect data from manual app testing

Most product and engineering teams aren’t collecting data from their manual testing because historically, testing management systems have been disconnected from targeted software experiences. While there is software that adds behind-the-scenes data to bug reports, there hasn’t previously been a testing analytics solution that records entire testing sessions.

There are so many reasons to collect data from manual testing, and certain ones will stand out more to you than others based on your company’s initiatives for utilizing data-led insights.

The top reasons are:

  • More detailed bug reports – data from manual testing can help you understand the behavioral and contextual factors leading up to an issue.
  • Transparent app activity – with any testing process, consistency is key. With automatically-collected testing data, you always have access to every app interaction, including UI IDs, login steps, etc.
  • Potential for deeper analysis – having this data sets the stage for further analysis. How you’ll use it depends on your manual testing processes and goals. Below, we explore the potential for usability studies and automated script writing.

How app instrumentation collects manual testing data

Data is collected in two key ways:

  • Bug submissions – when testers log a bug in Testlio Platform for apps that have set up our instrumentation SDK, the bug report will automatically load up important device data and tester activity.
  • Session recordings – Apart from more valuable bug reports, data can be collected at any time. Testers can hit the record button to generate a logline of everything that happens in that testing session. This ensures that all of the information needed to reproduce bugs is available and also sets the company up for further analysis of trends and issues.

Enhanced visibility into tester actions and device telemetry deliver rapid bug triage and issue resolution for engineering teams. And just as you would examine product data, the possibilities for analyzing testing sessions are endless.

How to get the most out of networked testing with app instrumentation

Networked testing offers several significant advantages, including the ability to increase your testing team’s size during important releases.

Here are the top ways to use the data from instrumentation:

Debug faster with comprehensive testing data

First and foremost, your engineers will be able to debug faster. A defect tracking tool is only as good as the data entered into it, and fortunately, our instrumentation SDK kit allows for automated data collection for every bug.

Here’s just some of the information that instrumentation automatically collects:

  • Device screenshots
  • Network connectivity
  • Tester location
  • Device orientation (portrait or landscape)
  • Memory
  • CPU
  • Actions leading up to the bug

Developers can access this information straight away, without having to follow up with testers to ask additional questions. Developers can also access the same level of detail as they would if running their code through a quick debugging scan, meaning that manual testing can meet their expectations for the information provided.

Analyze product usability

When collecting data on testing sessions, it’s also possible to analyze product usability. If it’s taking testers longer than expected to achieve certain tasks, then that might signal an issue for users as well.

Convert manual test loglines into automated testing scripts

App instrumentation yields a logline of everything that happened in a format that will enable future scriptless automated testing capabilities. Engineering teams can take the loglines from individual test cases and quickly turn them into automated test scripts.

Takeaways: why instrumentation is key to maximizing networked testing

Networked testing allows you to ramp up your testing team for big releases and have smaller levels of manual testing running around the clock.

It’s exciting to think of the potential for collecting data from everything your testers are doing. Immediately, the quick win is faster debugging, which offers a huge incentive to set this up.

And then there’s the future potential to turn that data into meaningful insights that ensure your mobile app is of the highest quality possible.