Instrumented testing: how to unlock a new era of manual testing impact

Getting quality bugs is only half the QA battle—you need issue reports showing actual testing session insights that could negatively impact your users.

And if you have skilled manual testers to help you find these bugs, the next problem becomes…how do you fix them? For that, you need contextual information about the steps and environmental conditions leading up to the bug.

Without this information readily available at your fingertips, you have to ask the tester follow-up questions about the bug, or you have to work off sometimes wrongful assumptions.

In this post, we introduce you to instrumented testing, whereby you collect insightful data from testing sessions, and we present important ways this helps you get more value out of manual testing than ever before.

What is instrumented testing?

Through an easy-to-install SDK, instrumented testing is a convenient way to gather structured information about manual testing sessions directly from mobile devices. Being able to automatically capture and convey testing session data enables rapid bug reproduction and issue resolution for engineering teams.

Instrumented testing works like this:

• You hook up Testlio’s SDK to your native iOS or Android app
• Testers test your app and record their testing sessions.
• Instrumented testing delivers loglines for every session, showing you the exact actions and app conditions at every step.
• Bug submissions come pre-loaded with important behind-the-scenes data such as mobile environment and app activity.

In essence, you’ve got app analytics alongside not only every bug but also every testing session.

But what value does this sort of automated data collection provide?

Top 5 ways app instrumentation delivers more value from manual testing

To better understand this new form of testing analytics, let’s explore 5 ways instrumented testing allows you to unleash greater impact from all manual testing efforts.

1. Faster debugging

Smoother debugging is the main value that any mobile app team will realize from instrumented testing. Whereas before developers were in the dark about certain conditions and activity, suddenly everything is clear.

For engineers who are debugging issues, this looks like this:

• No delays – there’s no need to ask testers for more information to try to reproduce the bug (so important when time zones are at play).
• Fits testing habits and expectations – when engineers test code in their own testing environments before committing it, they typically have this sort of information. They would consult their debugger in their local staging and get insights into what happened. The loglines from instrumented testing provide this level of detail but from manual testing work.
• Transparent – at the same time that it offers activity records, app instrumentation also provides automated screenshots and keyboard inputs, so there’s never any confusion or muddied waters about what actually happened during testing.

2. Consistency and time-savings with informative issue reports

Skilled manual testers should be able to write a great bug report. They should consistently provide the expected information to help engineers debug quickly.

However, sometimes testers aren’t able to provide all of the information that developers need. They might describe the mobile gestures and actions performed, without knowing the UI ID to include.

With instrumented testing, the same information is available with every bug submission. This information includes:

• Network connectivity
• Tester location (5-kilometer radius for privacy)
• Device orientation (portrait or landscape)
• Memory
• CPU
• Storage
• Battery
• Actions leading up to the bug

What’s more, it’s faster than ever to include all of this information. Since you can use the Testlio platform with your in-house team, this is good news. They’ll be able to submit quality bug reports consistently and they’ll spend less time on each one.

3. Usability studies

Testing analytics offers a ton of potential when it comes to usability. You might discover issues with how long it takes a tester to complete a certain task.

Just like website analytics gauges visitor satisfaction with sentiment benchmarks, so too are benchmarks required to get value from testing sessions. You’ll need to store testing sessions over time in order to find trends. If something repeatedly takes testers 50 seconds and you feel that it should only take users 20 seconds, then you’ve got a usability problem that you can explore further.

4. Manual test case optimization

Optimizing manual work is always smart. Can you help your team of manual testers save time on certain test cases so they unlock more time to spend on exploratory testing?

This is something we care about a lot at Testlio. We want our clients to get the most out of their team of networked testers. By analyzing manual testing, we can discover inefficient test instruction, rewrite test cases to reduce complication or context-switching, and thereby free-up more time to spend on breakage-prone feature sets.

5. Manual test case to automated script conversion

Instrumented testing offers another way to get even more value from manual testing.

The loglines provided by app instrumentation include the behavioral events, UI IDs, and conditions required for well-written test automation scripts. With some reviewing and editing, these loglines can be converted into scripts.

While the core value of app instrumentation is faster debugging, there are plenty more opportunities for utilizing the automatic collection of testing data.

Depending on your goals as an engineering team, you can make use of this new form of testing analytics in many different ways.

How to get started with instrumented testing

Currently, there are products on the market that offer metrics when testers submit bugs. However, they don’t record testing sessions and convert these sessions into loglines which can be analyzed and put to further use. Getting a recorded log of the entire testing session has a huge impact not only on your ability to reproduce bugs and debug faster, but this also offers the opportunity for deeper analysis into manual testing and usability.

Instrumented testing is a new feature inside of Testlio’s testing management platform, which offers:

• Access to our global network of skilled testers
• Ability to utilize your in-house employees alongside our freelance pros
• Test case management and bug reporting

When you use the Testlio platform, you can automatically collect valuable data from our network testers and your own internal team.

Want to debug issues faster than ever with automatically generated testing logs? Request information about instrumented testing.