Request info

Instrumented testing: how to unlock a new era of manual testing impact

Getting quality bugs is only half the QA battle—you need issue reports showing actual testing session insights that could negatively impact your users.

Without this information readily available at your fingertips, you have to ask the tester follow-up questions about the bug, or you have to work off sometimes wrongful assumptions.

What is instrumented testing?

Instrumented testing works like this:

  • You hook up Testlio’s SDK to your native iOS or Android app
  • Testers test your app and record their testing sessions.
  • Instrumented testing delivers loglines for every session, showing you the exact actions and app conditions at every step.
  • Bug submissions come pre-loaded with important behind-the-scenes data such as mobile environment and app activity.

In essence, you’ve got app analytics alongside not only every bug but also every testing session.

But what value does this sort of automated data collection provide?

Top 5 ways app instrumentation delivers more value from manual testing

To better understand this new form of testing analytics, let’s explore 5 ways instrumented testing allows you to unleash greater impact from all manual testing efforts.

1. Faster debugging

Smoother debugging is the main value that any mobile app team will realize from instrumented testing. Whereas before developers were in the dark about certain conditions and activity, suddenly everything is clear.

For engineers who are debugging issues, this looks like this:

  • No delays – there’s no need to ask testers for more information to try to reproduce the bug (so important when time zones are at play).
  • Fits testing habits and expectations – when engineers test code in their own testing environments before committing it, they typically have this sort of information. They would consult their debugger in their local staging and get insights into what happened. The loglines from instrumented testing provide this level of detail but from manual testing work.
  • Transparent – at the same time that it offers activity records, app instrumentation also provides automated screenshots and keyboard inputs, so there’s never any confusion or muddied waters about what actually happened during testing.

2. Consistency and time-savings with informative issue reports

However, sometimes testers aren’t able to provide all of the information that developers need. They might describe the mobile gestures and actions performed, without knowing the UI ID to include.

With instrumented testing, the same information is available with every bug submission. This information includes:

  • Network connectivity
  • Tester location (5-kilometer radius for privacy)
  • Device orientation (portrait or landscape)
  • Memory
  • CPU
  • Storage
  • Battery
  • Actions leading up to the bug

3. Usability studies

Testing analytics offers a ton of potential when it comes to usability. You might discover issues with how long it takes a tester to complete a certain task.

Just like website analytics gauges visitor satisfaction with sentiment benchmarks, so too are benchmarks required to get value from testing sessions. You’ll need to store testing sessions over time in order to find trends. If something repeatedly takes testers 50 seconds and you feel that it should only take users 20 seconds, then you’ve got a usability problem that you can explore further.

4. Manual test case optimization

Optimizing manual work is always smart. Can you help your team of manual testers save time on certain test cases so they unlock more time to spend on exploratory testing?

5. Manual test case to automated script conversion

The loglines provided by app instrumentation include the behavioral events, UI IDs, and conditions required for well-written test automation scripts. With some reviewing and editing, these loglines can be converted into scripts.

While the core value of app instrumentation is faster debugging, there are plenty more opportunities for utilizing the automatic collection of testing data.

Depending on your goals as an engineering team, you can make use of this new form of testing analytics in many different ways.

How to get started with instrumented testing

Currently, there are products on the market that offer metrics when testers submit bugs. However, they don’t record testing sessions and convert these sessions into loglines which can be analyzed and put to further use. Getting a recorded log of the entire testing session has a huge impact not only on your ability to reproduce bugs and debug faster, but this also offers the opportunity for deeper analysis into manual testing and usability.

  • Access to our global network of skilled testers
  • Ability to utilize your in-house employees alongside our freelance pros
  • Test case management and bug reporting
Jumbotron image