How to create a structured exploratory testing strategy

Kassidy Kelley , Managing Editor

January 4th, 2022

If you want your QA to go above and beyond basic expectations, structured exploratory testing is the secret weapon to elite functionality and good UX. When testing web and mobile apps, executing functional, regression-based scripted test cases is the minimum standard for ensuring that your software performs as expected. However, scripted functional testing can only validate if the system works as intended and can easily miss edge cases you wouldn’t even think to test. The room to be creative with exploratory testing can create richer feedback than a simple “yes, it works.” 

But the freedom of exploratory testing comes at a risk of random, ad-hoc results with no organization or uniformity across testers, test managers, and projects. While testers shouldn’t be constrained in their efforts to perform exploratory testing, creating a structured practice is an excellent tool for developing feedback, finding bugs, and improving UX. Here are some best practices on creating a systematic approach to exploratory testing that nets valuable and specific test/bug reports.

Set expectations on exploratory testing

Exploratory testing is a creative, experience-based approach in which design, execution, and learning are simultaneous activities. Test case designing, planning, and executing happen concurrently as the tester uncovers the app. The results of the tests conducted are used to design further tests. 

Exploratory testing doesn’t mean that testers launch into testing the app at random (this would be “ad-hoc” testing). Testers are usually strategically split into teams that assess the app from multiple angles, such as feature sets, actions and outcomes, account types, and more. 

For example, when testing an app for managing a family calendar of tasks, events, appointments, and to-do items, a tester needs experience with what features might be prone to cause bugs, such as complicated mobile gestures. They also need to stay in the moment and creatively assess new testing opportunities, such as adding two collaborators to a to-do list instead of one to test the appropriate notifications. 

For this reason, structured exploratory testing acts as a great compliment to functional testing when it is difficult to write test cases for specific features or to assess the overall user experience. We have a comprehensive guide to exploratory testing to check out as well.

Teach testers to be explorers

Many exploratory testing techniques and approaches can spark creative and engaging test scenarios that find complicated, overlooked bugs. Some great testers will seamlessly switch from planned regression test cases to a more creative exploratory structure, but some may struggle. To light the exploratory spark, teach your testers to:

• Create user stories. Put yourself in the shoes of one of your typical users (you could use a particular buyer persona for this). Follow their journey through the software and look for user-specific bugs.
• Ignore expectations. Create chaos! Don’t follow the precise steps outlined in the usual scripted test. Click three times where it says two. Go back a few steps, or leap far ahead. Use the back button or typical browser shortcuts. Change values, change your mind, change the page. What happens when you go off the beaten path?
• Make a mind map: Plot out every possible scenario you can think of based on the constraints you’ve been given, then go down every path.
• Try to break it: Use your experience with other similar applications to see if the software you’re testing has similar bugs. Don’t just confirm if something works – try to find the defects.
• Senior Testing Director at Testlio, Rudolf Elbrecht, also recommends using oracles and heuristics, like the Heuristic Test Strategy Model (HTSM), to shift testers towards exploratory testing. The HTSM is a guideword heuristics designed to help testers think about test strategy. One section, quality criteria categories, can give your testers insights into specific target categories to start exploring. 
• Capability. Can it perform the required functions?
• Reliability. Will it work well and resist failure in critical situations?
• Charisma. How appealing is the product design and functionality?
• Scalability. How well does the deployment of the product scale up or down? 
• Compatibility. How well does it work with external components and configurations?

The main difference between specific regimented test cases and exploratory testing is the guidelines and supporting material. Take the pressure off. Highlight that testers are already constantly learning, amending the tests, and reiterating even during scripted test execution. 

“When executing exploratory testing at Testlio, we give testers a certain mission or a focus without too many detailed steps and outcomes. Testers can use their own oracles and heuristics to plan, execute, and report findings,” Elbrecht said.

How To Structure Exploratory Testing

All testers should follow a structured exploratory testing process to develop a clear mission and a thorough understanding of requirements. These are tracked and documented for assessment after the test if any issues or questions arise. Add structure to your exploratory testings through a five-part session-based test management path:

1. Bug classification: The bugs and issues most commonly found in past projects are categorized, and the root causes are sought out. Based on these bugs, testers develop ideas for testing in the next run.
2. Creation of a test charter: Charters are statements that guide testing rather than scripts to follow. A team charter explains the plan, including where to start, what to test, how to test, who should perform what function, and other factors to consider. This helps distribute testing to cover the whole application.
3. Timeboxing: Two testers work together, uninterrupted, for 90 minutes. Addition or reduction of 45 minutes is allowed depending on the circumstances. 
4. Review: The test results are reviewed by assessing the issues found. Learnings from the test are recorded in test and bug reports, where the test coverage is analyzed.
5. Debrief: Finally, the results are compiled and compared with the initial charter. Additional testing may be recommended if necessary. 

Offer the right tools to support exploration

When creating a more systematic exploratory testing approach, QA leads should decide on documentation and test management tools and universally roll them out for testers to use. Since there is no pre-designed test document in exploratory testing, testers and team leads have more freedom and flexibility in choosing tools. Here are some exploratory testing tools for your toolkit: 

• Nimbus – Because Nimbus makes it easy to capture sections of a page and scrolling screenshots, it’s an excellent tool for testers.
• Loom – Though the screen-video recording tool Loom has exploded in its use cases and popularity, it began as a testing tool, and it is still a great one for filming the reproduction of bugs. 
• RapidReporter: This standalone tool allows testers to take notes during an uninterrupted testing session, with easy top-of-screen access and on-demand screenshots that can be exported in a CSV text file.
• Test management system – On a test management platform like Testlio, testers can follow an exploratory test structure with detailed instructions, including focus and out-of-scope areas. This ensures testers explore the parts of the app that make sense for the client and avoid wasting time on things that aren’t relevant. Furthermore, there should be transparency with all testers on the team, so everyone can see and search reports to avoid duplication. 
• Any.do– This timeboxing app can sync across entire teams, devices, and workspaces to track time per project, feature, or test case.