What is Exploratory Testing? QA without a map If regular scripted software testing follows clear directions on a map through your project, then exploratory testing starts with a compass and a blank map. There are software bugs that can’t be discovered by following a map: skilled explorers must create their unique map. Every project can benefit from having exploratory testing seeking out the hidden bugs lying off the beaten path. Here, we’ll discuss different ways of performing exploratory tests, creating test plans, some challenges you’ll face, the skills testers need, and where you can go to dig even deeper into the world of exploratory testing. Table of Contents What is exploratory testing? History of Exploratory Testing Why use exploratory testing Differences between scripted and exploratory testing Types of Exploratory Testing Unstructured exploratory testing Structured exploratory testing How to create a structured exploratory testing strategy Pros and Cons of exploratory testing Advantages Disadvantages Stages of exploratory testing 1. Strategize coverage before the test cycle 2. Set them off with precise requirements for documentation 3. Review submitted notes at the end of the cycle 4. Maintain open communication between testers and engineers Staffing an exploratory testing team What is exploratory testing? I told you what exploratory testing resembled: trailblazing — setting off into the uncharted wilderness. But what does that mean in more concrete terms? Exploratory testing searches for bugs without using pre-written test cases. Testers get information about what the project does and how it should perform and then are allowed to design tests as they go, following their intuition and experience to find bugs outside where scripted testing looks. Exploratory testing is an approach that uses scenarios generated ‘on the fly’ during the testing process. The more products a tester has tested, the more skills they have in their arsenal to predict potential issues – thus rendering the scenarios generated more productive. History of Exploratory Testing In the early days of software development, testing focused on following predefined test plans and executing scripted test cases. This approach was more formal and structured but often needed to catch up on subtle defects and user experience issues. In the late 1990s, Cem Kaner, a prominent figure in software testing, introduced the concept of session-based testing. He emphasized the importance of exploratory testing by encouraging testers to design and execute tests in a timeboxed session, capturing their observations, and sharing them with the team afterward. James Bach, another influential figure in software testing, further popularized exploratory testing as a disciplined and skillful approach to learning about the software and finding issues through active exploration and experimentation. With the rise of Agile methodologies in software development, exploratory testing gained more prominence. Agile teams embraced the iterative and adaptive nature of exploratory testing as it aligned well with the Agile principles of flexibility and responsiveness to change. Over time, various techniques and approaches have been developed to enhance exploratory testing. These include session-based testing, scenario-based testing, error guessing, and domain-based testing. These techniques provide testers with different perspectives and strategies to uncover defects. Need help prioritizing QA in your Agile planning? Head here. Why use exploratory testing It’s simply impossible to cover all the functional requirements of a system with test cases that follow a predefined script. So exploratory testing adds another verification layer by helping unearth issues not covered by test cases. Exploratory testing provides rapid feedback at the early stages of software development and reviews the product characteristics from a user perspective. Exploratory tests are also instrumental in uncovering impactful quality issues when teams need more time to prepare structured test cases. Differences between scripted and exploratory testing During a scripted test case, a senior-level quality assurance engineer has to take the time to write the scripts for other testers who use a detailed test plan filled with explicit instructions to complete the tasks with as little deviation as possible. Exploratory testers are provided with a charter of what to test, documentation of how the project works, and guidance for what areas to start with, behave like an improv troupe. They take off and go where their testing leads them, digging into places where they’ve seen problems before and trying out behaviors that found bugs in other projects. Instead of being partitioned out by a manager/test lead, exploratory planning, charting, assigning, and executing test cases happen simultaneously. This doesn’t mean that testers have no plan; it means that new scenario ideas are based on the actions of the product itself. Exploratory testing is crucial because even the best-scripted testing approaches have limitations. A scripted approach can only verify the parameters set within the script. So adding exploratory tests throughout the application development process is vital to ensuring no critical bugs/issues have slipped the net. Using exploratory testing in combination with other methods, such as running manual or automated test cases, performing sanity, and even performance or load testing (when needed), ensures better app quality and user experiences. Even companies that strongly favor scripted testing use exploratory methods. A common strategy is to let scripted testing solve most issues, saving exploratory testing until the end. Since testers act like users, it’s a final draft. A QA manager writing a script will also perform exploratory testing before writing it to be sure it’s ready or while writing it to get new ideas. Exploratory proponents agree that manual hands can be helpful for regression testing. To ensure a new feature isn’t breaking an old one, set procedures are often faster (allowing for more interaction with the new feature itself). Types of Exploratory Testing Unstructured exploratory testing But sometimes, the bugs live on the edges of functional areas or are the product of combining functions. Allowing your experienced testers time to probe any surfaces of the project can turn up unexpected defects in those places. Letting explorers’ follow their nose’ in unexpected directions can bring surprising benefits. While not the best choice for all your exploratory testing resources, allowing a team loose for a ‘bug-bash’ can find the bugs hiding out in the weeds where focused, area-specific testing will miss them. Timeboxed exploratory testing will prevent you from wasting time but still allow investigation of the spaces where testers have a gut feeling there might be bugs. Structured exploratory testing Thinking you might have to give up all control of your testing process if you use exploratory testing? No fear; you can still direct what functionality is being tested in any specific session and decide how much emphasis to put on an area throughout testing. Just like sending explorers out into unknown territory, you point them in the direction you want to be explored. Using a structured approach to exploratory testing, you break your project down into areas and direct your test team to concentrate on those based on your risk analysis. Breaking your sessions down to specific time windows, or timeboxing, allows you to emphasize testing by spending more time on an area rather than writing more tests for that feature. Testers will report defects found and the number of things they tried that worked adequately, providing another quality metric beyond bugs. How to create a structured exploratory testing strategy While you won’t provide a map of how to test as you would in scripted testing, you need some guidance. Letting the team charge off in all directions nets fewer defects than creating a test charter, a plan for who will test what and for how long. Keep your explorers from covering the same ground over and over. Create a set of user personas for your testers so they can choose activities that match up with what an actual user might do. Give them broad capabilities you want them to check. Functionality – Does it do what it should? Reliability/Robustness – Is it resistant to failures? Appearance/UX – Does it look good? Is the interface intuitive and easy to use? Scalability – Does it scale for more users? Compatibility – Does it work well with different hardware or add-ons? Define the rules in advance. Create clear bug classifications based on previous projects and root causes. Create a specific charter for who tests what and when. Timebox the testing sessions. Review results as they come in, documenting issues and new information. Circle back to compare your results with your test plan, allowing you to add more test areas pointed out by your finds or to increase attention on unexpectedly buggy sites. Pros and Cons of exploratory testing Exploratory testing allows flexibility and adaptability, enabling testers to respond quickly to changing requirements. It also promotes extensive test coverage by exploring various scenarios and user perspectives, uncovering defects that may have been missed in pre-defined test cases. However, its limitations include the lack of documentation, the subjective nature of exploratory testing, and potential staffing challenges. Advantages Rapid product feedback. Writing scripts requires an extensive amount of time. What’s more, the app has to be perfectly ready for use to warrant that initial test design phase. Exploratory testing is a definite plus for apps currently being developed and changed. Create new test cases to override obsolete ones (in the context of CI/CD processes, they almost always are!). Awesome testers are highly creative. They’re scientists and super-sleuths. By choosing exploratory testing, you invest in your team because they are way more invested in the project. Exploratory testing is fun and allows your team to use the full extent of their intelligence. Uncover more complex issues. A session-based approach allows testers to be method actors. They get in the heads of a particular user for each session. Say, a male user from China. Or a new user unfamiliar with this type of app. Sometimes sessions are based on other themes like areas of the app or cross-related functions. Disadvantages While there’s much to be gained from exploratory testing, it can be challenging to perform effectively, just like covering rough terrain is harder than gentle plains. No matter how expert your testers are, they must become familiar with the project being tested. The more they understand it, the more likely they can find bugs in it. Without scripted tests, it can be difficult to reproduce a defect. Make sure they have tools to aid in recording their steps, automatically if possible, so developers can see how they found a bug. Reporting where they tested, and the many positive results can be difficult. Provide robust reporting tools tailored to exploratory testing. Determining what features to test and whether to rely more on scripted testing should be done every cycle. Perform systematic risk analysis to see where defects can hurt you and concentrate testing there. Stages of exploratory testing As a quality assurance manager, you want to be flexible and allow your testers to be creative. But you also need structure to ensure coverage and great results. Here’s how to manage exploratory efforts that strike a balance. 1. Strategize coverage before the test cycle Look at the app or site, strategize, break it down, and then assign the specifics to your team. Assign: Focus points. Our test cycles last 24 hours and typically have specific focus points. If a new test cycle is devoted to taking videos, then there’s the focus. Other examples are new login integrations or updated user settings. We put these focuses in the cycle details. Task lists. Another way we guide testers is to give them test suites separated by areas or features. Within each test suite is a task list covering everything within that function—think of it like a checklist of everything a user can do within that product area. Pass/fail tests. No, tests with predefined steps and only pass/fail results are NOT exploratory. But even when using a primarily exploratory approach, it can be essential to have specific, pre-written tests for some areas of an app, like the vital functions. While some testers perform these critical actions, others can cover possible combinations. Other ways that QAMs can provide frameworks? Consider giving testers a detailed user persona. great user story. 2. Set them off with precise requirements for documentation Make sure your testers know how to submit bugs. Include steps with resulting behavior, a screenshot or video, and a log (unless it’s a UI issue). We want to make it easy for engineers to reproduce errors. Having transparent submittal formats means you don’t have to track your testers down for more info at the end of a cycle. Beyond Jira tickets, you need some form of notes. This way, your testers can tell you everything they did. When you have a remote team, oral debriefs and reports won’t likely be your go-to, but if you’re located in a central office, you can have a ten-minute conversation with each tester following their session when everything is fresh in their minds. 3. Review submitted notes at the end of the cycle Now that the testing is done double-check what has been tested. Quickly check that each tester explored the assigned focus points or features, then step back and compare the logs for coverage. If you find a couple of combinations missing, write up scripted tests and assign them in the next round. 4. Maintain open communication between testers and engineers Excellent communication skills are a necessity for great exploratory testing. The communication doesn’t stop there once they’ve reported all issues, bugs, and session notes. The back-and-forth dialogue will help them adjust to future testing cycles or projects. They’ll learn what irks the engineers, gain new insights into the project, and clarify the jargon of the app’s features. Staffing an exploratory testing team As you might have gathered, exploratory testers can’t be average or below average. You want seasoned explorers rather than someone who’s never been outside a city. Skilled and experienced testers work best on exploratory testing. With a background in the abstract concepts of testing and a grounding in the nuts-and-bolts work, a professional tester develops an intuition on where to look for bugs. Testers skilled at learning new tools and projects will fit into your testing paradigm smoothly and pick up the operation of your project quickly, letting them provide value early. Finding skilled testers can be difficult. Great testers are developed through years of practice, adept industry knowledge, and employer investment into education, not a frequent combination. Entry-level testers are undoubtedly easy to crowdsource, but finding qualified testers has been much more complicated for some organizations. This poses a significant QA challenge because, despite the market changes regarding talent, the needs and obligations of software development still have to be met. Many organizations have had to look for outside help, predominantly with freelance testers. Solving the growing crowdsourced testing shortage at your org requires a blend of awareness, third-party help, and some creativity. We’ve spent over 2,000 words discussing how following a map differs from exploring the unmapped territory and how that relates to testing. Scripted and exploratory testing have their place as valuable tools to flush out defects and measure the quality of a project. Using exploratory techniques on rapidly changing tasks, such as additional checks with scripted testing or digging deeper into more complex operations, will yield different bug finds and enhance your scripted testing. The drawbacks include needing a fresh style of test management and requiring more experienced testers. Balancing these benefits and costs will tell you what mix of tests you need, but remember, you always need some explorers to break that new ground beyond the edge of the map.