5 Lessons from over 100,000 Bug Fixes

Earlier this month we hosted a webinar with two of our finest QA managers, Jeff Scott and Kristi Kaljurand.

Jeff and Kristi shared some of what they’ve learned from reporting tens of thousands of fixed bugs — in essence, what makes high-quality software testing.

Want the full video experience? Check out the webinar recording here.

Without further ado, our lessons!

1. Flexibility makes amazing QA

“Flexibility is key in QA because as you know, we may receive a build request for testing at any time of the day, and that can be at the end of day 6PM,” Jeff says. “Right when we are ready to walk out the door, we receive a notification from the person of contact from our client saying ‘Can we perform sanity tests overnight’ and it’s key to have passionate flexible testers that have the knowledge and where-with-all, and is also familiar with the app that way we can easily pass down instructions, release notes, and so on. And they can provide us with excellent QA. With the results by morning. So all-in-all I believe just being passionate and also being a good listener and being able to be flexible is extremely key.”

2. Visual aids help teams understand test cases and their structure

“We like to build out something called a mind map, and this allows us to formally break out each and every feature and functionality,” says Jeff. “That will give us a better chance to create test cases and then formulate the proper test plan or task list.”

“It’s a good visual aid to see what are the main features of an app, and our clients really like to see it in this way,” says Kristi. “Actually those nice red dots can show you which features have the most issues. Like they say ‘A picture is worth a thousand words.'” 

3. Exploratory testing is awesome — but it needs structure 

“We do approach the testing structure in an exploratory way because regression testing only checks what you write down to check,” says Kristi. “Usually with regression testing, there’s not many new issues coming out. It only shows you what you write it to check. It shows you what’s supposed to work, works. The structured exploratory gives us, or our testers, to look at as an end user way, so how we write tests are basically can do something more, which allows our testers to think outside the box. If the expected result is written down, then they usually look at that. They don’t look outside.”

4. Issue reporting matters

“I think what frustrates developers is when they can’t really reproduce an issue, and I’m sure you have heard it, I’ve heard it,” Kristi says. ” So it is really important to try and write all the information down that the other person can reproduce the issue. Especially with working outside the windows, because you don’t have the privilege to go to the person and show that it’s not working on your machine. Writing down all the details, all the videos, I think our platform is a good one that helps developers too.”

5. There’s no “perfect” QA team structure

“It’s been proven time and time again that all structures work,” Jeff says “For instance, if you just have an internal QA team of two members that are just awesome individuals That can test web mobile connected devices all at the same time within the timeframe if an eight-hour workday, and get the work done then it’s fine. Or, if you have a structure of QA manager, a few test leads, an analyst, tester, all in-house, then you can also have vendors helping you out. It just depends on the actual work load and the platforms that’s needed coverage for.”