Why QA Testing Fails…and how to fix it: a Q&A with Kristi Kaljurand

, Managing Editor, Testlio

In today’s software-driven world, organizations are moving faster than ever to deploy new features and software fixes – because getting product updates into the hands of users at speed reduces the chances of risk and increases business value.
But rolling out continuous app changes while maintaining quality throughout the whole development process is not without considerable effort, and bottlenecks can occur even with the most well-meaning QA testing practices. 

Why is that? 

An excerpt from the 2019 book Enterprise Continuous Testing seemingly sums up the situation: “DevOps is all about removing the barriers to delivering innovative software faster. Yet, as other aspects of the delivery process are streamlined and accelerated, testing consistently emerges as the greatest limiting factor. A recent GitLab survey that targeted developers and engineers found that testing is responsible for more delays than any other part of the development process…. ” But while it can be easy to assume that QA teams simply need to test faster, the reality is more complicated.

To delve deeper into this topic, we asked our Senior Engagement Manager Kristi for her perspective on what could be some of the root causes of a lagging QA process.

About Kristi
Eleven years ago, Kristi started her professional career as a software consultant in a local Estonian IT company. After a few years, she took on a QA engineer role, and that’s how she discovered her true passion – software testing.
Kristi first joined Testlio as a freelance tester in January 2013. Soon after that, she started to work as a Community Test Lead and in May 2014, she joined the Testlio team on a full-time basis. She now serves as one of Testlio’s most Senior Engagement Managers.

Before joining Testlio, my background was in web testing, where things are quite black and white. If there’s a bug, then most likely it’s a bug. But in the mobile world, it’s very different. There are so many variables. Even if one person can reproduce a bug 100% on a given device, another one might be unable to reproduce the same bug. And something that can’t be reproduced predictably is going to be hard to fix.

Kristi Kaljurand, Testlio Senior Engagement Manager

Related: Meet Kristi

In what situations can QA testing fail companies?

In my experience, it usually boils down to a very simple matter: miscommunication and/or misaligned expectations. If the client’s expectations haven’t been mapped out or understood correctly, it’s safe to say that no matter the results you deliver, they will not be happy. I’ve seen this happen, especially when a well-meaning contact on the client’s side doesn’t have a QA background or an understanding of the testing process.
Here are three main situations where testing can fail to deliver exceptional results:

1. When focusing too much on one type of testing

Smoke testing is a great example.  It’s a preliminary level of testing that helps ensure the stability of the application. By design, smoke testing helps to quickly understand if the application ‘catches fire’ without going too deep into the testing process (hence the name ‘smoke tests’). The risk of only performing smoke tests is that it’s easy to miss critical bugs. 

The same applies to regression testing. Regression tests are usually very detailed and written to detect a particular bug after updating or modifying an app.  But, when you execute regression tests to evaluate that nothing breaks, you only make sure that the existing functionalities are working as expected. It’s akin to being a horse with blinders on, as your peripheral vision is limited, and you can only focus on a very narrow field. For example, you can’t expect regression testing results on par with exploratory testing.

2.  When testing in a hurry

Even with tight deadlines and narrow timeframes, spending time to define the testing scope properly is critical. Sometimes, the client will push a halfway completed application to testing because they have constraints and need to make deadlines – but in such a rushed scenario, the communication loop will quickly deteriorate, and the results will suffer. 

For example, we had a new client that needed to test quickly. It was their first time working with an app testing company like Testlio. We responded to their sense of urgency and put things into motion. In our rush, however, we learned that the client wasn’t quite ready for our speed. They struggled with how to share the app build and hadn’t assigned internal stakeholders to respond to the issues. So we didn’t get the information we needed as fast as we needed it, but the tests were already running. It was a learning opportunity for us to take a breath before diving in and to prepare clients for a faster testing process.

3. When prioritizing bug quantity over quality

Unfortunately, there is still this old-school thinking about testing – the more test cases you run, the more bugs you should have.

I had a client once who expressed disappointment that her first bug report didn’t include “a lot of” bugs. We walked her through the report, which identified a few highly critical issues. She came to realize that less is more, and the bugs we brought to the surface were, in fact, more valuable to her dev team.

When you focus on quantity rather than quality,  what happens is that cycle after cycle, the bugs just pile up. They don’t get prioritized, and it becomes challenging to make decisions and to take action (e.g., are they essential, does the end-user perceive it as a bug, etc.)

Related: The truth behind unlimited cycles

How can we fix these bottlenecks?

The most crucial element to a successful quality assurance engagement is to manage expectations proactively – across all stakeholders, at all levels, and all stages. I ask what does the client want? What results do they expect, and why? What value can we provide? How to convey or receive information concisely and effectively?

Consistently transparent communication is essential, including keeping the client informed of any potential blockers and/or concerns so that they are able to make the best decisions every time. In the same vein, when sending test results, the client’s report must be as clear and specific as possible and include what could not be tested, either by agreement or by a blocker issue. It pays to keep in mind that there are typically several people on the client-side reading reports and viewing results. Our reports provide precise descriptions, the “what” and “why,” and action items designed for multiple stakeholders. 

In your opinion, what’s wrong with QA processes right now?

One problem is the concept of quantity over quality. Many testing companies pay their testers by the bug, which incentivizes the wrong behavior – to uncover any issue, regardless of the impact or edge case. The testers get paid because they followed orders and discovered as many bugs as possible. What happens is that the client ends up with a long backlog of non-essential bugs. Not only does it take them more time to sift through and prioritize the bugs, but they also usually don’t fix them because the likelihood of most of the issues occurring for real users is slight.

Testing provides feedback, and, in my opinion, it’s inefficient and a waste of time and money if you’re not going to do anything with this feedback. If the bug is low priority and you don’t plan to fix it, then close it. No need to leave it lingering in the backlog while the bug list continues piling up. Unfortunately, not that many development teams focus on what to do with their issues list in the long run.

I see two ways to address the quantity over quality issue. The first is to only work with thoughtful testers with domain expertise and the right incentive structure to reward high impact bugs. The second way is to partner with a testing company that offers the option of expert project management. For example, I not only work with the best testers but also review their bugs to understand their impact and help to prioritize them for my clients.

While delivering a positive user experience remains of the utmost importance, I believe we all understand that there is no such thing as a bug-free app, so the focus should be on making sure it’s free of critical bugs instead.

My final bit of advice – make peace that you won’t be able to fix all bugs.

Jumbotron image

This is part of our series of articles titled “Ask a QA Engagement Manager”. Do you have a question for us?  Or, would you like to hear more about how Testlio can help you elevate your testing? Drop us a line and schedule a free demo.