Why QA Testing Fails…and how to fix it: a Q&A with Kristi Kaljurand

Why is that? 

To delve deeper into this topic, we asked our Senior Engagement Manager Kristi for her perspective on what could be some of the root causes of a lagging QA process.

About Kristi
Eleven years ago, Kristi started her professional career as a software consultant in a local Estonian IT company. After a few years, she took on a QA engineer role, and that’s how she discovered her true passion – software testing.
Kristi first joined Testlio as a freelance tester in January 2013. Soon after that, she started to work as a Community Test Lead and in May 2014, she joined the Testlio team on a full-time basis. She now serves as one of Testlio’s most Senior Engagement Managers.

Before joining Testlio, my background was in web testing, where things are quite black and white. If there’s a bug, then most likely it’s a bug. But in the mobile world, it’s very different. There are so many variables. Even if one person can reproduce a bug 100% on a given device, another one might be unable to reproduce the same bug. And something that can’t be reproduced predictably is going to be hard to fix.

Kristi Kaljurand, Testlio Senior Engagement Manager

In what situations can QA testing fail companies?

In my experience, it usually boils down to a very simple matter: miscommunication and/or misaligned expectations. If the client’s expectations haven’t been mapped out or understood correctly, it’s safe to say that no matter the results you deliver, they will not be happy. I’ve seen this happen, especially when a well-meaning contact on the client’s side doesn’t have a QA background or an understanding of the testing process.
Here are three main situations where testing can fail to deliver exceptional results:

1. When focusing too much on one type of testing

Smoke testing is a great example.  It’s a preliminary level of testing that helps ensure the stability of the application. By design, smoke testing helps to quickly understand if the application ‘catches fire’ without going too deep into the testing process (hence the name ‘smoke tests’). The risk of only performing smoke tests is that it’s easy to miss critical bugs. 

2.  When testing in a hurry

Even with tight deadlines and narrow timeframes, spending time to define the testing scope properly is critical. Sometimes, the client will push a halfway completed application to testing because they have constraints and need to make deadlines – but in such a rushed scenario, the communication loop will quickly deteriorate, and the results will suffer. 

For example, we had a new client that needed to test quickly. It was their first time working with an app testing company like Testlio. We responded to their sense of urgency and put things into motion. In our rush, however, we learned that the client wasn’t quite ready for our speed. They struggled with how to share the app build and hadn’t assigned internal stakeholders to respond to the issues. So we didn’t get the information we needed as fast as we needed it, but the tests were already running. It was a learning opportunity for us to take a breath before diving in and to prepare clients for a faster testing process.

3. When prioritizing bug quantity over quality

Unfortunately, there is still this old-school thinking about testing – the more test cases you run, the more bugs you should have.

I had a client once who expressed disappointment that her first bug report didn’t include “a lot of” bugs. We walked her through the report, which identified a few highly critical issues. She came to realize that less is more, and the bugs we brought to the surface were, in fact, more valuable to her dev team.

When you focus on quantity rather than quality,  what happens is that cycle after cycle, the bugs just pile up. They don’t get prioritized, and it becomes challenging to make decisions and to take action (e.g., are they essential, does the end-user perceive it as a bug, etc.)

How can we fix these bottlenecks?

The most crucial element to a successful quality assurance engagement is to manage expectations proactively – across all stakeholders, at all levels, and all stages. I ask what does the client want? What results do they expect, and why? What value can we provide? How to convey or receive information concisely and effectively?

Consistently transparent communication is essential, including keeping the client informed of any potential blockers and/or concerns so that they are able to make the best decisions every time. In the same vein, when sending test results, the client’s report must be as clear and specific as possible and include what could not be tested, either by agreement or by a blocker issue. It pays to keep in mind that there are typically several people on the client-side reading reports and viewing results. Our reports provide precise descriptions, the “what” and “why,” and action items designed for multiple stakeholders. 

In your opinion, what’s wrong with QA processes right now?

One problem is the concept of quantity over quality. Many testing companies pay their testers by the bug, which incentivizes the wrong behavior – to uncover any issue, regardless of the impact or edge case. The testers get paid because they followed orders and discovered as many bugs as possible. What happens is that the client ends up with a long backlog of non-essential bugs. Not only does it take them more time to sift through and prioritize the bugs, but they also usually don’t fix them because the likelihood of most of the issues occurring for real users is slight.

Testing provides feedback, and, in my opinion, it’s inefficient and a waste of time and money if you’re not going to do anything with this feedback. If the bug is low priority and you don’t plan to fix it, then close it. No need to leave it lingering in the backlog while the bug list continues piling up. Unfortunately, not that many development teams focus on what to do with their issues list in the long run.

While delivering a positive user experience remains of the utmost importance, I believe we all understand that there is no such thing as a bug-free app, so the focus should be on making sure it’s free of critical bugs instead.

My final bit of advice – make peace that you won’t be able to fix all bugs.

Jumbotron image