What is regression testing?

Before we dive into regression testing, let’s first understand what a regression is.

A software regression is any unwanted change that occurs from code changes. A real-world example of this is you take your car to a mechanic to get the air conditioning fixed, and when you get it back, the air conditioning is fixed but the gas tank sensor no longer works. Bummer.

In software, a regression can happen after a new feature is implemented. Let’s say an email photo sharing service introduces video compatibility, but after the capability is released, the basic function of sharing photos to a set group of email addresses is completely broken.

Given that a regression is an unintended change, then regression testing is the process of hunting for these changes.

regression testing

Why should it be automated?

The words “automated” and “regression testing” go hand-in-hand. When companies first consider how they can maximize their QA teams’ potential through automation, they look to regression tests.

This is because regression tests are naturally good candidates for automation. In general, automated test cases should be:

  • Stable
  • Repeated frequently
  • Simple
  • Worth maintaining

Because regression testing is for existing functionality, it requires constant repetition. Every release cycle should include a fair amount of regression testing to ensure that no new development causes bugs or breaks.

When regression testing is automated, it allows for checks into a variety of changes and frees up testers to conduct manual exploration into more unusual cases in the production environment. Not all regressions are caused by new features or the consequences of routine bug fixes. They can also be caused by database updates or new browser versions, in the case of a web app. A regression can also be an issue with efficiency and speed. Automating those cases which are stable and repeatable allows manual testers to spend more time on testing various environments and for merging more complicated cases together at a higher level.

Challenges and how to overcome them

As with other forms of automation, regression testing is subject to the challenge of test case maintenance.

The more test cases we automate, the more we can verify quality for existing functionality. But the more test cases we automate, the more test cases we must maintain. If we write the test cases too loosely, they’re too easily passed, but if we write them rigidly, they’ll need to updated and rewritten as the system changes.

The question of what to automate can only be answered with a deep dive into an individual application, but a good starting point when determining regression test candidates is to ask which components of the system have been affected by a code change?

With each new release, you can list out or create a mindmap for affected areas, then determine at the unit testing level, integration testing level, and system testing level which cases will need to be repeated and will be worthwhile to maintain.

Automated regression testing tools

Tools like Selenium and vTest can be used for automating functional as well as regression testing, allowing you to achieve more in a single platform. With Selenium, you can automate the test scripts, as well as the result reports.

While the execution of the tests are light years faster than manual tests, everything else about automated testing takes more time, which also means it can initially cost more money. Writing the tests and setting up the suite requires an initial time-consuming effort that will only pay off if the pre-defined tests truly overlap with an overall strategy to create excellent app coverage.

Automated regression testing services

In the long run, automated testing can save an organization money spent on human resources, but only if they are fully prepared for the investment.

With any turnover in QA employees, the proactive and retroactive processes for maintaining automated test suites can easily fall by the wayside. If a business wants to experience long-term cost savings by automatically testing for software regressions, but doesn’t have the internal resources to set up and maintain the suite, then working with an external service provider is an excellent option.

Some benefits are:

  • Fit automated testing into an overall test strategy
  • Get results continuously and for upcoming releases
  • No need to commit human resources
  • No risk of forfeiting ROI with inevitable organizational changes

regression testing

Practicing selectiveness

Ultimately, regardless of WHO does it and HOW it is done, regression testing requires that we practice selectiveness.

What features and functions are affected by new code?

What is the minimum amount of test cases that can cover the maximum amount of code?

Which unit level tests can be merged or eliminated?

By always (always!) being selective about what test cases require automation in the first place, you create a more stable automation suite that is less subject to inevitable software changes.

To learn how Testlio can maximize your automation budget and help you find regressions, contact us for a demo.

<p>Dayana is a QA engineer turned technology writer living in Milan, Italy. She’s always down for a smoothie.</p>