What is regression testing?

Before we dive into regression testing, let’s first understand what a regression is.

A software regression is any unwanted change that occurs from code changes. A real-world example of this is you take your car to a mechanic to get the air conditioning fixed, and when you get it back, the air conditioning is fixed but the gas tank sensor no longer works. Bummer.

In software, a regression can happen after a new feature is implemented. Let’s say an email photo sharing service introduces video compatibility, but after the capability is released, the basic function of sharing photos to a set group of email addresses is completely broken.

New call-to-action

Given that a regression is an unintended change, then regression testing is the process of hunting for these changes. The regression testing definition is simply an unintended issue caused by code change.

Regression: unintended issue caused by code change

A top 8 regression testing test case checklist

There are many scenarios when it makes the most sense to perform regression tests, including these top 8 test cases:

  1. Cases with frequent defects
  2. Functionalities highly visible to users
  3. Cases which verify core features of the product
  4. Cases of functionalities that have undergone recent changes
  5. All complex and integration test cases
  6. Boundary value test cases
  7. A sample of Successful test cases
  8. A sample of Failure test cases

Why should regression testing be automated?

The words “automated” and “regression testing” go hand-in-hand. When companies first consider how they can maximize their QA teams’ potential through automation, they look to regression tests.

This is because regression tests are naturally good candidates for automation. In general, automated test cases should be:

  • Stable
  • Repeated frequently
  • Simple
  • Worth maintaining

Because regression testing is for existing functionality, it requires constant repetition. Every release cycle should include a fair amount of regression testing to ensure that no new development causes bugs or breaks.

When regression testing is automated, it allows for checks into a variety of changes and frees up testers to conduct manual exploration into more unusual cases in the production environment. Not all regressions are caused by new features or the consequences of routine bug fixes. They can also be caused by database updates or new browser versions, in the case of a web app. A regression can also be an issue with efficiency and speed. Automating those cases which are stable and repeatable allows manual testers to spend more time on testing various environments and for merging more complicated cases together at a higher level.

Related: 8 steps to implement a manual regression testing strategy

Challenges and how to overcome them

As with other forms of automation, regression testing is subject to the challenge of test case maintenance.

The more test cases we automate, the more we can verify quality for existing functionality. But the more test cases we automate, the more test cases we must maintain. If we write the test cases too loosely, they’re too easily passed, but if we write them rigidly, they’ll need to updated and rewritten as the system changes.

The question of what to automate can only be answered with a deep dive into an individual application, but a good starting point when determining regression test candidates is to ask which components of the system have been affected by a code change?

With each new release, you can list out or create a mindmap for affected areas, then determine at the unit testing level, integration testing level, and system testing level which cases will need to be repeated and will be worthwhile to maintain.

When internal resources are in short supply or automated testing seems too daunting, using crowdsourced testing for overnight or weekend regression testing works wonders. For example, a leading provider of hotel accommodations with over 10 Million installs crowdsourced their regression testing to augmented their small in-house QA team. What had taken the internal team three weeks to test is crowdsourced to Testlio’s network and completed over weekends. The process is more efficient than adding additional full-time testers and provides the speed and flexibility necessary for today’s continuous development initiatives. Testlio’s do it for me (DIFM) approach to managed QA testing immediately takes the pressure off the in-house team so they can focus on higher-level testing. The entire process includes:

  • Twenty (20) testers to work in 2 to 3-hour bursts to knock out the regression testing over weekends
  • POS and localization testing
  • Comprehensive device, OS, and real user app version upgrade coverage, as not all of their users immediately install the latest updates

For more details on this and other use cases, read about Testlio’s regression testing capability or drop us a line.

Automated regression testing tools

Tools like Selenium and vTest can be used for automating functional as well as regression testing, allowing you to achieve more in a single platform. With Selenium, you can automate the test scripts, as well as the result reports.

While the execution of the tests are light years faster than manual tests, everything else about automated testing takes more time, which also means it can initially cost more money. Writing the tests and setting up the suite requires an initial time-consuming effort that will only pay off if the pre-defined tests truly overlap with an overall strategy to create excellent app coverage.

Automated regression testing services

In the long run, automated testing can save an organization money spent on human resources, but only if they are fully prepared for the investment.

With any turnover in QA employees, the proactive and retroactive processes for maintaining automated test suites can easily fall by the wayside. If a business wants to experience long-term cost savings by automatically testing for software regressions, but doesn’t have the internal resources to set up and maintain the suite, then working with an external service provider is an excellent option.

Eight examples of services offered by best-in-class automated testing companies include:

  • Alignment and integration of your engineering workflows with the automated testing service’s testing process
  • The design, build and maintenance of accurate test scripts
  • The management of test runs and setups for you
  • The review and validation of test results and defects, while also taking care of noise and flaky tests
  • Real device utilization
  • Access to build management tools
  • Automation run systems
  • The integration of reporting and insights

Some benefits are:

  • Fit automated testing into an overall test strategy
  • Get results continuously and for upcoming releases
  • No need to commit human resources
  • No risk of forfeiting ROI with inevitable organizational changes

If a customer asked you to demonstrate to him, within an hour, that your newest software is ready to use, what tests would you run? Are these the same tests that you are now performing in your first hour of regression testing? They should be.

Steven Woody

Practicing selectiveness

Ultimately, regardless of WHO does it and HOW it is done, regression testing requires that we practice selectiveness.

What features and functions are affected by new code?

What is the minimum amount of test cases that can cover the maximum amount of code?

Which unit level tests can be merged or eliminated?

By always (always!) being selective about what test cases require automation in the first place, you create a more stable automation suite that is less subject to inevitable software changes.

Jumbotron image

Do you need help with your regression testing? Testlio can help, schedule a free demo today.

Dayana is a QA engineer turned technology writer living in Milan, Italy. She's always down for a smoothie.