What is regression testing?

Before we dive into regression testing, let’s first define it and understand what a regression is.

To raise the stakes further, imagine you’re working on an airplane engine. You need to replace a component – say, a turbine disc. After you’ve done the job, you wouldn’t just check that the disk works in isolation; you’d also test the entire engine to ensure everything else functions as expected. For example, you’d want to ensure that you haven’t accidentally impacted how gases leave the combuster, that the compressor no longer maintains proper cabin pressure, or that the new disc caused any other damages.

In software, a regression can happen after a new feature is implemented. When you change or add a piece of code (like replacing the disc in the airplane), you need to not only verify that the new code works as intended but also run regression tests against the existing functions of the software (like the other parts of the plane) to make sure they haven’t been inadvertently affected by your changes.

Let’s say an email photo sharing service introduces video compatibility, but after the capability is released, the basic function of sharing photos to a set group of email addresses is completely broken.

New call-to-action

A top 8 regression testing test case checklist

There are many scenarios when it makes the most sense to perform regression tests. To explain the test cases, we’ll look at automating regression tests for an e-commerce website that recently added a discount coupon feature. Here are the top 8 test cases selected, tied with the regression testing strategy:

  1. Cases with frequent defects: User registration often had login errors. The login process needs thorough regression testing to ensure seamless user access.
  2. Obvious functionalities: Product browsing, a highly expected feature, must be tested to ensure the product information is correctly displayed.
  3. Cases that verify core features of the product: The checkout and payment process, a core e-commerce feature, requires testing to maintain trust and reliability.
  4. Cases of functionalities that have undergone recent changes: The newly added discount coupon feature requires rigorous testing to ensure it functions as intended.
  5. All complex and integration test cases: The shopping cart function, integrated with multiple features (product browsing, checkout, payment gateways), needs detailed testing to avoid failures.
  6. Boundary value test cases: Testing edge cases, such as maximum and minimum quantities for product purchases, ensures the system can handle limit conditions.
  7. A sample of Successful test cases: Test user registration and successful purchase with a discount coupon is essential for a smooth user experience.
  8. A sample of Failure test cases: Attempt invalid actions such as applying expired discount coupons or registering with an existing email to confirm error handling.

Why should regression testing be automated?

The words “automated” and “regression testing” go hand-in-hand. When companies first consider how they can maximize their QA teams’ potential through automation, they look to regression tests.

  • Stable
  • Repeated frequently
  • Simple
  • Worth maintaining

Because regression testing is for existing functionality, it requires constant repetition. Every release cycle should include a fair amount of regression testing to ensure that no new development causes bugs or breaks.

Challenges and how to overcome them

As with other forms of automation, regression testing is subject to the challenge of test case maintenance.

The question of what to automate can only be answered with a deep dive into an individual application, but a good starting point when determining regression test candidates is to ask which components of the system have been affected by a code change?

With each new release, you can list out or create a mindmap for affected areas, then determine at the unit testing level, integration testing level, and system testing level which cases will need to be repeated and will be worthwhile to maintain.

  • Twenty (20) testers to work in 2 to 3-hour bursts to knock out the regression testing over weekends
  • POS and localization testing
  • Comprehensive device, OS, and real user app version upgrade coverage, as not all of their users immediately install the latest updates

Automated regression testing tools

Automated regression testing services

In the long run, automated testing can save an organization money spent on human resources, but only if they are fully prepared for the investment.

  • Alignment and integration of your engineering workflows with the automated software testing service’s testing process
  • The design, build and maintenance of accurate test scripts
  • The management of test runs and setups for you
  • The review and validation of test results and defects, while also taking care of noise and flaky tests
  • Real device utilization
  • Access to build management tools
  • Automation run systems
  • The integration of reporting and insights

Some benefits are:

  • Fit automated testing into an overall test strategy
  • Get results continuously and for upcoming releases
  • No need to commit human resources
  • No risk of forfeiting ROI with inevitable organizational changes

If a customer asked you to demonstrate to him, within an hour, that your newest software is ready to use, what tests would you run? Are these the same tests that you are now performing in your first hour of regression testing? They should be.

Steven Woody

Practicing selectiveness

What features and functions are affected by new code?

What is the minimum amount of test cases that can cover the maximum amount of code?

Which unit level tests can be merged or eliminated?

By always (always!) being selective about what test cases require automation in the first place, you create a more stable automation suite that is less subject to inevitable software changes.

Jumbotron image

Dayana is a QA engineer turned technology writer living in Milan, Italy. She's always down for a smoothie.