How to identify regression tests?

Question

How can we detect suitable regression test-cases in the full test suite to use whenever we receive new build?

What are useful strategies to go about such a selection?

Answer 1

This doesn't have a single answer; it depends on the state of the code and the level you're testing at.

At a high level, regression tests are tests that you have for things that currently work in the product, and should continue to work the same way in the future. So in order to identify regression tests, you need to identify parts of the program that aren't expected to change.

For what I test, mostly, this is easy, because my company made a promise that any code written against supported interfaces will always work. So our regression bucket, at a minimum, consists of all our test cases that test things that are designated as supported interfaces. If you're testing some product that exposes an API, or is callable somehow, you should be able to do something similar, if those interfaces are supposed to be stable.

However, not all companies/projects/products make the same promise, in which case, you can't necessarily use prior test cases that tested interfaces as a regression suite (although ideally if there's an API, there should be documentation provided saying what has/hasn't changed that you can use, and instead of changing already existing interfaces, they should create new ones/extend the existing ones).

Depending on your use of external libraries, you might also consider putting regression testcases together for those as well, in case they make some sort of incompatible change that isn't documented.

Answer 2

It depends.

Every application is different. Every organization is different. Every organization and application has a different risk tolerance that determines how many of your test cases need to be part of your regression suite.

At one extreme, any test case that hasn't been deprecated by changes to the application is part of the regression test suite. This happens when the software/organization has a very low risk tolerance (e.g. medical device software, financial software, etc.)

At the other extreme, if any regression occurs, it's a smoke test at best.

In my experience most places are somewhere in the middle and perform regression against the most critical, highest impact parts of the application. Exactly how far they go depends on the organization, the tools they're using, and the nature of the application.

Answer 3

Finding good set of regression test cases may require good understanding of a domain. You may not want to select any random number of test cases for regression which may not be very common scenarios in the production. You may want to get help from BA or solution design team to understand most important & common scenarios. Many projects follow RBT (RISK based Testing) approach to prioritize their test cases. RBT defines likely hood of failure and impact of failure. These two parameters can serve good starting point for the team to find out most important scenarios.

Following criteria (but not limited) can be applied for selection of Regression Test Cases.

1. Scenarios which touch base most of the functional points of AUT
2. Scenarios which are more common in production (based on volume or market data)
3. Scenarios which verify core features of application or a product
4. Scenarios which are likely to fail Complex scenarios
5. Scenarios related to functionalities which have undergone changes in a recent past

Answer 4

I have found that basically all of the tests and test suites that I have used and created for unit and integrated tests are regression tests. They are tests to say that 'this did work and should work now' and then when they break you know you need to fix/change them and/or the underlying code. This has been true in the last 3 companies I have worked for. Note however that these positions all used ruby on rails and rspec-capybara-selenium for the testing framework. Different and older frameworks may present different opportunities.

Tests in a BDD/TDD environment may initially be written as tests ahead of the actual code. However once the tests pass and all the code is written, if these tests are kept around (which is normally the case) they effectively become regression tests for the future.

Tests that I might not consider as regression tests are performance and load tests. Even with these though, if the tests help establish a benchmark for what is reasonable and then a change is made which affects performance it can be visible in the performance/load tests. So they are acting as regression tests too, making sure that, despite changes, nothing regresses in the performance area.

Also in terms of "How can we detect suitable regression test-cases in the full test suite to use whenever we receive new build?" My answer to that is that you should talk with the product owner and application developers to determine what is suitable. It should be a team decision through shared understanding. "Detection" is not practical given the wide variety of situations.

At the end of the day I mostly consider tests == regression tests and don't worry too much about exact definitions, definitions aren't the important part of my job. The important thing is to identify key functionality and make sure it has tests.

Answer 5

To create a regression test case suite one must have thorough application knowledge or complete understanding of the product flow.

1. Start with breaking the application into features. Look for any labels or tags if added to test cases for product functionality. Like in Jira/test rail we have labels and tags added to test cases.
2. If there are then you can filter out test cases using a tag/label and can select them for your regression suite
3. Otherwise pick any feature and try to complete selection for one feature and then move towards next. Selecting test cases in flow as how product functions would be good.
4. Analyzing the defects reported by customers in last releases and selecting test cases for these areas would make suite more valuable
5. Select test cases validating the integration points, that is where most likely things can break

Above approach is used by us in our software testing company

Answer 6

Once test packs are created the main objectives of review and signoff process is to confirm test coverage and then categorise each and every test case into High, Medium and Low categories based on BI (Business Impact) and Likelihood of Failure. At the end of the review process you will have the entire test pack categorised. This approach helps to Identify a regression pack. When you have to run a regression depending of the functional areas the new piece of code is due to impact ( if known ) the pack can be pulled. If there is no time constraint you can choose to run all the P1, P2 and P3 test cases. If there is a time constraint and risk appetite of the project and its stakeholders, only P1 can be executed.

Hope this clarifies.