How can we detect suitable regression test-cases in the full test suite to use whenever we receive new build?
What are useful strategies to go about such a selection?
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.Sign up to join this community
This doesn't have a single answer; it depends on the state of the code and the level you're testing at.
At a high level, regression tests are tests that you have for things that currently work in the product, and should continue to work the same way in the future. So in order to identify regression tests, you need to identify parts of the program that aren't expected to change.
For what I test, mostly, this is easy, because my company made a promise that any code written against supported interfaces will always work. So our regression bucket, at a minimum, consists of all our test cases that test things that are designated as supported interfaces. If you're testing some product that exposes an API, or is callable somehow, you should be able to do something similar, if those interfaces are supposed to be stable.
However, not all companies/projects/products make the same promise, in which case, you can't necessarily use prior test cases that tested interfaces as a regression suite (although ideally if there's an API, there should be documentation provided saying what has/hasn't changed that you can use, and instead of changing already existing interfaces, they should create new ones/extend the existing ones).
Depending on your use of external libraries, you might also consider putting regression testcases together for those as well, in case they make some sort of incompatible change that isn't documented.
Every application is different. Every organization is different. Every organization and application has a different risk tolerance that determines how many of your test cases need to be part of your regression suite.
At one extreme, any test case that hasn't been deprecated by changes to the application is part of the regression test suite. This happens when the software/organization has a very low risk tolerance (e.g. medical device software, financial software, etc.)
At the other extreme, if any regression occurs, it's a smoke test at best.
In my experience most places are somewhere in the middle and perform regression against the most critical, highest impact parts of the application. Exactly how far they go depends on the organization, the tools they're using, and the nature of the application.
Finding good set of regression test cases may require good understanding of a domain. You may not want to select any random number of test cases for regression which may not be very common scenarios in the production. You may want to get help from BA or solution design team to understand most important & common scenarios. Many projects follow RBT (RISK based Testing) approach to prioritize their test cases. RBT defines likely hood of failure and impact of failure. These two parameters can serve good starting point for the team to find out most important scenarios.
Following criteria (but not limited) can be applied for selection of Regression Test Cases.
I have found that basically all of the tests and test suites that I have used and created for unit and integrated tests are regression tests. They are tests to say that 'this did work and should work now' and then when they break you know you need to fix/change them and/or the underlying code. This has been true in the last 3 companies I have worked for. Note however that these positions all used ruby on rails and rspec-capybara-selenium for the testing framework. Different and older frameworks may present different opportunities.
Tests in a BDD/TDD environment may initially be written as tests ahead of the actual code. However once the tests pass and all the code is written, if these tests are kept around (which is normally the case) they effectively become regression tests for the future.
Tests that I might not consider as regression tests are performance and load tests. Even with these though, if the tests help establish a benchmark for what is reasonable and then a change is made which affects performance it can be visible in the performance/load tests. So they are acting as regression tests too, making sure that, despite changes, nothing regresses in the performance area.
Also in terms of "How can we detect suitable regression test-cases in the full test suite to use whenever we receive new build?" My answer to that is that you should talk with the product owner and application developers to determine what is suitable. It should be a team decision through shared understanding. "Detection" is not practical given the wide variety of situations.
At the end of the day I mostly consider tests == regression tests and don't worry too much about exact definitions, definitions aren't the important part of my job. The important thing is to identify key functionality and make sure it has tests.
To create a regression test case suite one must have thorough application knowledge or complete understanding of the product flow.
Above approach is used by us in our software testing company
Once test packs are created the main objectives of review and signoff process is to confirm test coverage and then categorise each and every test case into High, Medium and Low categories based on BI (Business Impact) and Likelihood of Failure. At the end of the review process you will have the entire test pack categorised. This approach helps to Identify a regression pack. When you have to run a regression depending of the functional areas the new piece of code is due to impact ( if known ) the pack can be pulled. If there is no time constraint you can choose to run all the P1, P2 and P3 test cases. If there is a time constraint and risk appetite of the project and its stakeholders, only P1 can be executed.
Hope this clarifies.