We have a large test suite.
How can we decide which tests to choose for the regression testing ?
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.Sign up to join this community
Some factors you might choose to take into consideration:
As always, what you end up doing is a compromise between thorough testing and the time and resources you have available. Sometimes, much as we hate it, we have to cross our fingers and hope we caught everything important before release.
If you need to perform the regression on the whole application, let's say before a release, as a rule of thumb you execute the higher in priority and critical tests, the tests that cover the most changed parts of the application.
Considering the statement that you have a large regression suite of tests and therefore, probably the time doesn't allow you to run all of them, that should be your approach.
If you have a part of the application covered by automated tests, and assuming the automation already covers the most critical parts, then you execute those and in parallel go over the next critical and priority areas.
Basically you need to make sure that most important parts of your application under test (AUT) are covered by testing and work as expected.
Also note that besides the "critical areas" description I also used the "higher in priority" statement because, while a typo in your application or company name is not a critical bug, it's still a very important with huge priority to be fixed before the release in cause.
Actually a difficult question to answer. But here are my insights:
Regression is to verify that any new changes haven't broken the existing functionality. So keeping this in mind, it is clear that you have to think/choose those test suites/cases which are more important/make the impact.
I would suggest you to make the regression test suite on time (i.e. after you get the build for testing) This might sound weird but there is a practical reason for it. Once you get the build for QA, you can ask the developer about the impact areas and when you get to know those, you can pick those scenarios (by giving them high priority) and those few test cases which will cover the most important functionality of the application.
This above procedure actually works if you have a couple of hours to take out/choose the scenarios. I am using this technique since always, all the parts/modules are not important in a particular fix. Selecting all those scenarios which are not much important will defeat the very purpose of testing.
Let me know if you need any explanation in this. I would be happy to help.
It will depend very much on the details of your situation
Generally I consider the following factors when selecting some regression tests over others:
Note also that in a large number of organizations, whatever tests exists are all run and thus there is no separate regression tests, they are just the full test suite that is run for all builds.
Actually it is a good question not just for interviews; some more useful ideas:
the tests that found more bugs, since they find more bugs in unstable parts of the code.
the tests that found few if any bugs in the code, since they will keep the stable parts of the code stable.
one test per module so we have good coverage.
the fastest test to run.
tests with the most code/feature/other coverage.
tests which are the simplest to run.
etc. etc. etc.
If you want to do a good job at selecting some of the previously existing tests that you have in your large suite, much of the advice already offered is very sound advice such as (1) prioritizing by technical risk, business risk, financial impact, key client impact, (2) taking into account the size of the code changes introduced in this release, (3) taking into account the types and locations of the code changes in this release, etc.
If you followed those suggestions thoughtfully, you would have a regression suite that was far more effective and efficient than most regression test suites in use by typical software testing teams.
If you wanted to to a great job at selecting a manageable number of regression tests though, (assuming your existing tests have not been created using sophisticated test design methods) you should go further than that and generate a new set of regression tests that would be specially designed to accomplish the following goals:
My company, Hexawise, has worked with more than 100 Fortune 500 firms to help teams achieve truly great regression test suites on hundreds of projects. By helping them systematically eliminate wasteful repetition from test sets, we can frequently help teams reduce the number of regression tests they're executing by more than 50%. By systematically covering all simple interactions between test inputs in the regression test suite, the optimized test sets we help our clients create with our test design tool, our clients consistently achieve higher testing coverage than their much larger original test sets.
The difference between doing a good job and doing a great job in this area is large. That's because existing test sets have far more wasteful repetition from test to test than people tend to realize and because existing test sets have more coverage gaps than people tend to realize. The slides below are from an online presentation about test selection optimization that explains the concrete reasons why this is the case:
Complete presentation available[here].1