Which tests to choose from a large regression test suite? [closed]

Question

We have a large test suite.

How can we decide which tests to choose for the regression testing ?

Answer 1

It depends.

Some factors you might choose to take into consideration:

• Time - for manual regression testing, the amount of time you have available is a major factor in what you test.
• Priority - ideally you'll have your tests prioritized by how critical they are to the application: I usually work this way:
  • Core function - The most used and most important features. These are chosen on the basis that if these features can't be used, the release is effectively useless. This category can also include "people will die if this software goes wrong".
  • Not core but high impact - This category would include hugely embarrassing bugs like misspelling the product name as well as problems with frequently-used supporting functionality.
  • Frequently broken - This category covers the most fragile parts of the system, the areas that your team knows are more likely to have regression issues.
  • Your biggest customer will walk if this is broken - Depending on how much revenue your biggest customer brings in, particularly compared to everyone else, this can jump to your first priority.
  • Anything that affects financial data - This may or may not be relevant to your software, but if it is, you can guarantee that your customers want the financials to be accurate. Again, this can be your first priority if the core of your business is financial data.
  • Everything else - prioritized using a risk/impact matrix that should ideally be re-evaluated regularly.
• Release Size - It's relatively normal to perform less regression on a patch release than on a major release, especially when most of your regression is done manually.
• What Changed - Particularly if you're working with well-designed software that doesn't have spaghetti code under the hood, manual regression can be focused around the areas that changed in the release. If your code base has a lot of dependencies and spaghetti code, this isn't a viable strategy unless you're really pressed for time.

As always, what you end up doing is a compromise between thorough testing and the time and resources you have available. Sometimes, much as we hate it, we have to cross our fingers and hope we caught everything important before release.

Answer 2

If you need to perform the regression on the whole application, let's say before a release, as a rule of thumb you execute the higher in priority and critical tests, the tests that cover the most changed parts of the application.

Considering the statement that you have a large regression suite of tests and therefore, probably the time doesn't allow you to run all of them, that should be your approach.

If you have a part of the application covered by automated tests, and assuming the automation already covers the most critical parts, then you execute those and in parallel go over the next critical and priority areas.

Basically you need to make sure that most important parts of your application under test (AUT) are covered by testing and work as expected.

Also note that besides the "critical areas" description I also used the "higher in priority" statement because, while a typo in your application or company name is not a critical bug, it's still a very important with huge priority to be fixed before the release in cause.

Answer 3

Actually a difficult question to answer. But here are my insights:

Regression is to verify that any new changes haven't broken the existing functionality. So keeping this in mind, it is clear that you have to think/choose those test suites/cases which are more important/make the impact.

I would suggest you to make the regression test suite on time (i.e. after you get the build for testing) This might sound weird but there is a practical reason for it. Once you get the build for QA, you can ask the developer about the impact areas and when you get to know those, you can pick those scenarios (by giving them high priority) and those few test cases which will cover the most important functionality of the application.

This above procedure actually works if you have a couple of hours to take out/choose the scenarios. I am using this technique since always, all the parts/modules are not important in a particular fix. Selecting all those scenarios which are not much important will defeat the very purpose of testing.

Let me know if you need any explanation in this. I would be happy to help.

Answer 4

It will depend very much on the details of your situation

Generally I consider the following factors when selecting some regression tests over others:

• The happy path tests
• The most critical money making paths
• Speed of the test
• Uniqueness of test and if similar tests provide some coverage
• End-to-end integration tests can reduce the need for using unit tests for regression testing

Note also that in a large number of organizations, whatever tests exists are all run and thus there is no separate regression tests, they are just the full test suite that is run for all builds.

Answer 5

42

Actually it is a good question not just for interviews; some more useful ideas:

• First ask for more information, for example the purpose of the tests, where is the product in its life-cycle, some history of the product, it's customers and testing etc.

Then choose:

• the tests that found more bugs, since they find more bugs in unstable parts of the code.

• the tests that found few if any bugs in the code, since they will keep the stable parts of the code stable.

• one test per module so we have good coverage.

• the fastest test to run.

• tests with the most code/feature/other coverage.

• tests which are the simplest to run.

etc. etc. etc.

Answer 6

If you want to do a good job at selecting some of the previously existing tests that you have in your large suite, much of the advice already offered is very sound advice such as (1) prioritizing by technical risk, business risk, financial impact, key client impact, (2) taking into account the size of the code changes introduced in this release, (3) taking into account the types and locations of the code changes in this release, etc.

If you followed those suggestions thoughtfully, you would have a regression suite that was far more effective and efficient than most regression test suites in use by typical software testing teams.

If you wanted to to a great job at selecting a manageable number of regression tests though, (assuming your existing tests have not been created using sophisticated test design methods) you should go further than that and generate a new set of regression tests that would be specially designed to accomplish the following goals:

• Systematically eliminate all wasteful repetition between tests
• Systematically cover all simple interactions between your test inputs (because so many defects are caused by simple interactions that do not necessarily get covered by requirements-driven test selection approaches)
• Incorporate the prioritization goals listed above (by adjusting the systematically generated tests, as necessary)

My company, Hexawise, has worked with more than 100 Fortune 500 firms to help teams achieve truly great regression test suites on hundreds of projects. By helping them systematically eliminate wasteful repetition from test sets, we can frequently help teams reduce the number of regression tests they're executing by more than 50%. By systematically covering all simple interactions between test inputs in the regression test suite, the optimized test sets we help our clients create with our test design tool, our clients consistently achieve higher testing coverage than their much larger original test sets.

The difference between doing a good job and doing a great job in this area is large. That's because existing test sets have far more wasteful repetition from test to test than people tend to realize and because existing test sets have more coverage gaps than people tend to realize. The slides below are from an online presentation about test selection optimization that explains the concrete reasons why this is the case:

Complete presentation available[here].1