Which kind of Manual Test cases should be automated for Regression Testing?

Question

Automating each and every Test case would be futile. So which kind of Test cases should be automated so that more time is not spend on writing the automated test cases than actual manual testing of those test cases?

Answer 1

Automation is on ongoing process, it never stops and it can always be improved.

You should first prioritize the test cases, and then start automating from the most important ones. Focus on the functional cases first. For example, you should first automate a basic flow, smoke test, happy flow etc.

After all the important test cases are automated and the process is stable, you will have time to automate the rest of the test cases that are less important and have a lower impact.

Even if you spend more time automating a specific test then running it manually, long term you will gain more time if that test case is automated.

I also need to mention that, from the beginning, you should carefully design your automation framework so that is easy to maintain, and you don't have to lose time trying to refactor or improve the framework in the future.

Answer 2

Well, according to classical approach you should consider the costs of automated tests implementation as the investments and thus choose the tests whose automation would give you the maximum ROI.

The challenge here is to calculate NetProfit value since this depends on your particular project specific and what you actually need to minimize (either the cost of testing or the time of testing).

You should take into account the average number of builds in release, test priorities (which are to be revised since the value of the test might change), time of each particular test execution and estimated time of the implementation and maintenance of the test and test execution environment.

Then you sort your tests by ROI value and plan the implementation according to the resources you have.

Answer 3

It takes a lot of effort to manage and prioritize tests. Coverage is one of several good approximations.

One way to prioritize is to use tools to measure which parts of application are used most often (which is different from coverage), and focus testing accordingly to cover those parts.

Another way is to use expert opinion and rules of thumb: Write a "happy path test" for most important functionality from user POV, and for functionality which is important for safety/security (cost will be high if your company was sued for providing wrong info). Measure test coverage and add more to to increase it.

Coverage is of course separate complicated issue (do your tests need to cover error handling which is rare and hard to trigger? etc).

Answer 4

Software Testing is primarily a risk-based activity so test cases should be chosen wisely for frequent execution, aiming maximum return with least investment.

There are primarily three types of test suites executed during each release of a software application: regression tests, release specific tests, and defect fixes verification tests.

I would select the below test cases as part of regression suite:

1. that have frequently caught bugs: Some areas in the application are so error-prone that they usually fail following a small coding change. We can keep track of these failing test cases throughout the product cycle and cover them in the regression test suite.

2. that verify core features of the application: Prior to designing the test cases, identify all the core features of the application. Ensure that test cases cover all functionality mentioned in the requirements document. One can make use of a traceability matrix to make sure that no requirement is left untested.

3. for functionalities that have undergone recent changes: Maintain the history of functionality changes for test case documentation in order to identify the test cases to include in the regression suite.

4. Covering integration scenarios: This includes all those integration test scenarios on the functional boundary where two modules handshake and pass data and integration happens.

5. that are complex in nature: Some system functionality may only be accomplished by following a complex sequence of UI events.

6. that have high business priority: Prioritize the test cases as they relate to business impact and critical and frequently used functionalities. It is always helpful if an analysis is completed to determine which test cases are relevant. One idea is to classify the test cases into various priorities based on importance and customer use. The selection of test cases based on priority will greatly reduce efforts spent on regression testing.

7. On a “case-to-case” basis: There is always room for expert judgment and experience to pick test cases on a case-to-case basis, depending on each company/domain/project/team/ feature etc.

Answer 5

There are several factors that I consider when determining which manual test cases to automate

• How often tests are run manually
• How easy the manual emulators and simulators are to use
• How long the manual tests take to get the same feedback as the automated cases
• The availability and cost of manual testers
• How many variations need to be run
• What unit and integration automated test coverage exists
• The inability of humans to do large number of boring things consistently

Overall I follow the test type categorization for each group of tests (often representing workflows) of:

• One smoke test
  Usually that the first page is displayed before any data entry (catches 404's, 500s)
• Several sad path tests
  For when the user or system has an error
• One Happy path tests
  When everything works for the given workflow
• Optional Happy/Sad path tests
  When there is an optional part of the workflow, e.g. a non-required screen

For raw time comparison (leaving aside human factors), its really just a matter of what timeframe you look at and number of runs. For example:

A manual test take 30 minutes.
Automating it takes 4 days, e.g. 4x8=32, 32*60= 1920 minutes and then once automated takes 1 minute to run.

Therefore if this test is run 4 times a day (common in places with CI/CD) then over 6 months, manual would be 183x10 = 3660 mins whereas automated would be 1920 (to write) + 183x1 = 2103

Just as importantly for many development shops that want to do rapid iterations, agile development, CI/CD, etc. it it just not possible to do large scale manual testing in a fast and efficient manner. Even with 1000 off-shore folks bashing away, co-ordinating them starts to take a huge amount of time. Plus humans tend to want continual wages of course.

Overall, as with so many Quality Assurance and Quality Engineering issues I feel it comes down to The Value Proposition. Show how the costs, savings and revenue play out over time and automation can be an easier sell.