Automating each and every Test case would be futile. So which kind of Test cases should be automated so that more time is not spend on writing the automated test cases than actual manual testing of those test cases?
-
11. you ALWAYS spend more time on writing an automated test than running it manually. 2. "not more time is not spend on writing the automated test cases than actual manual testing of those test cases" is not really a good measurement in this case. Even if executing automated tests take more time than doing them manually they still can be more usefull than manual (depending on how much more "more time" is of course).– EmbeddedMay 4, 2018 at 14:29
5 Answers
Automation is on ongoing process, it never stops and it can always be improved.
You should first prioritize the test cases, and then start automating from the most important ones. Focus on the functional cases first. For example, you should first automate a basic flow, smoke test, happy flow etc.
After all the important test cases are automated and the process is stable, you will have time to automate the rest of the test cases that are less important and have a lower impact.
Even if you spend more time automating a specific test then running it manually, long term you will gain more time if that test case is automated.
I also need to mention that, from the beginning, you should carefully design your automation framework so that is easy to maintain, and you don't have to lose time trying to refactor or improve the framework in the future.
Well, according to classical approach you should consider the costs of automated tests implementation as the investments and thus choose the tests whose automation would give you the maximum ROI.
The challenge here is to calculate NetProfit value since this depends on your particular project specific and what you actually need to minimize (either the cost of testing or the time of testing).
You should take into account the average number of builds in release, test priorities (which are to be revised since the value of the test might change), time of each particular test execution and estimated time of the implementation and maintenance of the test and test execution environment.
Then you sort your tests by ROI value and plan the implementation according to the resources you have.
It takes a lot of effort to manage and prioritize tests. Coverage is one of several good approximations.
One way to prioritize is to use tools to measure which parts of application are used most often (which is different from coverage), and focus testing accordingly to cover those parts.
Another way is to use expert opinion and rules of thumb: Write a "happy path test" for most important functionality from user POV, and for functionality which is important for safety/security (cost will be high if your company was sued for providing wrong info). Measure test coverage and add more to to increase it.
Coverage is of course separate complicated issue (do your tests need to cover error handling which is rare and hard to trigger? etc).
-
Peter, implicit in your answer is the idea that the tester and testing is an outsider to the system/code under test. By using "Coverage", "used more often", and "expert opinion" (meaning to me a customer of the system.) we as testers can become insiders. I like your answer - Thanks! Jun 3, 2018 at 22:45
-
@JohnBurley Testers are outsiders in the sense that they need less understanding of intimate details how the code works (if they understand more, they could be developers). It is good to have a second pair of the eyes who understand the requirements but were not involved in development to test the requirements. Developers could be blind to some detail or use some unsaid assumption, hence the benefit of second pair of eyes. Jun 4, 2018 at 15:12
-
@JohnBurley "Expert opinion" is opinion by someone who understand system intimately - lead developer or system architect. Customer is not expert about how the system works. Customer is expert about how (and for what) system is used, but customer understands only customer-facing surface of the system, and does not have (and does not need) understanding of the intimate details of the system (as long as his/her mental model is close enough approximation). Jun 4, 2018 at 15:15
Software Testing is primarily a risk-based activity so test cases should be chosen wisely for frequent execution, aiming maximum return with least investment.
There are primarily three types of test suites executed during each release of a software application: regression tests, release specific tests, and defect fixes verification tests.
I would select the below test cases as part of regression suite:
that have frequently caught bugs: Some areas in the application are so error-prone that they usually fail following a small coding change. We can keep track of these failing test cases throughout the product cycle and cover them in the regression test suite.
that verify core features of the application: Prior to designing the test cases, identify all the core features of the application. Ensure that test cases cover all functionality mentioned in the requirements document. One can make use of a traceability matrix to make sure that no requirement is left untested.
for functionalities that have undergone recent changes: Maintain the history of functionality changes for test case documentation in order to identify the test cases to include in the regression suite.
Covering integration scenarios: This includes all those integration test scenarios on the functional boundary where two modules handshake and pass data and integration happens.
that are complex in nature: Some system functionality may only be accomplished by following a complex sequence of UI events.
that have high business priority: Prioritize the test cases as they relate to business impact and critical and frequently used functionalities. It is always helpful if an analysis is completed to determine which test cases are relevant. One idea is to classify the test cases into various priorities based on importance and customer use. The selection of test cases based on priority will greatly reduce efforts spent on regression testing.
On a “case-to-case” basis: There is always room for expert judgment and experience to pick test cases on a case-to-case basis, depending on each company/domain/project/team/ feature etc.
There are several factors that I consider when determining which manual test cases to automate
- How often tests are run manually
- How easy the manual emulators and simulators are to use
- How long the manual tests take to get the same feedback as the automated cases
- The availability and cost of manual testers
- How many variations need to be run
- What unit and integration automated test coverage exists
- The inability of humans to do large number of boring things consistently
Overall I follow the test type categorization for each group of tests (often representing workflows) of:
- One smoke test
Usually that the first page is displayed before any data entry (catches 404's, 500s) - Several sad path tests
For when the user or system has an error - One Happy path tests
When everything works for the given workflow - Optional Happy/Sad path tests
When there is an optional part of the workflow, e.g. a non-required screen
For raw time comparison (leaving aside human factors), its really just a matter of what timeframe you look at and number of runs. For example:
A manual test take 30 minutes.
Automating it takes 4 days, e.g. 4x8=32, 32*60= 1920 minutes and then once automated takes 1 minute to run.
Therefore if this test is run 4 times a day (common in places with CI/CD) then over 6 months, manual would be 183x10 = 3660 mins whereas automated would be 1920 (to write) + 183x1 = 2103
Just as importantly for many development shops that want to do rapid iterations, agile development, CI/CD, etc. it it just not possible to do large scale manual testing in a fast and efficient manner. Even with 1000 off-shore folks bashing away, co-ordinating them starts to take a huge amount of time. Plus humans tend to want continual wages of course.
Overall, as with so many Quality Assurance and Quality Engineering issues I feel it comes down to The Value Proposition. Show how the costs, savings and revenue play out over time and automation can be an easier sell.