Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.
Sign up to join this community
Suppose, your company develops mobile apps for different platforms with different teams & technologies.
Which points would you put on the checklist to analyse the ROI (or usefulness) of automation?
For example:
What else is important?
@Alexei - I have observed, if you follow below points in checklist you will definitely think of high percentage of ROI :-
Manual vs Automated – Testing vs Checking
Automate Regression Tests
The main reason you want to automate a test, is because you want to execute the test repeatedly on every new release. If the test requires to be executed only once, then the effort to automate the test can outweigh the benefits.
Regression tests are required to be executed repeatedly as the software under test evolves. This can be very time consuming and a boring task for QA to have to run regression tests every day. Regression tests are good candidates for test automation.
Design Tests Before Automating Them
It is always a good practice to create the test cases and scenarios before starting to automate the tests. It is the good test design than can help in identifying defects, automated tests only execute the test design.
The danger in jumping straight to automation is that you’re only interested in making the script to work and usually only automate positive and happy flow scenarios rather than thinking about the other possible scenarios that can be tested.
Remove Uncertainty from Automated Tests
One of the key points of automated testing is the ability to give consistent results, so that we can be certain that something has actually gone wrong when a test fails.
If an automated test passes in one run and fails in the next run, without any changes on the software under test, we cannot be certain if the failure is due to the application or due to other factors, such as test environment issues or problems in the test code itself.
When there are failures, we have to analyse the results to see what had gone wrong, and when we have lots of inconsistent or false positive results, it increases analysis time.
Don’t be afraid to remove unstable tests from regression packs; instead aim for consistent clean results that you can rely on.
Review Automated Tests for Validity
You will be alarmed by the sheer number of automated tests that are outdated, just don’t check for anything or are not checking the most important verifications!
This could be a symptom of jumping straight to automation without spending enough time before hand planning on what needs to be done and designing good test scenarios.
Always have a colleague to review the automated tests for validity and sanity. Make sure tests are up to date.
Don’t Automate Unstable Functionality
As a new feature or functionality is being developed, many things can go wrong and even the feature may no longer be applicable because the business have changed their mind.
If you started automating tests as the feature was being developed, the tests need to be updated many times as the feature evolves and can be quite daunting trying to keep up with all the changes. And if the feature is no longer applicable, all that effort on test automation is wasted.
Therefore, it is always best to automate a functionality once it has been stabilized and less subject to change.
Don’t Expect Magic From Test Automation
The primary reason for test automation is to free up QA time for interesting exploratory testing and to give confidence to the team that the application is still in good order as new changes are delivered.
Don’t expect automation to find lots of bugs. In fact, the number of bugs found by automation is always much less than manual and exploratory testing.
Don’t Rely Solely on Automation – Beware of Passing Tests
Aim for Fast Feedback
Understand the Context
Don’t Automate Every Test
100% Test Coverage is not possible since there can be millions of combinations. We always execute a subset of possible tests. The same principle applies to automated testing.
To create an automated script, it requires time and effort, and aiming for “Automating Every Test”, we require lot of resource and time, which in many cases is not possible.
Use Test Techniques in Test Automation
Don’t Automate Chaos
In order to get the most out of your automated testing, a good QA process should be in place. If the QA process is chaotic and we add automated testing to that chaos, all we get is faster chaos.
Try to answer questions like, What to automate, When to automate, When to execute the automated tests, Who shall automate the tests, What tools should be used for test automation, etc…
You have mixed questions which fall in multiple categories
Each Factor need to analysed separately
If you have thought through Automation Strategy (Functional/ Regression), Development efforts, Savings (Immediate / Recursive Release over Release) you will have answers for the same.
I think if you want to use ROI you are most likely talking to business/management and giving metrics about code coverage etc. will have little to no impact. In my opinion this will actually leave you two main aspects to analyze:
Increased speed to give ok for go-live - all the time gained can usually be converted to financial gain. You can factor in also the speed of finding bugs if you have the information, but that feels like huge overhead
Increased detection of regressions. you can put a price on these if you match how big of a revenue loss the bugs (that were missed in regression testing and went to live) caused for the company.
But are you certain you want to stick to ROI? Usually when questions like this come up means that some of the stakeholders do not have enough understanding what qa's do and why automation is useful - in my experience a short presentation for concerned party is enough to get over this hurdle.
