Suppose, your company develops mobile apps for different platforms with different teams & technologies.

Which points would you put on the checklist to analyse the ROI (or usefulness) of automation?

For example:

  • Is the app native/hybrid?
  • Could we modify the source code?
  • How many releases are planned in the next year?
  • How big is the user base?
  • Are there any unit-test done?
  • What programming languages does the team master?

What else is important?

  • 2
    Profitability is poor term to use IMO. The 'profitability' of thorough testing (which automation would be a part of) is essentially the company's profitability (if the company is a web based company). I think ROI would be a better choice as the cost comparison is more about the return on the investment and not how much money you will make from automation. – Paul Muir Jan 7 '15 at 16:25

You have mixed questions which fall in multiple categories

  • Automation Coverage (Functional / Regression Maintenance)
  • Automation Design Aspects - Maintainability
  • Automation ROI timeline vs Cost of Development

Each Factor need to analysed separately

  • How do you develop automation, Is it extensible, How easy to modify / add new cases ?
  • Duration of development, What is the effort saved in manual testing
  • What is the coverage of Automation for Functional / Regression cases - Device Coverage ?

If you have thought through Automation Strategy (Functional/ Regression), Development efforts, Savings (Immediate / Recursive Release over Release) you will have answers for the same.

@Alexei - I have observed, if you follow below points in checklist you will definitely think of high percentage of ROI :-

Manual vs Automated – Testing vs Checking

  • Avoid comparison between manual and automated testing. They are both needed as each serves a different purpose. Automated tests are a set of instructions written by a person to do a specific task. Every time an automated test is run, it will follow exactly the same steps as instructed and only check for things that is being asked to check.
  • On the other hand, during manual testing, tester’s brain is engaged and can spot other failures in the system. The test steps may not necessarily be the same every time, as the tester can alter the flows during the testing; this is specially true in case of exploratory testing.

Automate Regression Tests

  • The main reason you want to automate a test, is because you want to execute the test repeatedly on every new release. If the test requires to be executed only once, then the effort to automate the test can outweigh the benefits.

  • Regression tests are required to be executed repeatedly as the software under test evolves. This can be very time consuming and a boring task for QA to have to run regression tests every day. Regression tests are good candidates for test automation.

Design Tests Before Automating Them

  • It is always a good practice to create the test cases and scenarios before starting to automate the tests. It is the good test design than can help in identifying defects, automated tests only execute the test design.

  • The danger in jumping straight to automation is that you’re only interested in making the script to work and usually only automate positive and happy flow scenarios rather than thinking about the other possible scenarios that can be tested.

Remove Uncertainty from Automated Tests

  • One of the key points of automated testing is the ability to give consistent results, so that we can be certain that something has actually gone wrong when a test fails.

  • If an automated test passes in one run and fails in the next run, without any changes on the software under test, we cannot be certain if the failure is due to the application or due to other factors, such as test environment issues or problems in the test code itself.

  • When there are failures, we have to analyse the results to see what had gone wrong, and when we have lots of inconsistent or false positive results, it increases analysis time.

  • Don’t be afraid to remove unstable tests from regression packs; instead aim for consistent clean results that you can rely on.

Review Automated Tests for Validity

  • You will be alarmed by the sheer number of automated tests that are outdated, just don’t check for anything or are not checking the most important verifications!

  • This could be a symptom of jumping straight to automation without spending enough time before hand planning on what needs to be done and designing good test scenarios.

  • Always have a colleague to review the automated tests for validity and sanity. Make sure tests are up to date.

Don’t Automate Unstable Functionality

  • As a new feature or functionality is being developed, many things can go wrong and even the feature may no longer be applicable because the business have changed their mind.

  • If you started automating tests as the feature was being developed, the tests need to be updated many times as the feature evolves and can be quite daunting trying to keep up with all the changes. And if the feature is no longer applicable, all that effort on test automation is wasted.

  • Therefore, it is always best to automate a functionality once it has been stabilized and less subject to change.

Don’t Expect Magic From Test Automation

  • The primary reason for test automation is to free up QA time for interesting exploratory testing and to give confidence to the team that the application is still in good order as new changes are delivered.

  • Don’t expect automation to find lots of bugs. In fact, the number of bugs found by automation is always much less than manual and exploratory testing.

Don’t Rely Solely on Automation – Beware of Passing Tests

  • Automated regression tests can give a sense of confidence for the team because regression tests should still pass as new functionality is delivered.The team start relying on the tests and having a good set of regression tests can act as a safety net.

Aim for Fast Feedback

  • Quick feedback is one of the objectives of automated tests, because developers are keen to know if what they have developed works and hasn’t broken current functionality.

Understand the Context

  • Tests can be automated at any layer, Unit, API, Service, GUI. Each layer serves a different purpose for testing. Unit Tests ensure that the code works at class level, that it compiles and the logic is as expected. Tests at this layer are more verification than validation.

Don’t Automate Every Test

  • 100% Test Coverage is not possible since there can be millions of combinations. We always execute a subset of possible tests. The same principle applies to automated testing.

  • To create an automated script, it requires time and effort, and aiming for “Automating Every Test”, we require lot of resource and time, which in many cases is not possible.

Use Test Techniques in Test Automation

  • The test techniques that you learned in ISTQB, are not just for manual testing. They are also applicable to automated testing. Techniques such as Boundary Value Analysis, Equivalence Partitioning, State Transition Testing, Pairwise Testing can provide a lot of benefits in automated testing.

Don’t Automate Chaos

  • In order to get the most out of your automated testing, a good QA process should be in place. If the QA process is chaotic and we add automated testing to that chaos, all we get is faster chaos.

  • Try to answer questions like, What to automate, When to automate, When to execute the automated tests, Who shall automate the tests, What tools should be used for test automation, etc…

I think if you want to use ROI you are most likely talking to business/management and giving metrics about code coverage etc. will have little to no impact. In my opinion this will actually leave you two main aspects to analyze:

  • Increased speed to give ok for go-live - all the time gained can usually be converted to financial gain. You can factor in also the speed of finding bugs if you have the information, but that feels like huge overhead

  • Increased detection of regressions. you can put a price on these if you match how big of a revenue loss the bugs (that were missed in regression testing and went to live) caused for the company.

But are you certain you want to stick to ROI? Usually when questions like this come up means that some of the stakeholders do not have enough understanding what qa's do and why automation is useful - in my experience a short presentation for concerned party is enough to get over this hurdle.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.