17

I've been looking at calculating the ROI on automated test on a project that I've been working on. However, I'm a little stuck when it comes to determining a good method for this. Simply counting the defects found when executed on the continuous integration sever doesn't really help as many developers run the test locally before checking in code, so I can accurately count how many defect have been found. So my question is how would I calculate this? My gut feeling is that they have massively helped, but I need to quantify this!

Improve this question

9 Answers 9

Reset to default
13

The easiest way I can think of is to base it on how many hours the qa people are saving by not having to regression test all the scenarios covered by the automated test. If the response is they wouldn't regression test it, the answer to that that they should. The ideal scenario is that the entire application should be tested from one release to the next, but this in most situations just isn't possible. This is where automated testing really gives lift. You can have the QA subject matter expert write up the scenario, and then not worry about having to test it again.

Improve this answer
1
  • 1
    I've used this approach, we invested 5800 hours approx. which saved the project team form executed around 28000 hours approx, a total saving of 22200 hours (again approx.)!
    – stuartf
    Commented May 17, 2011 at 21:42
6

I'm sure that the automated tests have helped and there should definitely be a ROI present. The question is, since there are several different types, what kind of ROI are you getting for the automated tests you have.

  • Time savings - This one should be more quantifiable than other types of ROI. Time savings comparing manual test execution to automated execution should be useful in building a good case.
  • Repeatability - Automated tests are executed precisely the same way every time they are executed. This might not seem like a big return, but consider how many steps would need to be executed manually if these tests were not present and then consider how many mistakes might be made with that effort.
  • Traceability - Automated tests (usually) provide some type of evidence of execution. It could be a traditional log or a simple pass/fail output from execution, but either way, automated tests can keep better track of what was measured during execution and provide the data with less variation than a manual tester.
  • Availability - Automated tests can be kicked off manually or automatically to run at any hour of the day. They don't get tired, sick, or take vacations, and they can be made to work weekends without special arrangements.

These are just a few off the top of my head, but you will be the best person to determine where these tests benefit you the most. Sit down and really think about the positive additions these test have provided for your project.

Improve this answer
4
  • 3
    Repeatability is actually a disadvantage if you're looking to find bugs - the natural human variation that comes in with manual testing allows far greater coverage. Good if what you mainly want is to be reasonably sure that a failure indicates some change since last run though, which is generally what you want from a regression suite. And you can write your automation to introduce variation.
    – testerab
    Commented May 11, 2011 at 20:33
  • 3
    @testerab are automated tests really supposed to find new bugs? As you say, regression tests mainly exist to make sure old bugs don't return. Exploratory testing (which really can't be fully replaced with automated tests) is what will find new bugs and utilize human variation to its fullest.
    – Steven
    Commented May 11, 2011 at 22:27
  • 1
    Absolutely agreed that exploring is what finds more bugs: but exploring doesn't have to be purely manual. It just needs to have a human brain in the loop somewhere. This article is an interesting read: developsense.com/blog/2010/09/…
    – testerab
    Commented May 11, 2011 at 23:57
  • 2
    Our regression suite is there not to find new bugs but to give us a clear indication that the build is good enough to perform further testing. I think that it is important to have a deterministic regression suite and have another suite for fuzz testing that will try to find new defects.
    – stuartf
    Commented May 17, 2011 at 21:40
6

I can very quickly think of three benefits that should be somewhat measurable. (1) Freeing up testers from the monotony of repeating regression tests and allow them to focus on new development and areas that need their attention. Add up the time they are not spending on regression & show how that goes to new projects.

(2) Accelerated testing window. Especially when R&D is checking in code frequently, catching a regression problem sooner gives them a much narrower window of when the bug was introduced. Think about it, if I can say Test X passed 3 days ago & failed today, the developer only has 3 days of code to worry about instead of 3 months worth. Track how quickly those are caught compared to the days of manual regression.

(3) Provides a much better level of testing for smaller releases. In service packs, patches and hot fixes that have a very tiny testing window you there isn't always time to manually run the regression tests, but if they are automated you now have the option of running those tests even in that short time. Count the number of regression tests run in a service pack that never would have been possible before.

Improve this answer
3

I can't tell whether the OP is a tester or a developer. I'll assume they're a tester.

It is difficult to measure the rate of return on the investment on automated tests because the relative cost and the return is speculative.

Here is my personal experience. In some cases, an automated test may be the only way to reproduce a problem or to verify that a change has made the product better. In mosts tests, automated tests uncover bugs that manual testing may have uncovered anyway.

Testers who have the skills to write automated tests are more expensive that those who do not. And testers who prefer to automate may, intentionally or subconsciously, avoid testing features that are difficult or cumbersome to automate.

It may make more sense to start by figuring out the rate of return of your QA organization as a whole.

Improve this answer
2
  • Should the answer change depending on who asks the question? It would seem it's rather objectively asked, which means it wouldn't matter.
    – corsiKa ♦
    Commented May 10, 2011 at 20:56
  • @glowcoder My comments about the rate of return of a QA organization are not relevant to the OP's question if the OP is asking from the perspective of a developer -- outside of the QA organization -- who writes automated tests. Ditto for my comment about testers who write automated tests.
    – user246
    Commented May 10, 2011 at 22:45
3

It's hard to put a price on catching a bug before it goes out the door to a customer. It may be a small problem that can be corrected in the next release, or it may be the showstopper that keeps you at the office frantically trying to fix it so that you can get it to the customer as soon as possible.

If you are relying on your automation to check functionality so that you can avoid checking that functionality manually before shipping, then I believe the ROI can be calculated for:

  1. Hours your team has spent on similar customer issues, emergency or otherwise.
  2. Customer's satisfaction and trust in you as a vendor.
Improve this answer
2

First question: why do you feel that you need to quantify this? Are you being asked to provide metrics? Would a qualitative evaluation be more valuable here?

It also sounds to me like you're focusing at the wrong level - i.e. on too small a group: it's not so much about making the testers more effective, or the developers more effective, but about making the team as a whole more effective. Can you get feedback from the developers about how it's helped them? Is there anything that it's allowed you to do that you wouldn't have been able to do before (turn around an urgent change in 2 days without having to spend 3 weeks on regression testing, for example, or add functionality partway through a project which let you keep/win a major customer), that would be appealing to your stakeholders?

I can't write an answer about metrics without linking Cem Kaner's paper on metrics.

Improve this answer
3
  • I've not be asked so much to provide figures, however, I would like to be able to justify this approach on new projects, and like always management talk in savings and cost.
    – stuartf
    Commented May 10, 2011 at 21:13
  • 2
    You can dig yourself into a big hole though by providing figures that aren't a good measure - once you start down that route, they then become a target. Sometimes it's better to resist the temptation to throw a whizzy chart into your report.
    – testerab
    Commented May 10, 2011 at 21:19
  • Problem is so much of what Testers do and provide is subjective and doesn't have a direct cost, so its hard to convince bean counters.
    – MichaelF
    Commented May 13, 2011 at 20:07
1

I don't see this as an exact science so you need to make some assumptions. But if you can put some numbers, money or time, in what it cost to find a bug in development (dc), test (tc) and prod (pc). You can then give some different scenarios, let say automation decreases the number of bugs found in dev with x, in test with y and in prod with z. Then you have an estimate that is total savings = x*dc + y*tc + z*pc - (cost of automation).

That's a really simple model to get you started. I haven't used it myself but that's how I would start if I was to write an ROI for automation.

Improve this answer
0

Keep in mind that automated testing will, for the most part, only catch things that "flesh-and-blood testers" have already caught. This is a good thing, because we want them thinking about new ways and things to test, not going over the same ground again and again.

A rule we had at my last firm (and I'm trying to get at my present firm) is for every bug, there's a test to catch it. This has a big impact. If you have a bug, and you release a patch that says "Hey customers, we have listened to you! We have fixed this bug you reported! Yay for us!" and then it turns out you didn't actually fix it, not only are they mad because you didn't fix, you've lost some trust from your customers.

So - how much is your customers trust worth to you?

Improve this answer
1
  • Priceless is the real answer, however, I it would be nice to know the actual cost.
    – stuartf
    Commented May 10, 2011 at 21:16
0

There are online services which allow to calculate ROI: 1. ROI Calculator 2. ROI Calculator2

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.