How do you calculate your return on investment on automated tests?

Question

I've been looking at calculating the ROI on automated test on a project that I've been working on. However, I'm a little stuck when it comes to determining a good method for this. Simply counting the defects found when executed on the continuous integration sever doesn't really help as many developers run the test locally before checking in code, so I can accurately count how many defect have been found. So my question is how would I calculate this? My gut feeling is that they have massively helped, but I need to quantify this!

Answer 1

The easiest way I can think of is to base it on how many hours the qa people are saving by not having to regression test all the scenarios covered by the automated test. If the response is they wouldn't regression test it, the answer to that that they should. The ideal scenario is that the entire application should be tested from one release to the next, but this in most situations just isn't possible. This is where automated testing really gives lift. You can have the QA subject matter expert write up the scenario, and then not worry about having to test it again.

Answer 2

I can very quickly think of three benefits that should be somewhat measurable. (1) Freeing up testers from the monotony of repeating regression tests and allow them to focus on new development and areas that need their attention. Add up the time they are not spending on regression & show how that goes to new projects.

(2) Accelerated testing window. Especially when R&D is checking in code frequently, catching a regression problem sooner gives them a much narrower window of when the bug was introduced. Think about it, if I can say Test X passed 3 days ago & failed today, the developer only has 3 days of code to worry about instead of 3 months worth. Track how quickly those are caught compared to the days of manual regression.

(3) Provides a much better level of testing for smaller releases. In service packs, patches and hot fixes that have a very tiny testing window you there isn't always time to manually run the regression tests, but if they are automated you now have the option of running those tests even in that short time. Count the number of regression tests run in a service pack that never would have been possible before.

Answer 3

I'm sure that the automated tests have helped and there should definitely be a ROI present. The question is, since there are several different types, what kind of ROI are you getting for the automated tests you have.

• Time savings - This one should be more quantifiable than other types of ROI. Time savings comparing manual test execution to automated execution should be useful in building a good case.
• Repeatability - Automated tests are executed precisely the same way every time they are executed. This might not seem like a big return, but consider how many steps would need to be executed manually if these tests were not present and then consider how many mistakes might be made with that effort.
• Traceability - Automated tests (usually) provide some type of evidence of execution. It could be a traditional log or a simple pass/fail output from execution, but either way, automated tests can keep better track of what was measured during execution and provide the data with less variation than a manual tester.
• Availability - Automated tests can be kicked off manually or automatically to run at any hour of the day. They don't get tired, sick, or take vacations, and they can be made to work weekends without special arrangements.

These are just a few off the top of my head, but you will be the best person to determine where these tests benefit you the most. Sit down and really think about the positive additions these test have provided for your project.

Answer 4

It's hard to put a price on catching a bug before it goes out the door to a customer. It may be a small problem that can be corrected in the next release, or it may be the showstopper that keeps you at the office frantically trying to fix it so that you can get it to the customer as soon as possible.

If you are relying on your automation to check functionality so that you can avoid checking that functionality manually before shipping, then I believe the ROI can be calculated for:

1. Hours your team has spent on similar customer issues, emergency or otherwise.
2. Customer's satisfaction and trust in you as a vendor.

Answer 5

First question: why do you feel that you need to quantify this? Are you being asked to provide metrics? Would a qualitative evaluation be more valuable here?

It also sounds to me like you're focusing at the wrong level - i.e. on too small a group: it's not so much about making the testers more effective, or the developers more effective, but about making the team as a whole more effective. Can you get feedback from the developers about how it's helped them? Is there anything that it's allowed you to do that you wouldn't have been able to do before (turn around an urgent change in 2 days without having to spend 3 weeks on regression testing, for example, or add functionality partway through a project which let you keep/win a major customer), that would be appealing to your stakeholders?

I can't write an answer about metrics without linking Cem Kaner's paper on metrics.

Answer 6

I can't tell whether the OP is a tester or a developer. I'll assume they're a tester.

It is difficult to measure the rate of return on the investment on automated tests because the relative cost and the return is speculative.

Here is my personal experience. In some cases, an automated test may be the only way to reproduce a problem or to verify that a change has made the product better. In mosts tests, automated tests uncover bugs that manual testing may have uncovered anyway.

Testers who have the skills to write automated tests are more expensive that those who do not. And testers who prefer to automate may, intentionally or subconsciously, avoid testing features that are difficult or cumbersome to automate.

It may make more sense to start by figuring out the rate of return of your QA organization as a whole.

Answer 7

Keep in mind that automated testing will, for the most part, only catch things that "flesh-and-blood testers" have already caught. This is a good thing, because we want them thinking about new ways and things to test, not going over the same ground again and again.

A rule we had at my last firm (and I'm trying to get at my present firm) is for every bug, there's a test to catch it. This has a big impact. If you have a bug, and you release a patch that says "Hey customers, we have listened to you! We have fixed this bug you reported! Yay for us!" and then it turns out you didn't actually fix it, not only are they mad because you didn't fix, you've lost some trust from your customers.

So - how much is your customers trust worth to you?

Answer 8

I don't see this as an exact science so you need to make some assumptions. But if you can put some numbers, money or time, in what it cost to find a bug in development (dc), test (tc) and prod (pc). You can then give some different scenarios, let say automation decreases the number of bugs found in dev with x, in test with y and in prod with z. Then you have an estimate that is total savings = x*dc + y*tc + z*pc - (cost of automation).

That's a really simple model to get you started. I haven't used it myself but that's how I would start if I was to write an ROI for automation.