Is the return on investment of any particular kind of testing measurable?

Question

It is easy to find claims about the benefit of different kinds of testing, particularly when the testing technique requires purchasing a product or spending money on consultants. Has anyone ever attempted to measure the ROI of a testing technique? If so, how did you do it and were you successful?

Answer 1

Measuring ROI of just about anything is subjective because it often depends on context. For example there are many test automation ROI calculators and formulas running around the Internet, but all assume that we all automate our tests the same way, and they fail to take into account intangible value from automated tests (e.g. confidence, stability, etc.)

I and others have done case studies to evaluate benefits and limitations of various testing approaches in controlled settings. Of course, controlled settings don't necessarily mimick 'real-world' environments which is why some people reject these types of studies. But, sometimes even similar studies reveal a pattern that is rejected in favor of more commonly accepted folklore and emotional attachment.

Bottom line, I don't think there is a one size fits all way to measure ROI of any testing approach; however, I do think that we can assess various testing approaches in different situations and derive patterns of benefits and limitations of those test approaches within appropriate contexts.

Answer 2

Yes, it can be measured

but NOT always quantitatively !

Some examples which I have had experience with -

Quantity -

1. Man hours/$s saved by automating a set of regression tests which were previously taking X no. of man hours

2. Further to above, those Man hours saved yielded in these number of new bugs detected( because testers could focus on "other" stuff)

3. Adopting this new tool, has reduced the app compile time by x number of minutes .

4. Adopting this new tool , has reduced the test execution time by x number of times .

In tangible -

1. This process change/tool has resulted in reduced response time by tech support and hence we have recieved postivie feedback from customers (include testimonials)

2. This process changes has resulted in improved communication between cross functional teams ( example - critical issue highlighted earlier as compared to the "older" process)

3. This process has brought out the conflict , in the open . This conflict resulted in us not delivering our last deliverable in time, the last time.

4. These changes have resulted in a happier team (elicit feedback) and new energy amongst the team.

Personally , I have experienced examples like the above,where you can advocate new initiatives/changes and try & capture both the tangible & intangible ROI on those.

Answer 3

Never in terms pf $ for $. We do however have to justify money being spent, for example we need another member for the execution team because we can only get to 50% coverage with out existing staff, or we need that test equipment since it will allow us performance testing in real life throughputs. In my industry (embedded, consumer devices) and position (one of many test teams, faraway from upper management) it is almost impossible to produce a meaningful ROI estimation.

Answer 4

This is the only thing I have come across, though I'm sure there have been other studies:

An Initial Investigation of Test Driven Development In Industry