Means of Quantification of QA impact

Question

I am currently looking into methods to quantify the effect of QA on a team, be this in terms of overall productivity improvement or net effectiveness. In doing so I am evaluating various methods by which I can quantify this.

For example, how many bugs are raised and at what stage of the agile process. Then from that calculate the developer time saved.
There is also potential to couple the ongoing saving of automated tests and their associated maintenance costs. Or to measure the most effective means of QA involvement

What metrics do you currently use and how you compile the dataset for effective comparison?

Edit: Please dont respond with "test early". That is not a sufficiently robust argument for planning future qa expansion and project involvenent

Answer 1

By the success of the company.

Buy-in for QA will need to come from the top rather then the justification being looked for in data.

You will, over time, be able to point to things like

• some major bugs prevented from reaching customers
• performance issues anticipated and planned for
• unusual bugs discovered for certain conditions
• more new customers
• more retained customers

However statistics like number of bugs per project or person or team, or bugs per sprint, are often not very useful as the various factors that contribute are so variable. Measureing number of bugs or bugs per developers also doesn't focus on other issues that affect users such as usability, design, branding, device compatibility, preferences, etc.

Given that numbers and stats are so important to folks that run the company you can also try to amass and present information on:

• significant bugs found pre-production
• best practices for usability to help increase customer retainment and brand loyalty
• best practices for long term maintainability and reduced cost.
• analysis of real user experience to show actual pain points.
• who's using what platforms, devices, browsers and versions and in what proportion.

One example (2016) - we AB test changes to our car insurance web site. We recently update our forms to correctly highlight errors, NOT rely solely on red and give better feedback. This was an effort championed by the QA department based on accessibility best practices. Result: 1.5 million dollars a year more in revenue.

Answer 2

This is sort of a slippery slope sort of thing to get in, because if you measure by amount of bugs...that's not always accurate. But in some cases it is difficult for managers/etc... to really see the value of QA. Especially in cases where people expect a completely bug free system or otherwise "QA is failing to do their job".

The easiest thing to remember is the time spent. So without QA lets say QA finds 5 bugs on a project during testing time. And lets pretend without QA only 2 bugs would've been found. Each step through the process till release a bug costs more and more time. (Like for example a bug in production wastes 10 hours of time, instead of just 1 hour if a bug is found during test). That's sort of the easiest "financial" argument IMO.

Either way, i'd be careful before diving into metrics.

Answer 3

I've never used metrics in this way, however, you can build a case with some care and a lot of data mining.

First find out the average time cost per bug found by customers. You're going to need to find out how much time is spent in diagnosis, fixing, devising workarounds, and pushing patches, and average over the total number of bugs (this will never be anything but a crude measurement because no two bugs are equal, but if your management wants numbers, it's a way to give them numbers).

Next, look at the average time cost per bug found by the test team. This is also going to be a very crude approximation, but can be done via the amount of time spent per bug.

Rinse and repeat the time cost per bug calculations for each step of the process as best you can. What you should find is that on average each bug caught before release saves the company some number of hours developer time. If your company charges a per-hour fee for custom work, use that number to calculate savings, because time spent working on bug fixes is time not spent working on customer requests or new features.

As long as you emphasize that it's an average and the actual differential is going to vary (also, that a good test team is going to catch all the critical bugs prior to release, so the company shouldn't have to face the cost of a humiliating public failure), you have a starting point for discussion.

Never use these metrics to evaluate the team performance. I can't stress this enough. These are averages and act as proxies for customer goodwill retained by relatively bug-free releases. They don't say anything about the performance of the team as a whole or individual team members.

Answer 4

Metrics are misleading. You can catch 10 000 useless bugs, but if the one that kills a few patients in the hospital is not caught, you're dead. Well, dead in terms of career, not as much as the hospital's patients.

But metrics can be useful to convince people who believe only in them, unfortunately. If, and only if, the rationale behind the metrics is to convince your hierarchy QA is useful, then begin to think about what they want to hear. Usually, it's about money(adapt if they have other things in mind), and you can compare the cost of an early correction to the cost of a late correction. You can add also the "image" effect of being on delay.

There is literature covering the topic. It can help you. But remember : the real effect is not measurable in numeric terms. Its only real usefulness is to convince people who don't move as long as they don't have numbers that QA is useful. The real bottom line is that without a QA team, in the modern software world, you're going to vanish from the market. That's why both Michael Durrant & Rsf did insist in their excellent answers that they don't need metrics. They are not useful for doing a proper job.

Answer 5

Do you have the opportunity to compare two projects delivered one with and one without the QA effort? If projects are comparable you should realize that bugs reported by customers are less both as amount and severity. If not, your QA team could not be set up as it should. Non conformities or variances reported by customers are a huge price to pay not only in terms of money but even of reputation.