3

I am currently looking into methods to quantify the effect of QA on a team, be this in terms of overall productivity improvement or net effectiveness. In doing so I am evaluating various methods by which I can quantify this.

For example, how many bugs are raised and at what stage of the agile process. Then from that calculate the developer time saved.
There is also potential to couple the ongoing saving of automated tests and their associated maintenance costs. Or to measure the most effective means of QA involvement

What metrics do you currently use and how you compile the dataset for effective comparison?

Edit: Please dont respond with "test early". That is not a sufficiently robust argument for planning future qa expansion and project involvenent

  • We don't. Be careful with what you wish for, there is a good chance you might end up proving that testers waste time rather than save it. I s there a reason you need to prove this ? – Rsf Jan 14 '16 at 12:29
  • What we are trying to establish is the ROI of testers and QA fulfilling differnt roles and functions within the company. Are we deploying our resources effectively, is there a skills gap? Are to many bugs reaching to late a stage in the cycle where an additional tester might have mitigated it. That kind of thing. Although its nice to assume testers are unquantifiable, we are. Otherwise no one would hire us other than to tick a box! – ECiurleo Jan 14 '16 at 16:28
  • 1
    This can often vary depending on what type of product/company you are working within. For example are we talking about an eConmerce website, a physical device with embedded software, a POS app running on an iPad, manufacturing plant systems, etc.? What metrics you record and how you analyze them will very with your product. How well other things are documented outside of QA will also affect what insights you can gain. I only ask so that I can write more specific answer. – CodeBreaker Jan 14 '16 at 18:55
  • Ecommerce, with public facing ingestion systems. We have a wealth of data which is dev focus from jira and other tools (which are shared by the testers and agile team). – ECiurleo Jan 14 '16 at 20:37
5

By the success of the company.

Buy-in for QA will need to come from the top rather then the justification being looked for in data.

You will, over time, be able to point to things like

  • some major bugs prevented from reaching customers
  • performance issues anticipated and planned for
  • unusual bugs discovered for certain conditions
  • more new customers
  • more retained customers

However statistics like number of bugs per project or person or team, or bugs per sprint, are often not very useful as the various factors that contribute are so variable. Measureing number of bugs or bugs per developers also doesn't focus on other issues that affect users such as usability, design, branding, device compatibility, preferences, etc.

Given that numbers and stats are so important to folks that run the company you can also try to amass and present information on:

  • significant bugs found pre-production
  • best practices for usability to help increase customer retainment and brand loyalty
  • best practices for long term maintainability and reduced cost.
  • analysis of real user experience to show actual pain points.
  • who's using what platforms, devices, browsers and versions and in what proportion.

One example (2016) - we AB test changes to our car insurance web site. We recently update our forms to correctly highlight errors, NOT rely solely on red and give better feedback. This was an effort championed by the QA department based on accessibility best practices. Result: 1.5 million dollars a year more in revenue.

| improve this answer | |
1

I've never used metrics in this way, however, you can build a case with some care and a lot of data mining.

First find out the average time cost per bug found by customers. You're going to need to find out how much time is spent in diagnosis, fixing, devising workarounds, and pushing patches, and average over the total number of bugs (this will never be anything but a crude measurement because no two bugs are equal, but if your management wants numbers, it's a way to give them numbers).

Next, look at the average time cost per bug found by the test team. This is also going to be a very crude approximation, but can be done via the amount of time spent per bug.

Rinse and repeat the time cost per bug calculations for each step of the process as best you can. What you should find is that on average each bug caught before release saves the company some number of hours developer time. If your company charges a per-hour fee for custom work, use that number to calculate savings, because time spent working on bug fixes is time not spent working on customer requests or new features.

As long as you emphasize that it's an average and the actual differential is going to vary (also, that a good test team is going to catch all the critical bugs prior to release, so the company shouldn't have to face the cost of a humiliating public failure), you have a starting point for discussion.

Never use these metrics to evaluate the team performance. I can't stress this enough. These are averages and act as proxies for customer goodwill retained by relatively bug-free releases. They don't say anything about the performance of the team as a whole or individual team members.

| improve this answer | |
1

Metrics are misleading. You can catch 10 000 useless bugs, but if the one that kills a few patients in the hospital is not caught, you're dead. Well, dead in terms of career, not as much as the hospital's patients.

But metrics can be useful to convince people who believe only in them, unfortunately. If, and only if, the rationale behind the metrics is to convince your hierarchy QA is useful, then begin to think about what they want to hear. Usually, it's about money(adapt if they have other things in mind), and you can compare the cost of an early correction to the cost of a late correction. You can add also the "image" effect of being on delay.

There is literature covering the topic. It can help you. But remember : the real effect is not measurable in numeric terms. Its only real usefulness is to convince people who don't move as long as they don't have numbers that QA is useful. The real bottom line is that without a QA team, in the modern software world, you're going to vanish from the market. That's why both Michael Durrant & Rsf did insist in their excellent answers that they don't need metrics. They are not useful for doing a proper job.

| improve this answer | |
  • Our image is actually very good in the company. We are respected as experts/specialists within the scrums. In the same way you can perform A/B of paired programmers against traditional, i feel this can be extended to testing capacity. Is automated testing yielding the results from the investment or should we improve our exploratory testing capacity for example – ECiurleo Jan 14 '16 at 16:32
0

This is sort of a slippery slope sort of thing to get in, because if you measure by amount of bugs...that's not always accurate. But in some cases it is difficult for managers/etc... to really see the value of QA. Especially in cases where people expect a completely bug free system or otherwise "QA is failing to do their job".

The easiest thing to remember is the time spent. So without QA lets say QA finds 5 bugs on a project during testing time. And lets pretend without QA only 2 bugs would've been found. Each step through the process till release a bug costs more and more time. (Like for example a bug in production wastes 10 hours of time, instead of just 1 hour if a bug is found during test). That's sort of the easiest "financial" argument IMO.

Either way, i'd be careful before diving into metrics.

| improve this answer | |
  • 1
    This is effectively telling the corporation "trust me, qa saves money". That isnt really enough and not what the question is asking – ECiurleo Jan 14 '16 at 16:36
  • Well something you can do is if you can find some statistical data on a project where QA was done and find the hours spent fixing X issues. and then compare it to the amount of time X issue costed in production is probably the best "basic" case to show them. I believe gazzz02x2z showed a chart like that, but it's always best if you have a relevant case in your company. I personally think saying "Hey we found X bugs in this one project" might sort of set a precedent that they will start measuring you by (Which is not a good thing) – Mercfh Jan 14 '16 at 16:45
0

Do you have the opportunity to compare two projects delivered one with and one without the QA effort? If projects are comparable you should realize that bugs reported by customers are less both as amount and severity. If not, your QA team could not be set up as it should. Non conformities or variances reported by customers are a huge price to pay not only in terms of money but even of reputation.

| improve this answer | |
  • Actually i have 3 comparible projects coming up. All agile all with the same number of devs. How many testera and what skills we deploy to each are open. Its a great opportunity to compare the return of different methods. But unless you use some form of metric defined at the start that is consistent, the opportunity will be lost – ECiurleo Jan 14 '16 at 18:40
  • Given that it's not straightforward finding a standard metrics for measuring the impact of QA (except for the higher cost) before the completion of the project. As I wrote in my answer above you can quantify the amount of bugs and related severity reported for a project delivered with the involvement of QA and of a different one delivered without QA and then evaluate the difference. – Luca Giuffrida Jan 16 '16 at 8:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.