How do I set specific (%) QA targets on reduction in bugs for the year?

Question

We're starting from a relatively small number of bugs currently, hence my challenge. In other words, bug slippage is currently quite low already, but I have been tasked to make it even lower.

If we had a situation, for example, where we had a rolling number of e.g. 100 customer-reported bugs open at any given time, then saying something like "reduce this to 50" would be easy. But since the bug slippage is currently quite low (although not yet measured precisely as it is difficult to do so), I am struggling with a way to come up with targets (numbers or relative numbers).

EDIT: Some additional context:

So the technology here is web scraping. Web scrapers scrape a large amount of structured data from specific sites, and what the client pays for is the data produced. Before data produced by any spider gets delivered, it goes through QA, often several rounds + fixes until it appears that the data is of sufficient quality. While this testing is manual and therefore not the most efficient, it is effective. The proportion of data issues that get missed in QA (and caught by customers) is therefore relatively small. The do though find issues, usually for one or some of the following reasons:

• They are consuming the data in a way not reproduced by QA
• They consume data from a web page or pages that QA didn't have time to check and which differ in some way from most pages being scraped.
• The customer has ways of knowing the expected total amount of items to be scraped e.g. from a large e-commerce site that QA do not have
• etc.

I already know the steps to take to reduce the amount of slippage; that's not the topic of this question. What I am looking for is for ways to set targets for the reduction in slippage, if not in % terms, then at least in some quasi-quantitative way.

Answer 1

If I understand you correctly your defect ratio is all ready very low. What effort are you going to put in reducing it to an even lower percentages? Question the ROI here, even if you technically show its possible to lower the slippage, maybe it is not worth it money wise.

Nevertheless I believe in trends, if you want a good year target the trend should be going down. If you want to use statistics read up on significant, your minimal target should meet the significant boundary. This is something you can calculate based on previous year defect ratio data. Maybe a data scientist within your company can help you.

You will need to take the ratio of defects vs changes in account. If more new crawlers or features are added then also expect an higher amount of defects. You could use a changes vs defects metric, but in larger products complexity might also play a big role as a third axis.

Personally I would rather execute a root-cause analyses for each defect and find a way to prevent it the future. Metrics can give you a false sense of security if you do not define it well enough.

Answer 2

Here is what I do, maybe this will be helpful. First, for all customer issues, we do a root cause analysis - but capture 2 bits of information: the true root cause AND the reason the bug slipped through. You can see the template of the analysis here in this article about RCA

Now, you should have a breakdown, by percentage, of the bugs that slip. For example, 22% data consumption, 45% no time, 18% large-data, 15% other... Then, think about which reasons you can affect by making improvements. So, in this scenario - maybe the "no time" category is largest and most in your control.

Answer 3

I recommend not setting specific % targets.

Instead:

• Focus on championing quality and measuring what the company values as quality.
• Learn what clients consider to be their performance targets which will differ from client to client.
• See if you can find data to reflect bottom line savings for profit making companies.
• Get senior management buy-in to long term quality.
• Learn more about your customer demographics so you can address specific concerns.

Answer 4

As some of the other folks said, I'd be reluctant to commit to a percent of defects reduction, especially since the number of defects that are found in the field at any given year is very dependent on your product cycle. Unless you can be sure that all of your users are always using the most recent version of the product, there's an unknown/unknowable number of defects that are currently baked in to your numbers. And even then, if you have 15 customers, and 5 of them bought the product, but haven't actually used it yet, they're probably going to find some number of defects whenever they do start using the product, and that's out of your control. And at some level, would be a good thing. More defects can mean lower quality, but it can also mean more users, or more usage, or different usage, or a number of other things. If you're providing some sort of managed service, you might be able to ignore some of the above, but not all of it.

If you do want to reduce defects, you need to understand them. There's a methodology called Orthogonal Defect Classification that you could take as inspiration, if not use it exactly. It does require a fair amount of historical data, and also a defined development and test process, and depending on the size of your development/test team, may not be appropriate to use as is.

But you could use part of it, namely, review all of the defects found in the field, and think about what it would have taken to find the defect prior to it being found in the field. Was it a design issue? A load/stress issue? A serialization issue? Related to a specific hardware or software configuration? A problem with some sort of recovery code? Functional regression? You need to be very honest with yourself/your organization when doing this review. At some level, once you know that a problem exists, you can probably envision ways you could have found that specific problem. But you need to think about problems in terms of classes, not individual problems.

So if, for example, if there was a video card driver that revealed a problem, the way you would have increased your chance of finding that class of problem would have involved running a host of test systems with different video cards, and many different video card driver levels. And that would be an ongoing effort, because nvidia and AMD put out drivers on a regular basis, so you'd need to test all combinations of currently supported levels of the software every time a new driver was released.

So, after you've reviewed the defects, sorted them into classes, and then thought about how you'd find that class of problems, then you can put proposals for reducing defects together. It's possible you'll find a class or classes of defects that you could find quickly and easily, and that would be great. And there may be other classes that could have been prevented entirely if the developers had code reviews, or used different compiler options, or whatever. But after any of those, there's a business decision to be made, as there are going to be either tradeoffs or some sort of investment required, and the powers that be will need to decide if finding that class of defect is worth the investment, either as a one-off cost or an ongoing investment. Or, if they don't want to make the investment, but do want to reduce the likelihood of a certain class of defect being found in the field, some other classes of defect are likely to increase as you reduce one sort of testing to focus on another sort. (Although you may be able to, by analyzing the problems you find in test, find areas/methods of testing that aren't finding many problems, so you could potentially recommend certain areas as being less risky to reduce than others.)

So, I'd be very reluctant to commit to a percent of defects reduction. What I would commit to, after doing the analysis, is making changes to the development/test processes that would reduce the likelihood of classes of defects, and then give the powers that be a list of other classes of defects and what the cost of reducing them would be.

And then you need to keep doing the analysis, probably for several years, to see if you were correct, and you have reduced the number of defects of that class (relative to product usage), or if you need to make some sort of course correction.