What data-driven techniques exist for determining that enough testing has been done?

Question

I have seen some SQA groups use defect arrival or "discovery" rates in order to determine whether a piece of software is ready to be released. The idea is that the rate at which defects are uncovered will slow to an acceptable level.

What are these techniques? How is an "acceptable" defect arrival rate determined?

Edit: Daniel's answer gives a nice list of items. To elaborate on the question: what are the methodologies that allow setting of appropriate rates of decline / discovery of defects?

Edit 2: This link shows an example of the methodology. Has anyone used this or something like it?

Answer 1

There are lots of ways to do this; some of the data points I have used in the past are:

• Build breakage: is the automated build compilation/test failure rate going up or down?
• Assuming an Agile development model, are the user stories complete and have they been reviewed by QA + stakeholders
• Has the code freeze been effective? Are developers still checking in new features? This usually means there is more testing to be done
• Is there a lot of code merges going on at the end of the project? This tends to cause instability and could be a sign you need to do more targeted testing
• Automated testing confidence. The better your automated test coverage, the more confident you can be when the code changes
• Are there any blocker bugs left open?
• Is the find rate of critical bugs increasing or decreasing?

Answer 2

I don't think that there can be such a concept as an "acceptable" defect arrival rate, in the sense of a target number as you typically want to discover them as fast as possible. So, in english, you want to find as many bugs as you can.

The key way to read that metric is that you want a consistent arrival rate over time, i.e. you want the defect arrival trend to be fairly flat and never zero, say between 15-25 per day as opposed to 2 one day and 50 the next.

Once you have your bugs being discovered consistently, you want to look at the bug fix rate.

The simple answer is to average the number of bugs fixed per day, and divide the total number of bugs by the average. That is approximately how many days until you reach zero bugs. So, if you are fixing 5 bugs per day and you have 200 active bugs, the earliest that you will ship is in 40 working days time. If you want to ship sooner, you will need to stop adding features and focus on fixing more bugs.

The same information can be used in reverse to calculate a maximum allowable bug count. Say you only have 40 days until your desired ship date, and you are fixing 5 bugs per day as in the previous example. If you active bug count is over 200 today, you will probably miss your target. This number continuously decreases so in 2 weeks time, with 30 working days to go, your bug count should be at the 150 mark if you are going to hit your ship date.