Defect Removal Efficiency confusion

Question

I have read several articles by Capers Jones, who states that:

Defect Detection Efficiency (DDE): percentage of defects found after release
Defect Removal Efficiency (DRE): percentage of fixed defects when compared to customer reported issues.

In the further text, he gives an example of DRE: when the development team found and fixed 90 defects, and the customer then finds 10, DRE=90%.

There is a note about the difference between DDE and DRE, stating that some companies choose not to fix all found defects before release, making DRE thus lower.

But how are those unfixed issues reflected? Let's say I found 100 defects and fix only 50. The customer, for some reason, found only 5. Following the stated process, DRE then should be 50/50+5=90%, even when I left 50 defects unfixed.

Is that truly correct? To me it seems counterintuitive that when I leave 50% of defects unfixed, in a situation when the customer found nothing, DRE would be 100%.

Answer 1

Bugs that are not fixed are comprised of many categories:

• they were unknown
• they were were not found
• they are viewed as features
• they are not well understood
• they are not important enough to fix
• they are not worth fixing based on the effort required
• they are not worth fixing based on the life span on the product
• they are not worth fixing because customers expect them and use workarounds

Using only the categories of "we found them" vs "customers found them" is too crude a measure for improving quality. Such a simplification will not address the complexities of the underlying issues and will also likely lead to human manipulation and misrepresentation to achieve other personal goals.

Answer 2

Stephen Kan's Metrics and Models in Software Quality Engineering Second Edition cites Capers Jones' Programming Productivity and states that Defect Removal Efficiency is not related to fixing the defects, but simply finding them. The equation presented is Removal Efficiency = (Defects found by removal operation) / (Defects present at removal operation) * 100% = (Defect found) / (Defect found + Defect not found or found later) * 100%.

Using a concrete example: If the development team finds 90 defects, fixes 50, presents 40 as known issues at release, and external stakeholders find 10, then Defect Removal Efficiency would be 90%.

It may appear that this makes Defect Removal Efficiency and Defect Detection Efficiency the same. However, I've usually seen the Defect Detection Efficiency with an additional part of the definition: it considers the defects that could and should have been found by an activity. If you consider the entire development process and have a robust enough process, then perhaps you should be able to find all of the defects. However, if you look at it activity-by-activity, some defects will be more likely to be found in different activities. For example, a code review or inspection may find an automated test that doesn't conform to the requirement which could hide a defect in the implementation, while dynamic analysis may find an injection vulnerability - you probably don't want to put finding injections on the code reviewer since that could be very hard, so if that defect escaped into production, you may find that your dynamic analysis is insufficient.

I would agree with Michael Durrant, though - just keeping track of who found a defect is insufficient. Especially when you consider iterative and incremental development models. There are plenty of good reasons why a defect could be found and not fixed, so getting into understanding things like why defects weren't found or why a found issue was not fixed is often more important and more interesting.