Capturing DEFECT LEAKAGE metrics is important but how it is captured; is more important
First of all, it should not be captured to assess the performance of testers only.
Why is it important to capture this metric?
- It is very important to calculate defect leakage in order to assess the overall quality of the product delivered by the team (Developers, testers and other stakeholders).
- There might be some defects leaked because of code merge mistakes
- Some requirements might have been captured incorrectly and were pointed out as defects during UAT
For more reasons, please refer to the Reasons for leaked defects
There will always be some defects reported by customer. I am in complete agreement. But, what if there are blunders from the project team?
It is important to evaluate this metric and take steps in order to avoid; at least a few mistakes that have been made by the project team.
In the example, quoted in the question; the RCA of client reported bugs should be conducted using 5 whys or fishbone so that all the major causes can be found. Even if the analysis is not done then how would an organization take steps to avoid such instances in future? How will the processes mature? And from my experience, I can assure you that a serious analysis would reflect on actual issues but on the other hand, no analysis or casual analysis would put everything on testers. Because, the easiest thing to say is that defects were leaked and testers were responsible for testing. So, they are to be blamed.
I want to quote here that in one of our projects the defect leakage was too high. So, it started with the analysis of client reported defects. And after the RCA was completed, it was concluded that the reason for high defect leakage was "understaffed testing team". And this finding was shared with all the projects so that they can learn from it and avoid such instances in future.
For example, in one of the companies that I worked, the permitted limit for defect leakage was 7% and defect leakage was calculated as follow:
Defect Leakage = (Num of client reported defects/100 PDs of estimated effort)X100
I can understand why the denominator was not "Total number of defects found".
This was an organization level standard which was gradually reduced to 4% in 2 years after putting this metric in place.
Even if the defect leakage was within the limits, we still use to go ahead and review all the client reported defects and do an RCA. I think that this was important otherwise our organization would have never been able to reduce the target from 7 to 4%.
Moreover,we also use to filter the client reported defects because:
Most of the times, while logging a bug, users did not care to assign proper Priority and Serverity.
And sometimes ended up reporting requirements and enhancements as defects. I mean the functionalities that users think as in scope but was never a part of requirements shared with us.
However, we had another metric which was "Defect Removal Efficiency" and it was calculated as follow:
Defect removal efficiency = [(Total Internal Defects)/(Total Internal+client reported defects)] X100
Where
Internal Defects=Doc review defects, code review defects, defects found during internal testing
External Defects=Internal defects + client reported defects