- I wonder how can we apply the defect arrival rate to discover the progress in our testing projects from developers and testers side ?
- Is there a risk by applying such approach ?
There aren't any black and white rules in testing, everything is project and product dependent.
"Testing is context dependent"
The decision to use metrics in a project should consider many factors, a few would be:
- How critical is the product
- Do the organization and product follow a zero bug policy
- Can we release the product with known bugs
- Team maturity
- Release pace (How fast a product needs to be released)
Defect arrival rate:
Defect arrival date shows how many new defects were identified during the specified unit of time. Example, the number of new bugs per sprint, or per month etc.
In most companies more than defect arrival rate, number of open bugs is considered for making release decision. Meaning how many bugs under different severity and priority are there still there to be fixed.
If company has a zero high bugs and ignore low bug policy, then the product will be released only after all high bugs are fixed.
Advantages of Defect arrival curve:
- It helps in deciding whether development sprint is shorter than required, meaning do we need to have a longer sprint for ensuring quality development.
- Do we bring more tasks or user stories to the current sprint than we could deliver, this might be affecting the quality of sprint.
- Do we need split user stories again to smaller user stories, and need to have a more efficient effort estimation process.
- Do we need to restructure our test pyramid and ensure testing is pushed to the left of the pipeline. Meaning more unit, integration and API testing.
- Having identified more defects (even minor) in the last sprint before release could be an indicator that there may be defect masking in the system. Meaning minor bugs may be masking more critical ones, do we need to spend more time testing it rigorously.
- Is there more defect spillage from unit test > integration > backend > Front end. This indicates we need to improve our testing approaches in the left of the pipeline.
- All KPIs should be used for process change and release decision making, it should not be taken as a way to validate the efficiency of developer or tester.
- Open bugs are sometimes more efficient than defect arrival curve in agile teams but can use it to improve the agile process
This post in Linkedin by @Avinash Magdum looks interesting and could give more metrics relating to defect: