4

Looking for a way to measure the efficiency of software testing I came upon the following formula

Testing Efficiency = (No. of defects Resolved / Total No. of Defects Submitted)* 100

from e.g. here.

But this does not look like a measure for testing efficiency, but more for resolving the found defects. Finding the defects is one thing, to fix them is not a testers responsibility. So how can it be a 'Testing efficiency', in what sense?

Also, what does the actual number say? 0% means none of the defects have been resolved, 100% means all of the defects have been resolved. What does a number of '80%' signify? Is it just a number reflecting the current state of a software release to see that 80% of the defects have been fixed? Can I see more from that number? Does the time-distribution tell me something with more insights?

Improve this question
1
  • 1
    Time and time again, I see questions on definitions. There's a trend for us to latch on to terms and less on goals. If the metric doesn't help you identify and reach your goals, bin it and find a new one. Simple. =)
    – corsiKa ♦
    Commented Feb 15, 2017 at 20:33

3 Answers 3

Reset to default
4

Test Efficiency is a measure of the relevance of the bugs being reported. A low efficiency would imply that the test team are reporting many bugs that aren't worth fixing. This is pretty limited, and simplistic.

I've had better success with a pie chart that shows the "resolution" for all of the resolved bugs. This shows the resolution categories like "fixed", "Cannot reproduce", "duplicate of another bug", "won't fix - business decision", etc... This pie chart allowed us to have a conversation about test efficiency. Turns out, we were opening lots of bugs that were duplicates - so we added a search capability to the bug tracking system. Also, lots of bugs were closed for business decision, but the PM didn't review. So, we added a PM review before closing bugs like that. Before this, we were only fixing 50% of the bugs reported. Afterwards, it was 80%.

It's hard to put a number on what is good, but the measure can allow you to ask "is this a problem?", and then give some insights on how to fix the problem if it exists.

Improve this answer
1
  • 1
    +1 I thought the metric the op describes look worthless, but if you take say resolved is fixed and compare it to other closing types it starts to make sense. It comes down to how good are reported defects.
    – Niels van Reijmersdal
    Commented Feb 15, 2017 at 20:00
3

Testing efficiency, among other metrics, is a merely a guideline. It does not tell the whole picture, it has to be put into context in order to make some sense.

It is more of an indication used by management when they make an assessment. E.g. when a senior manager asks a project lead, "Recently we have put some investment into fixing defects, how does it go?" A project lead may use testing efficiency to demonstrate how effective the money is spent.

One scenario I found interesting is:

  • A test manager I used to work for was very fond of measuring his team's efficiency with sprint story points. Once, we had a meeting, he showed that this team's efficiency dropped during last sprint as the story points we completed was 1 point less than previous records. He spent the entire 1 hour meeting discussing why our efficiency dropped.

Metrics like sprint story points and testing efficiency are more of guidelines, they are not strictly quantitive and should be combined with other metrics in order to get the whole picture.

Improve this answer
1

Like a lot of other metrics it doesn't mean a lot, especially on its own. Suppose you combine this with a "code coverage" metric and a "defects found in production" metric- you can get some meaning out of it.

The name might refer to something like "how good was the product tested".

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.