2

What are the best metrics for evaluating a software QA team's performance?

I am looking for metrics similar to the ones below:

1. Test Efficiency

 Test Efficiency = ((A-C)  /  (A+B))*100

 A = Defects found by test team

 B = Defects Found by customer

 C = Total number of Invalid defects found during Phase "A". 

2. Defect Leakage can be calculated using

 Defect Leakage = (A  /  (B-C))*100 

 Where

 A = Total Defects found in production.

 B = Total defect found by test team.

 C = Total number of Invalid defects found during Phase "B". 

3. Defect Rejection Ratio

 Defect Rejection Ratio = ( A  /  B) * 100

 A = Number of defects rejected by development team

 B = Total number of defects raised

This metric is important for measuring the efficiency of a tester: while recording defects, a tester should not waste development team’s time by creating invalid defects.

2

There is no best metric

I've yet to meet a tester who wants to create invalid defects, miss defects, or raise defects that are later rejected. If this is happening a lot, the problem is probably not with your test team, it's with something else in your process.

Looking at your suggestions in a little more detail:

Test Efficiency/Defect Leakage (proportion of defects missed by the test team/found by the customer or escaping to production)

  • Each project is different - One can have good resourcing, customers who actively work with the team and help to ensure that all communication between the customer liaison and the team is clear. Another can have the customer tell you what they want then say nothing until after delivery when you discover that what the customer said they want is not what they actually need. The test team is not responsible for this.
  • Application complexity matters - adding new features to a large, complex application will probably have many more bugs escape to the customer than a small, self-contained application.
  • Feature-complete/On Time/Thoroughly Tested - Pick any two - If your software is constrained by customer-imposed deadlines, features are usually not sacrificed. Testing is. If your testers are reporting that they've tested the highest priority areas based on your and their understanding of customer needs, but don't have the time for more, it's your responsibility as a test lead or project lead to ensure the the customer understands the risks they're taking with their aggressive timeline, not to beat your team with an unrealistic metric that will penalize them every time this happens.
  • Customers report defects to get new features free - This is way, way too common. I've seen it where a customer will sign off on a document that explicitly says something will not be done, then report the lack of that same thing as a defect. Very few tracking systems distinguish between these cases and actual defects, and many companies will implement these stealth feature requests as a bug fix to preserve customer goodwill.

Invalid Defects/Rejected Defects

  • What guarantee do you have that the specification or user stories are complete enough that the rejected defects aren't simply different views or expectations?
  • Who is raising and who is rejecting them? I've worked with programmers who would reject a defect in layout where a screen wasn't readable because the design spec didn't include the number of pixels to use for spacing.
  • Was the defect invalid, trivial, or simply not of sufficient priority to fix right now. This makes a big difference - and is not accounted for by naive metrics.
  • How fast are the requirements changing? I've worked in so many areas where the requirements (no matter how they're expressed) change several times a day. Maybe the tester has a more up-to-date version than the developer, or vice versa.

The best way to measure your team is to know what they're doing and what problems they're having. The kind of metrics you're talking about in this questions are the kinds of things that can be used to measure problems like cross-team communication issues, external communication issues, team and department challenges. Using them to measure team performance is inaccurate, in my opinion abusive, and will at best get your numbers gamed and at worst leave you with a non-functional, resentful team.

1

tester should not waste developer team’s time by creating invalid defect.

I think testers should also not waste developers team's time with trivial defects.

Gaming the system: The problem with metrics is that people will try to game the system.

Example: Personally I think I am pretty good in finding a lot of defects that actual users will not run into. Some examples might be inconsistent margins/paddings, incorrect use of language, errors when submitting gigabytes of data in forms and click flows resulting in errors which actual users would never follow. Reporting these issues could possible waste countless hours of development time.

Based on your statistics I might be great tester, since I find huge amounts of "valid" defects.

Metrics: Quoting some metrics from another question.

The best KPI's for testing are ones that you wouldn't expect.

  • Can the team ship with a boring level of predictability ?
  • Are customers happy ?
  • Is the product selling ?
  • Are there very few critical issues found in production ?
  • Are budgets and schedules being met ?

These co-incidenty are 'whole of team' KPI's which testing plays a key part of.

Also read this question and its answers: What is a good KPI for software QA?

0

It depends very much on what you want to reach. Some testing approach is possible to do completely without metrics. There should be reason for some measurement and metrics collection.

Question is too wide to be accepted, byt you can take some examples of the QA metrics here: http://www.getzephyr.com/resources/whitepapers/qa-metrics-value-testing-metrics-within-software-development

Not the answer you're looking for? Browse other questions tagged or ask your own question.