How can you measure the efficiency of the QA team?

What KPIs would you set up?

  • 5
    I'm assuming KPI = Key Performance Indicators? My question is why would you want to measure efficiency? Wouldn't effectiveness be a more relevant measure? Commented Jul 3, 2012 at 2:39
  • 5
    The only valid measurement of quality is WTF/minut. The other question is 'How to use it?' :) Commented Feb 20, 2013 at 13:43

8 Answers 8


KPIs can be dangerous - it's very easy to measure the wrong thing or worse, reward the wrong thing. The general rule is that people will do more of the things that get rewarded.

Some of the things you should consider:

  • If you use the number of bugs raised by the team, you will see bugs raised for things like misplaced pixels. Modifying this to use the number of bugs above a specified severity level will get your misplaced pixels classified to urgent.
  • Using the number of bugs tested or cleared as a KPI tends to encourage testers to be less thorough and focus on checking that the bug as described is fixed without touching anything that might affect their statistics.
  • Throughput of testing per person is equally dangerous: the temptation to test less thoroughly in order to meet the standard set for the indicator will arise.
  • Any metric you use will be heavily dependent on context - a tester working with a developer who unit tests thoroughly, makes sure his integration is clean and checks that he's doing the right thing will not find many bugs in that developer's work. A project that's had its timeline cut to meet an external deadline will generate more bugs and have more bugs reach the customer than one that's been well-planned and executed without major time or resource stress.
  • No one metric can capture the complexities of testing. Who is doing the better job, the tester who works through a complex new feature and finds a dozen severe bugs in a week, or the tester who works through a complex new feature and finds one catastrophic bug in that week? The answer is neither: both testers are finding information that needs to be surfaced, but the information they find differs because they're working in different areas with different developers.
  • Almost no-one has the resources for multiple testers to perform the same tests against the same software. That means that no measurement can capture performance because there is nothing that can be directly compared. There are too many variables involved, including how much pressure the tester is under to finish testing so the software can be released (no, this isn't ideal, but it's how it works in a lot of places).
  • No matter what measure you use, it's best to use it as an informational tool: this team averages this much testing for this many bugs so if we're going to use them we need to schedule this much time. Using a metric to rank your testers will give you results that are... less than you desired.

I hope this list helps you decide what you need to do.

  • 1
    Some of the ideas behind what you say here have much wider application. I wish that more people who thought about metrics thought so clearly. +1
    – Kazark
    Commented Feb 21, 2013 at 21:02
  • This is an old comment, but this brought some insight. Im at a new company and they are really trying to figure out how to measure the performance of QA. I agree with your comment however do you think there "is" really a good way to measure it?
    – Mercfh
    Commented Feb 18, 2016 at 5:11
  • 1
    @Mercfh - not using metrics. The only real way to measure QA performance is the old-fashioned method of being aware of what individual team members are doing, what problems they encounter, and how they handle those problems. Testing is a "soft" skill (i.e. no or few objectively best ways to do things) that's heavily dependent on context: those don't take well to measurement.
    – Kate Paulk
    Commented Feb 19, 2016 at 12:40
  • There's a way, but it's not one anyone likes, which is how many defects are found in the field, in the tested code? Unfortunately, this may not be something that you can know for several years, depending on take-up rate and how long the code is in use, and changes between the test/development assumptions and how the product is eventually being used will skew things as well. Commented Sep 7, 2016 at 22:39
  • 1
    @KevinMcKenzie - that metric also requires a degree of filtering, because a lot of what gets reported from the field as defects - or treated as defects - is actually small customer requests or a mismatch between team beliefs and customer beliefs. Neither of which should be considered when trying to measure team performance.
    – Kate Paulk
    Commented Sep 8, 2016 at 11:26

The best KPI's for testing are ones that you wouldn't expect.

  • Can the team ship with a boring level of predictability?
  • Are customers happy?
  • Is the product selling?
  • Are there very few critical issues found in production?
  • Are budgets and schedules being met?

These coincidentally are 'whole of team' KPI's which testing plays a key part of.

  • 3
    +1 for "whole of team" KPIs - makes little sense to measure testing separately from what we're meant to be helping to deliver.
    – testerab
    Commented Jul 6, 2012 at 19:36
  • So it isn't possible to have "good testing" without having "good everything else"? Commented Feb 20, 2013 at 14:29
  • @joe ... possibly ... but the "best tested" product, that is a commercial failure, is still a failure, and by derivation, the testing has been complicit in that failure as well. Commented Feb 21, 2013 at 21:30
  • 1
    @Bruce - I mostly agree. Which is why I always try to hitch my "testing wagon" to a winning team. You can't test success into a product. And it's hard to feel good about a great testing job if the rest of the company sucks. Still, I can't necessarily call your list "KPI's for testing". More like "KPI's for products or companies". Commented Feb 21, 2013 at 21:36
  • 1
    @joe ... I agree with that. I suppose that was my point. In my mind/opinion testing is part of the team, and succeeds or fails if the team succeeds or fails, and shouldn't be measured independently, which, technically, doesn't answer the question. Commented Feb 21, 2013 at 22:10
  1. % of Rejected Defects - makes sure your QA really understands what he is doing.
  2. Time to test a feature - how long does it take the QA to actually test a feature
  3. % of Escaped defects - how many defects were found by customers post a release
  4. Usability grade of your system
  5. I'd also count how many enhancements requests your QA is opening - making sure they also provide new ideas for the system
  • I like this, but I believe some would argue QA's responsibility is not suggesting new features. The QA is unlikely to be immersed in the business knowledge of the application (to the extent an end user is) and is probably going to suggest things that make testing more convenient rather than making using the tool itself more convenient.
    – corsiKa
    Commented May 22, 2013 at 15:19

For QA, the best two KPIs I can think of are:

  • The number and severity of bugs escaping into the wild
  • The cleanliness of the testing itself (i.e., testing the correct things as given in the requirements for the software/module in question)

As well, estimation variance of time required to perform testing activity can be used.

Just remember - KPIs are indicators, not end-states. They indicate where you have strong and weak points and should be used to call your attention or as monitors against a baseline. They are not themselves a goal or a solution...but they do help you discover goals and solutions.

  • 2
    How does one measure "cleanliness"? Commented Feb 20, 2013 at 13:20

The simplest KPI would be that quickly after the team gets a release, the rate defects are found is high but then quickly tapers off as they find fewer and fewer. However, you will get what you measure, and games can be easily be played with these sorts of measures - i.e the team can work very hard for the first two days and then slack off completely and hey presto it looks like you're ready for release as they're just not finding bugs.

There are of course other KPIs that could be used, such as the proportion of defects found before/after release - but that, of course, is a very slow lagged measure.

KPIs are a great idea, but in real life, things are not as simple as a KPI unfortunately.

  • 2
    KPIs are a great idea in testing? Really? Perhaps you should qualify your statement. Commented Jul 3, 2012 at 18:45
  • 4
    In principle / theory KPIs are a good idea, but in practice they are frequently dangerous because they modify behaviour in an inappropriate direction. In the same sense that "In theory there is no difference between theory and practice - but in practice there is."
    – Sean
    Commented Jul 3, 2012 at 22:07

Any KPI you choose is going to fall afoul of the time-cost-quality triangle.

Can't have em all

You can only optimize two of the three.

So let's say you pick bugs found in the field as your KPI. Then you obviously want LOADS of testing done, to make sure it's exhaustive. The problem? That costs time and money to achieve that quality.

What about low time as your KPI - all tests completed in 3 days? That's pretty awesome if your team does that right? Unfortunately, as you can probably predict, the quality will likely suffer, OR you can throw lots of money at it (thousand testers?) but that'll shoot the costs up.

Do you count numbers of tests? I can break tests cases into as big (assert: the software works) or as little as you like ("page x field y accepts value 1 as input").

When it comes down to it, setting KPIs changes the goals - people focus on the KPIs RATHER than what they're meant to measure - how well the quality of the software is being assured. And even that is often subjective.

So then, how about we just go with subjectivity? Often sounds painful to developers/testers who like numbers, but frankly, there are stakeholders and customers, and at the end of the day, it's what they think of the software that matters. So ask them. Do annual surveys. Get them to rate aspects of the software. Is it getting worse or better? What's changing? Have there been more incidents coming in? Ask the customers what you can do better for their KPIs to improve.

  • There are ways to keep quality high while shortening testing time. For instance, shorten tedious tasks like regression testing, verifying environment health, or setting up test data via automation. Sure, there will be some investment in the beginning but in the end you will gain more time for doing additional tests. This way you are able to optimize three elements in the triangle: cost (less testers doing tedious tasks), time (shorten regression time) and quality (new bugs can be discovered in the gained time).
    – dzieciou
    Commented Jan 19, 2014 at 7:58

Many good examples here. Another that we use extensively in my company, Net Promoter. I do Net Promoter surveys for my quality team, and use that information to help us improve.

I send a brief survey to our stakeholders (developers, PM, leadership, ops, customer support, etc.): - How likely are you to recommend the QA team to another group? (0-10, with 10 being highest) - (text box) Please tell me why you answered the way you did.

You get the NP by subtracting detractors (0-6 score) from the promoters (9-10). Higher is better. This gives you a number to measure, and hopefully improve over time. More importantly, you see the verbatim comments from your stakeholders - for opportunities to improve, or great things to reinforce.

Good luck

  • And you use those as KPIs to measure the efficiency of your QA Team? Commented May 24, 2013 at 19:19

The only things that should matter to a Test department are the number and severity of defects found after release. If we lived in a world perfect for testers, that would be all that is required.

However, it's also necessary for a wider business to know if their testing department is cost effective. After all, you could have 1,000 testers on a project and release something bug free but at what cost?

One of the few KPIs that I've found useful is Defects Per Hour (of testing). Whilst this is subjective and dependent on factors outside of Test, it is a handy guideline for how well things went.

At the very least, it's a starting point for a conversation between Test Lead and Management and when coupled with the Post Release Defects count, you can build up a reasonable picture of how well Test did.

For example

Lost Post Release Defects + Low Defects Per Hour - Good initial coding

Lost Post Release Defects + High Defects Per Hour - Well tested product

High Post Release Defects + Low Defects Per Hour - Poorly tested product

High Post Release Defects + High Defects Per Hour - Product released too early

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.