What is a good KPI for software QA?

Question

How can you measure the efficiency of the QA team? What KPIs would you set up?

Answer 1

Many good examples here. Another that we use extensively in my company, Net Promoter. I do Net Promoter surveys for my quality team, and use that information to help us improve.

I send a brief survey to our stakeholders (developers, PM, leadership, ops, customer support, etc.): - How likely are you to recommend the QA team to another group? (0-10, with 10 being highest) - (text box) Please tell me why you answered the way you did.

You get the NP by subtracting detractors (0-6 score) from the promoters (9-10). Higher is better. This gives you a number to measure, and hopefully improve over time. More importantly, you see the verbatim comments from your stakeholders - for opportunities to improve, or great things to reinforce.

Good luck

Answer 2

1. % of Rejected Defects - makes sure your QA really understands what he is doing.
2. Time to test a feature - how long does it take the QA to actually test a feature
3. % of Escaped defects - how many defects were found by customers post a release
4. Usability grade of your system
5. I'd also count how many enhancements requests your QA is opening - making sure they also provide new ideas for the system

Answer 3

For QA, the best two KPIs I can think of are:

• The number and severity of bugs escaping into the wild
• The cleanliness of the testing itself (i.e., testing the correct things as given in the requirements for the software/module in question)

As well, estimation variance of time required to perform testing activity can be used.

Just remember - KPIs are indicators, not end-states. They indicate where you have strong and weak points and should be used to call your attention or as monitors against a baseline. They are not themselves a goal or a solution...but they do help you discover goals and solutions.

T

Answer 4

The best KPI's for testing are ones that you wouldn't expect.

• Can the team ship with a boring level of predictability ?
• Are customers happy ?
• Is the product selling ?
• Are there very few critical issues found in production ?
• Are budgets and schedules being met ?

These co-incidenty are 'whole of team' KPI's which testing plays a key part of.

Answer 5

KPIs can be dangerous - it's very easy to measure the wrong thing or worse, reward the wrong thing. The general rule is that people will do more of the things that get rewarded.

Some of the things you should consider:

• If you use the number of bugs raised by the team, you will see bugs raised for things like misplaced pixels. Modifying this to use the number of bugs above a specified severity level will get your misplaced pixels classified to urgent.
• Using the number of bugs tested or cleared as a KPI tends to encourage testers to be less thorough and focus on checking that the bug as described is fixed without touching anything that might affect their statistics.
• Throughput of testing per person is equally dangerous: the temptation to test less thoroughly in order to meet the standard set for the indicator will arise.
• Any metric you use will be heavily dependent on context - a tester working with a developer who unit tests thoroughly, makes sure his integration is clean and checks that he's doing the right thing will not find many bugs in that developer's work. A project that's had its timeline cut to meet an external deadline will generate more bugs and have more bugs reach the customer than one that's been well-planned and executed without major time or resource stress.
• No one metric can capture the complexities of testing. Who is doing the better job, the tester who works through a complex new feature and finds a dozen severe bugs in a week, or the tester who works through a complex new feature and finds one catastrophic bug in that week? The answer is neither: both testers are finding information that needs to be surfaced, but the information they find differs because they're working in different areas with different developers.
• Almost no-one has the resources for multiple testers to perform the same tests against the same software. That means that no measurement can capture performance because there is nothing that can be directly compared. There are too many variables involved, including how much pressure the tester is under to finish testing so the software can be released (no, this isn't ideal, but it's how it works in a lot of places).
• No matter what measure you use, it's best to use it as an informational tool: this team averages this much testing for this many bugs so if we're going to use them we need to schedule this much time. Using a metric to rank your testers will give you results that are... less than you desired.

I hope this list helps you decide what you need to do.

Answer 6

The simplest KPI would be that quickly after the team gets a release, the rate defects are found is high but then quickly tapers off as they find fewer and fewer. However, you will get what you measure, and games can be easily be played with these sorts of measures - i.e the team can work very hard for the first two days and then slack off completely and hey presto it looks like you're ready for release as they're just not finding bugs.

There are of course other KPIs that could be used, such as the proportion of defects found before/after release - but that of course is a very slow lagged measure.

KPIs are a great idea, but in real life, things are not as simple as a KPI unfortunately.