How can you measure the efficiency of the QA team? What KPIs would you set up?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
KPIs can be dangerous - it's very easy to measure the wrong thing or worse, reward the wrong thing. The general rule is that people will do more of the things that get rewarded.
Some of the things you should consider:
I hope this list helps you decide what you need to do.
The best KPI's for testing are ones that you wouldn't expect.
These co-incidenty are 'whole of team' KPI's which testing plays a key part of.
The simplest KPI would be that quickly after the team gets a release, the rate defects are found is high but then quickly tapers off as they find fewer and fewer. However, you will get what you measure, and games can be easily be played with these sorts of measures - i.e the team can work very hard for the first two days and then slack off completely and hey presto it looks like you're ready for release as they're just not finding bugs.
There are of course other KPIs that could be used, such as the proportion of defects found before/after release - but that of course is a very slow lagged measure.
KPIs are a great idea, but in real life, things are not as simple as a KPI unfortunately.
Many good examples here. Another that we use extensively in my company, Net Promoter. I do Net Promoter surveys for my quality team, and use that information to help us improve.
I send a brief survey to our stakeholders (developers, PM, leadership, ops, customer support, etc.): - How likely are you to recommend the QA team to another group? (0-10, with 10 being highest) - (text box) Please tell me why you answered the way you did.
You get the NP by subtracting detractors (0-6 score) from the promoters (9-10). Higher is better. This gives you a number to measure, and hopefully improve over time. More importantly, you see the verbatim comments from your stakeholders - for opportunities to improve, or great things to reinforce.
Any KPI you choose is going to fall afoul of the time-cost-quality triangle.
You can only optimize two of the three.
So let's say you pick bugs found in the field as your KPI. Then you obviously want LOADS of testing done, to make sure it's exhaustive. The problem? That costs time and money to achieve that quality.
What about low time as your KPI - all tests completed in 3 days? That's pretty awesome if your team does that right? Unfortunately, as you can probably predict, quality will likely suffer, OR you can throw lots of money at it (thousand testers?) but that'll shoot the costs up.
Do you count numbers of tests? I can break tests cases into as big (assert: the software works) or as little as you like ("page x field y accepts value 1 as input").
When it comes down to it, setting KPIs changes the goals - people focus on the KPIs RATHER than what they're meant to measure - how well the quality of the software is being assured. And even that is often subjective.
So then, how about we just go with subjectivity? Often sounds painful to developers / testers who like numbers, but frankly, there are stakeholders and customers, and at the end of the day, it's what they think of the software that matters. So ask them. Do annual surveys. Get them to rate aspects of the software. Is it getting worse or better? What's changing? Have there been more incidents coming in? Ask the customers what you can do better for their KPIs to improve.
The only things that should matter to a Test department are the number and severity of defects found after release. If we lived in a world perfect for testers, that would be all that is required.
However, it's also necessary for a wider business to know if their testing department is cost effective. After all, you could have 1,000 testers on a project and release something bug free but at what cost?
One of the few KPIs that I've found useful is Defects Per Hour (of testing). Whilst this is subjective and dependent on factors outside of Test, it is a handy guideline for how well things went.
At the very least, it's a starting point for a conversation between Test Lead and Management and when coupled with the Post Release Defects count, you can build up a reasonable picture of how well Test did.
Lost Post Release Defects + Low Defects Per Hour - Good initial coding
Lost Post Release Defects + High Defects Per Hour - Well tested product
High Post Release Defects + Low Defects Per Hour - Poorly tested product
High Post Release Defects + High Defects Per Hour - Product released too early
For QA, the best two KPIs I can think of are:
As well, estimation variance of time required to perform testing activity can be used.
Just remember - KPIs are indicators, not end-states. They indicate where you have strong and weak points and should be used to call your attention or as monitors against a baseline. They are not themselves a goal or a solution...but they do help you discover goals and solutions.