Testers Performance Indicators / Metrics

Question

What are the key performance indicators which you all are using in your company with respect to testers? As with any metric it can easily be skewed as testers and developers will often play to the metric, but are you all successfully utilizing any metrics in your projects? How can one best offset people working only towards these metrics? How do you balance these metrics between testers who automate, and those who do exploratory testing?

Answer 1

There are several things that you need to remember before starting this task:

1. You do not want to measure just for the sake of measurement; instead, you want to achieve a clearer view of what you have (or do) and how can you make it happen better.
2. You do not want to for your team nor their performance to be only driven by improving the measured results, for some cases, the measurement results may indicate improvements while the underlying performance that was allegedly being measured will be actually worsens. (Please see a live example documented in this question "Bug hunts and possible alternatives?").
3. Be sure of what you want to measure (team performance vs. individual's achievements vs. process compliance).

Keeping the above in mind, I would suggest dividing your task into the following categories:

1. 50-60% to Project's performance - this is a completely objective measure where the team member will be evaluated based on his/her performance during projects' work based on factors related to Productivity, Customer Satisfaction, Process Compliance and Personal Attributes.
2. 10-20% to Business Related Objectives - these objectives measure what can a team member do to increase the value of the business (e.g. building capacity for a certain skill or new tool).
3. 10-20% to Team Development Objectives - these objectives measure what can a team member do to make sure that a certain skill is properly shared and delivered to the rest of the team (e.g. Deliver training for the team on a skill or tool).
4. 5-15% to Personal Development - these objectives ensure that all team members personal improvement is also rewarded (e.g. acquiring a certain certification).

We have been using this model described briefly above for more than 5 years for now and we have achieved several benefits including achieving relatively low resources turnover and the fact that most of the team members feel that building an increasing on their skills falls in parallel with enriching their resumes.

However, if your question refers more to measuring or grading the outputs of your team members, then consider understanding the quality of the delivery of the team member outputs after, for example, peer review comments or team lead reviews to get a better understanding on the personal competency of each resource. However, I would remind you again not to get too much indulged in this area as it could lead to negative side effects that -in most cases- ruin the team spirits.

Answer 2

Scott,

Tester performance isn't something that you can measure easily. There are too many other dependencies.

Some examples:

• If you go with number of bugs opened/closed/whatever that's highly dependent on the quality of the material each person is testing. Someone who's got a load of fertilizer will open a lot of bugs quickly and close them quickly because there's a whole lot of really obvious and easily fixed bugs. Someone who's testing something that's mostly clean but has a hidden intermittent problem may not report many bugs at all, but the one they do report could be a show stopper. Someone who gets the best code in the system could legitimately not find anything.
• If you evaluate on lines of code for automators, you're ignoring the impact of data-driven testing where you can add 500 new tests without adding a single line of code (I've done this).
• If you look at number of bugs in projects Tester X worked on, then you also have to account for whether Tester X had sufficient time to test, how complex that project was compared to the projects and features your other testers worked on, and so forth.

Possibly the only valid measure is the tester equivalent of WTF per minute - WTF per test case document and WTF per minute when reviewing script code. Anything else is too dependent on factors outside your tester's control.

Answer 3

Before you attempt to implement a metrics program, I strongly urge you to read "Measuring and Managing Performance in Organizations" by Robert D. Austin.

You owe it to your team and to yourself to learn about the people-related aspects of measurement systems, and how "measurement dysfunction" occurs. And you may learn about the appropriate level of supervision needed to get your team to succeed the way you want them to (which may or may not involve metrics).

Answer 4

I have never found a metric that cannot be manipulated. I think it is good to measure things, e.g. statistics from your bug tracking system, but you must keep them in perspective. Although it happens all the time, plugging metrics into a formula is usually not a good way to manage people.

I also think there are mediocre statistics and bad statistics but no great statistics. For example, number of bugs opened/closed/reopened by a tester is mediocre; number of lines of code written by a developer is bad.

You seem to recognize that automation and exploratory testing are quite different. I think you may be able to measure, in a rough way, the relative effectiveness/cost of automation and exploratory testing. Comparing the job performance of individuals in those different roles is difficult.