8

I often hear people talking about about Performance Metrics and Key Performance Indicators, but how to define quality engineering productivity exactly? What is it and how to measure? And what are some best practices to increase?

How can I evaluate the performance, productivity and improvement of a Quality Engineer?
Is it depending on time spend at the workplace, number of bugs found, certifications..etc. or something like that?
What are the possible reasons to say that the team need a considerable improvement?

  • 1
    Possible duplicate of sqa.stackexchange.com/questions/3381/what-is-a-good-kpi-for-software-qa/ – Kate Paulk Dec 8 '16 at 12:21
  • @KatePaulk : I already saw that question and answer. But my question is clear, I didn't get the exact picture of quality engineering productivity. – prinz Dec 9 '16 at 3:44
4

Is is quite hard to assess the productivity of a QA engineer.
It's easy to try and use things like 'time spend at the workplace', 'number of bugs found', 'certifications', etc.

The problem is that these are not really the measures you want. Base on a performance review on number of bugs logged and guess what happens? - Tons of bugs get logged, many of which are questionable.

This means you have to look more to:

  • What automated tests have been written
  • Bugs logged that represented significant revenue changes
  • Initiative taken in testing new areas
  • A proactive approach in ticket follow-up to make sure they don't go stale
  • Efforts to reduce technical debt
  • Work done that increases technical debt (this is a negative measure)
  • Work done to pair with developers
  • Quality of follow-up for difficult to reproduce bugs
  • Initiative taken on closing older bugs

It is very important that these factors should be discussed regularly (e.g. weekly) in 1:1's so that there will not be any significant surprises for the employee.

The value of certifications and formal training will vary among industries and company size. In some cases this will be be a critical measure for the company concerned, in other cases not. Think banking vs a startup for an example.

3

Managers like metrics because it gives them a (seemingly) objective way to measure the performance of their people and reward/correct them. There are better ways to reward or correct performance.

As an exercise in choosing metrics for measuring your team members, ask yourself this: "How can this metric be gamed?" If someone can gain financially from gaming a metric, it will be gamed.

You should also ask yourself why you want to perform such a task in the first place. If you are doing it to decide who needs to get cut from your team or receive an amazing bonus or raise, there are better ways.

James Bach recommends managers walk around in the developer pit and talk to people, listen to people, and get a sense of how the team is functioning on a regular basis. This will tell a manager more about his people's performance than any stack-ranked arbitrary metric.

A metric that measures team performance (developers and testers as a single team) is a list of bugs reported by customers, sorted by release. If the issue was found pre-production and not addressed, it still counts. This shows the delta between what the customer wanted and what we thought they wanted.

Since top management likes to measure the money, I have in the past used a story-based metric to show top management how valuable my team was. At raise/bonus time I would poll the bug repository for all the urgent issues opened by each tester. I would hand the list to each tester and have them guesstimate the value of the top 10 or so of each of their found issues had they not been found. Value can include such things as lost revenue, lost productivity, defecting customers, loss of company reputation, or anything else that can be quantified in a dollar amount. The dollar amounts were added up by each tester and always exceeded their annual salary. The lists of bug IDs, titles and dollar amounts were handed up the management chain as a demonstration of test team members' effectiveness.

There were many estimations and assumptions that went into each line item; I would help them come up with reasonable assumptions. These were available to anyone who wanted to challenge the assumptions. Nobody ever did.

1

Quality of a product should be the only measure to check engineers productivity.

Time spent at the workplace : A engineer can stay in office for 16 hours doing nothing. so that's not a good fit for productivity check.
Number of bugs: Number of bugs can vary application to application and totally based on the quality work done by developers. So this as well is not a good fit.
Certifications: If a engineer is good with studies only and fails in practical work, this case doesn't fits for him as well.

Delivered product's quality is the only measure which we can use to check performance of a Quality Engineer.

Not the answer you're looking for? Browse other questions tagged or ask your own question.