Should a salary bonus be based on bugs in released code or other metrics?

Question

I've received a job offer recently with a significant portion of the pay based on yet to be determined "objective" performance metrics. We are supposed to sit down together after I've worked there awhile to develop and agree on the metrics that will be used. The only metric suggested by the employer so far is the number of bugs in released code.

Is this workable? Is it a fair metric or, as I fear, will it introduce dysfunction? Are there other metrics that could/should be used? Some of the issues that concern me are defining what a bug is, who finds the bug (does it only count when the customer finds it?), how do bugs found in old releases count, does a bug caused by a requirements gap count, what happens when other people whose pay is not effected by this metric, such as developers, help out with testing before a big release, what about the conflict between finding all the bugs and getting the product shipped, etc.

In other words, are there reasonable performance metrics that could be used?

Answer 1

Tying cash rewards to metrics like bug counts is always a very risky business, one that is almost certain to introduce dysfunction.

I never want people on my QA team to have to worry about their bonus when I just want them to be doing the right thing.

Imagine you are the tester working with a single developer (who happens to be your best friend) on a project that is planned to be released just before fiscal year end (when the bonus period ends). The developer has his bonus tied into getting this release done on time. You have a bonus tied into making sure the release doesn't have any bugs in it after it is released.

On the day before the release you discover a not-too-serious bug. What do you do? You could stay silent, let the code get released so that your friend earns his bonus, and hope that the bug isn't discovered so that you earn your bonus. You could write the bug report immediately causing your friend to either argue and try to get you over-ruled or lose his bonus.

This sort of thing happens all the time when bug reports are used as metrics and applied to bonuses.

You have pointed out several of the flaws in using bug count metrics. Here's one viewpoint: http://www.allthingsquality.com/2010/04/misuse-and-abuse-of-bug-counts.html

Release decisions are business decisions. Sometimes, it's more important to release code to meet an immediate business need - even if it contains bugs - and then fix the bugs later. You don't deserve to get penalized for that.

Unfortunately, there are no good systems for individual bonuses that must be tied to "metrics". The thinking is that "if we tie his bonus to a 'metric', it will motivate him correctly, and will be an objective and fair way to measure him. This thinking is simply wrong. Yet companies know this and still continue to force manager to create such bonus programs, and everyone suffers. Trust me - it's no fun for a manager either.

If you haven't already, you should read "Measuring and Managing Performance in Organizations" by Robert D. Austin. He talks about dysfunction caused by metrics/measurement systems.

I want my team to be empowered to simply "do the right thing" in all contexts, without having to worry that it will cost them bonus money and without worrying how to game the system so they maximize their bonus.

I strongly prefer that all bonuses are earned only based on company goals (sales and profit for example), and not on lower-level or individual goals. To me, this is the proper way to motivate individuals, and to reward them. Unfortunately, I've worked at many companies where I am forced to do otherwise.

You won't be able to tell how this will work out for you until after you have accepted the job offer, and have been working there for over a year.

Good luck!

Answer 2

In addition to what everyone else has said, metrics used for performance bonuses are a seriously bad idea in the software world.

Metrics are helpful as a descriptive tool, but that's about it in my view.

Here's some of the reasons why:

• every software project is different. That means that even if all the same things are measured, comparisons won't be valid. Project A might have released on time with few bugs, but that was because it was small, carefully scoped, and was a greenfields development, where Project B, which was late and horribly buggy was one of those death march projects with constantly shifting resources, no-one knew what was supposed to happen, and it relied heavily on existing code with a lot of known (and unknown) bugs. You simply can't compare bug counts, lateness, or any other metric - and if you're the tester working Project B, you will get shafted if metrics like this are used for your performance reviews.
• every metric can, and will, be gamed. I know places where bugs raised against developer code counted against them in performance reviews. Anything small was handled completely off the system to avoid hurting the developer - unless the tester didn't like the developer, in which case all bets were off.

I'd suggest you take a look at my answer to What is a good KPI for software QA? - it summarizes all my disagreements with metrics for performance reviews. I'd also recommend discussing the question of metrics with this potential employer and feeling out how they respond when you point out how little use bug counts or other such metrics are for evaluating how well you're doing your job. (In my experience leads and team members know who the stand-out people are and who the dead-weights are. If they feel that management trusts them to do their jobs without micromanaging, they'll let management know of any problems.)

Answer 3

Can you prevent code with known bugs from going out? If you can't actually control the thing being measured, it's a bad metric.

What if the right thing for the business is to release the code, with bugs, to get it out faster so users can start benefitting? Sometimes having some software is better than none (but not always!). If the metric could ever reasonably be opposed to doing the right thing for the business, it's a bad metric.

I would guess that it's your job to inform about product quality, not to control it. Unless you are allowed, empowered, and expected to halt the release until all the known bugs get fixed with no repercussions, this metric doesn't measure how well you do your job.

Take a look at the Wikipedia page on SMART goals (goals that are specific, measurable, attainable, relevant and time-bound). Many metrics suggested for testers fail to be attainable because they reflect factors that are not under the testers control.

It's very hard to suggest better metrics without knowing your role in-depth. Testing is a very context-sensitive role. Load and performance testing efforts often require a lot of effort for relatively few bugs - but those bugs are often really important bugs. Testing a smoothly run project with clear requirements and senior devs who write great code can mean few bugs filed and probably a very fast testing effort. Testing a buggy project with unclear requirements can mean a long, drawn-out testing effort with lots of bugs filed and still could have bugs escaping despite excellent test work.

Cem Kaner is a well-respected name, and he co-authored an article on Software Engineering Metrics that focuses on measuring testers. He suggests a method that involves qualitative analysis, considering each task that a tester is expected to perform and defining what it means for that task to be done well, then evaluating it. The article ends with this caveat:

We have not seen this type of evaluation fully implemented and don't know of anyone who has fully implemented it. A group of test managers has been developing this approach for their use, and many of them are now experimenting with it, to the extent that they can in their jobs.

Our intuition is that there are some challenging tradeoffs. The goal of this approach is not to micromanage the details of the tester's job, but to help the test manager and the tester understand which tasks the tester is doing well and which not. There are usually many ways to do a task well. If the scoring structure doesn't allow for this diversity, we predict a dysfunction-due-to-measurement result.

Answer 4

Metrics are useful, but they should be supplemented with subjective judgement. It's useful to have something measurable to use as the basis for management decisions such as promotions, raises, and bonuses. Of course any metric can be manipulated to defeat its purpose, so a competent manager will use their own subjective judgement too.

The bonus formula will change. Next year's bonus formula is unlikely to be the same as this year's bonus formula. I have never worked anywhere where the bonus formula remained the same for more than a year.

If you don't like the formula, you should suggest improvements. If you have an issue with the bonus formula, you should talk to your manager about it. If you and the manager are reasonable, you should be able to work something out that is more fair.

The right metrics depend on what you do. You can't measure what a tester does based solely on the released bug count. To find a better set of metrics, you need to think about what the job entails. For example, are you in charge of measuring the software's performance? Do you need to document a manual test suite? Do you need to write automated tests? Will you be involved in usability testing? Does it matter whether the bugs you report are clear and complete?

Answer 5

The problem with known performance metrics is that it changes behaviour. Not necessarily in the way you'd imagine.

When the plague was going on in Europe, rewards were set up for people catching/killing rats and presenting the bodies. Pay per carcass, nice and simple right? Until people worked out they could breed rats and make more money that way...

Similarly, if say, you're rewarded per bug, on a website, it's incredibly easy to be pedantic and to break stuff down into smaller bugs. Instead of "poorly written text on page X" it could be "word starts at wrong height", "typo in sentence", "spelling mistake", "poor choice of words" and so on. Eventually you're getting a much higher noise to signal ratio.

This doesn't mean small bugs are bad, just that the focus may change. Any reward system has to be carefully thought about. One method is to assign higher rewards to bigger priority bugs. I've seen this fail too - the 'average' priority/severity on bugs suddenly started increasing as people subjectively started rating them higher. Observation/confirmation bias and all that.

In addition, you may find developers starting to point out known bugs to testers - bugs that may not even be hit in a production environment. One could argue that a bug is a bug, but now you've got developers spending more time looking at bugs than fixing/writing new stuff.

In general, it makes a lot more sense for the rewards to be based on the whole company performing - if a sales target is hit, or whatever the metric might be, then everyone gets rewarded. This encourages some teamwork, unity and less "us vs them" mentalities.

Answer 6

(In addition to user246's post)

I am generally skeptic about defining quality based solely on quantitative metrics.

Although the number of bugs found in production does indicate what was missed by the testers, it misses a few things like:

• When you joined, did you have enough tools to efficiently test?

• What's the general quality of code? Let's say if the incoming code has a lot of bugs, and you spend most of your time with functional bugs, you might not be able to spend more time doing exploratory testing and finding potential issues that can crop up further down e.g. A bug introduced in a code in release v1 may not surface until release v5.

• During the year of evaluation, were you and/or your team actually building infrastructure for test automation? If so, you might not be able to spend enough time trying to catch all bugs but you ended up building an automation which can be of significant value to the company

  ... and a few more

Answer 7

Many of the current answers are possible. If the company is commited to using bonus KPIs that are related to bug count, it would be possible to create another layer to this concept.

Something like social scoring/votes for every bug entry a particular engineer makes. The best way to quantify the value of a bug entry in terms of what it actually is (severity let's say) and when this bug entry occurs (it comes at a very early stage rather than arbitrarily at the end of a development/project cycle or release).

Social scoring can pretty much follow the same model as Stack Exchange where different people can vote up or down any given bug entry. This allows more stakeholders to have a say and there are numbers (votes) that can be used for KPI records.