5

Recently a custom metric was presented to our team during a meeting about software quality.

The metric was: number of found bugs divided by the sum of dev time (aggregated as team average and also per developer)

How are those numbers calculated?

We have a team of testers which do manual testing (based on new features and test scenarios). If a bug is discovered, it will be tracked in our internal bug tracker. Additionaly, the overall dev time is recorded as well.

After doing some research in the net I couldn't find anything about such a metric. I would say that it is related to defect density (bugs per loc) but I'm not sure.

Does this kind of metric make sense? Does it measure dev quality or software testing quality? And is there a common term for this metric?

Improve this question
2
  • Definitely not per developer--that's inviting bad behavior, both by management and by developers wishing to avoid unwanted scrutiny.
    – c32hedge
    Commented Apr 2, 2019 at 14:42
  • 1
    I agree - there was a lot of discussion after the meeting but nothing productive came up. I personally think that every bug found by our internal tests is 'better' than a bug which a customer reports. But this cannot be measured with such a metric.
    – ventiseis
    Commented Apr 2, 2019 at 15:10

6 Answers 6

Reset to default
11

No

It absolutely is not valuable to measure bug count per hour of dev time. It is especially bad to measure bug count per hour for individual devs.

Devs on more complex or difficult code will generally have more bugs/hour than devs working on cleaner, more straightforward code. Similarly, devs will produce more bugs/hour at the end of a long workday than they do at the beginning.

Unclear requirements will generate more bugs than clear, well-structured requirements.

If devs are penalized for excess bugs, some testers will start reporting problems informally in order to avoid being the cause of devs being penalized. Others could well decide to be petty and create a lot of small bugs to penalize someone they dislike.

In addition, there is a difference between a single application-breaking bug and a number of small trivial bugs - which is worse for the application? The trivial bugs would be counted as more damaging than the application-breaking bug with this kind of metric.

In my opinion the only way to use something like a bug count/hour is as an average used to forecast approximately how much padding you need to leave in the schedule for bug fixes: if team A typically produces 0.05 bugs/hour and 1 in 5 typically need to be fixed, then in 500 hours there will probably be 25 bugs generated, of which 5 are likely to need fixing before release. If the typical time per bug fix including testing is 5 hours, you'd include padding of around 25 hours to cover likely bug fixes.

(caveat: the numbers are not real - they're just for illustration of how the metric should be used)

Improve this answer
4
  • Your answer is built on assumption that devs will be penalized for bugs. Metrics are not always for the penalties. Metric might show correlation with different factors (for example they might show that some people require more training) and reducing the impact of those factors might improve overall performance. Hourly rate is taken because of flexible working schedule that might be used within particular project so that values are normalized for those who work 8 hours a day and for those who work 6 hours.
    – Alexey R.
    Commented Apr 2, 2019 at 16:52
  • I agree that our team members have different working hours per week and that is some kind of normalization required. But on the other hand, every team member is doing different things and therefore producing different kinds of bugs, which makes them hard to compare! No difference is made between fixing a misspelled word in a descriptive text and an annoying exception causing an appcrash. So I wondered if it would be helpful if there was some kind of bug classification...
    – ventiseis
    Commented Apr 2, 2019 at 20:49
  • All it engineers do different things. If I would pay for software development I would consider such justification as the attemt to put off the responsibility for the result. Whatever metric you imagine that could demonstrate the quality of the processes you will eventually come to subjective assumptions on some level of that metric computation which wont let you to tell if A works better than B because they do different things.
    – Alexey R.
    Commented Apr 2, 2019 at 22:25
  • 4
    Murphy's Law of metrics: If a metric can be misused against individuals, it will be.
    – c32hedge
    Commented Apr 3, 2019 at 14:20
4

As Kate says NO

The common term for kind of metric from those with experience is:

A bad metric

I'll go even further and say any kind of bug count is likely to be gamed
The industry has done thngs like this is the past, e.g. 'lines of code to measure productiviuty'. You can imagine where that leads! Although maybe how few lines would be good? NO! That would lead to single line code golf competitions! Not maintainable software!

I strongly recommend using different measures of quality.

Here are some good metrics:

  • Mean time to recover from a failure (make changes, run tests, release fix)
  • Number of users compared to yesterday / last week / last month / last year
  • Amount of revenue compared to past periods
  • Conversions - measures of customers reaching certain points on workflows
  • Device usage comparison to past periods

Also as Vertax implies, not all bugs are equal or take equal effort to fix.

I use the "1000 color issue bugs" is less important than 1 "I can't use the payement POST form for payments (the only revenue source)" bug, but perhaps more than "I have to click twice". It all depends on too many factors that are specific to your situation.

Improve this answer
2

No,

The reasons are many. First, bugs are not quantifiable. You can't add them together and then infer amount of time spent. Secondly, bugs that may be very simple to fix, (maybe even just one line changed) may take many hours of debugging to figure out what is wrong. As much as we would love our solutions to have adequate automated test coverage, and for bug reports to include enough information to drill down to the actual issue, it won't always be obvious, and because of that, every developer, might have a different amount of time just to understand what needs fixing. (If you discuss these during a sprint, and there is adequate documentation, that helps).

A better metric might be to look at, bugs as far as it relates to certain features/parts of the code logically. Bugs have a weird tendency to cluster, and sometimes that's a sign that you are rushing too quickly through development to done, when some bugs can be caught, and fixed, before introducing the bug into the backlog. Now, does that mean, you should ban bugs from being entered on features during a sprint? No, because there may always be hidden defects introduced, that are not even triggerable until some future point at which a change is made that exposes the faulty bit of code.

It is better to use them as a measure of code health, figuring out where to focus resources to fixing, than as a measure of Dev or Tester 'quality/goodness'.

Improve this answer
1

Yes

Metrics are usually tacked to manage them. If such the metric was introduced by business then they consider them valuable for their goals. For example if they pay for dev resources on per hour rate they might want to compare such rate with the metric measured on other team and decide to handover development to the better one.

They might want to know how much they "invest" in defects for number of different reasons. Except the one described above they might vary the dev team, excluding or including different persons (for example when someone leavs for vacation or a sick leave) and check how that value changes. Thus they could detect underperformers and make management decisions (for example provide training or combine the teams of the people whose synergy effect shows its maximum).

They take hourly values to normalize the measurement taken for the people working for different number of hours a day.

Improve this answer
5
  • 1
    One reason that was mentioned is that the amount of work for the manual testers is too much. If they would find fewer bugs, testing could be done faster and more effective. So the goal would be having fewer bugs before manual testing starts.
    – ventiseis
    Commented Apr 2, 2019 at 15:17
  • Why not. This is nice goal which can be managed using that metric
    – Alexey R.
    Commented Apr 2, 2019 at 17:01
  • 1
    I understand your reasoning and it makes sense from a management view. From the developers perspective it is not easy to accept such measurement because the tasks assigned to the developers differ in a lot of ways: from rolling out new features to maintaining the old code from others. Different tool support, different frameworks, even different workplace conditions - a lot of variables are influencing the day-to-day work which are all summed in the number hours of development time.
    – ventiseis
    Commented Apr 2, 2019 at 21:24
  • Software development is a matter of subjectivism. That does mean no obective metrics can be applied at all. That does not mean the business owners have to assess their employees in terms of "good boy" and "bad boy". Dont think of metrics as of the instrument of prosecution. It's an instrument of tuning processes.
    – Alexey R.
    Commented Apr 2, 2019 at 22:17
  • 3
    There are other alternatives to motivating and inspiring good work other than 'good/bad'. Which is not mentioned by the OP. This is using a strawman / binary argument.
    – Michael Durrant
    Commented Apr 3, 2019 at 10:32
1

I saw your reply "One reason that was mentioned is that the amount of work for the manual testers is too much. If they would find fewer bugs, testing could be done faster and more effective. So the goal would be having fewer bugs before manual testing starts." – ventiseis Apr 2 at 15:17

I think business, or who ever came up with this idea, has good intentions - to increase code quality. But there are better ways to do it - unit tests, integration tests, static analysis with data flow or code reviews.

Improve this answer
0

Maybe

If you use metrics as trends, which trigger you to do a root-cause analysis and take appropriate action any metric might be valuable.

A bug count per hour of dev time metric feels like a code complexity metric. If a feature takes more hours to build it probably is more complex and probably has more defects.

Does it have more value compared to defects per feature? What does the timespend really add? How do you plan to use it? What is the actual result you hope to get when you monitor and steer on this metric? Is it really a KPI? or is it just some data you could research once in a while.

Good metrics are hard, experiment and adapt.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.