-1

My company wishes to use two primary KPIs to help improve quality.

  • Number of bugs (less is better)
  • Number of tests (more is better)

Will improving these figures actually improve quality?

Improve this question
3
  • 3
    How is this question different from other KPI questions we already have?
    – Niels van Reijmersdal
    Commented Apr 17, 2019 at 11:59
  • 1
    Thank @NielsvanReijmersdal . Updated title to refer to issue with specific KPIs
    – Michael Durrant
    Commented Apr 17, 2019 at 12:05
  • Nice seed question--it's unfortunate the downvoters (possibly?) didn't recognize it as such
    – c32hedge
    Commented Apr 23, 2019 at 20:09

4 Answers 4

Reset to default
6

No

Some general guidelines:

  • Any metric can and will be gamed. It is possible to have an application with thousands of passing unit tests that crashes on startup. It is also possible for bug count metrics to drive actual bug counts underground.
  • Using a metric as a guideline to indicate the team's current status will work. Using one to push the team to change/improve their ways will not.
  • The proposed metrics have favor in some organizations because they are easy to collect, not because they have value.
  • The way to improve quality in any product or organization is for everyone involved to be actively working to improve quality in their part of the organization. This includes ensuring that nobody feels the need to take short-cuts to meet deadlines, accepting that bugs will happen, and that when taken far enough the root cause is usually some variation of:
    • product complexity, where there what appears to be a simple problem causes unforeseen knock-on effects. This is usually caused by the developers/testers being insufficiently familiar with the application in test, although if the application is large enough or complex enough, it's possible that nobody can be sufficiently familiar with the application to prevent unforeseen circumstance bugs.
    • communication issues, where there is a mismatch between what customers expect, what developers/testers are told, and what actually gets built. It's entirely possible for nobody to be at fault in these cases - and just as possible for everyone to be at fault
    • schedule pressures, forcing developers/testers to take shortcuts to meet deadlines outside their control.
  • Metrics imposed without the agreement of a team will generate resentment and mistrust, which inevitably leads to gaming the metric.
  • Not all bugs are equal, and not all tests are equal. Using bug counts as KPIs effectively ranks the showstopper bug that prevents users logging on as equal to the typo on a page only ever seen by top level users and used maybe once a year. Using test counts as KPIs effectively ranks tests of the most commonly used functionality as equal to tests of "set it and forget it" functionality.

I have yet to see any good reason for using bug counts or test counts as a KPI.

Improve this answer
2
  • Great answer Kate, ty
    – Michael Durrant
    Commented Apr 17, 2019 at 15:40
  • +1, I always learn something new by reading Kate's answers, although it takes long time to read :)
    – Vishal Aggarwal
    Commented Apr 18, 2019 at 0:53
3

In a recent automation project, actually by reducing the test count drastically improved the overall test suite effectiveness multiple times. We almost reduced 80% test cases and designed test cases from scratch which is effectively & economically striking at the right points of the application.

We are getting far more effective test coverage with 100 test cases compared to earlier where we had 500 legacy tests which nobody bothered to review in long time.

Although this only could become possible when Business agreed to invest significant time to review test cases together with the QA team.

Improve this answer
0

The "number of bugs and numbers of tests" KPIs are often used.
This is unfortunate as they are actually just symptoms of underlying problems.
When companies try to improve those two figures without diving deeper into the why and underlying causes they will not improve quality for the long term. Or tomorrow.

In order to actually improve quality - which will lead to less bugs - companies should focus on a combination pf both technical and business stats that workers can actually affect by their daily activities and that the business want to see improving such as

  • Grading of code
  • Speed of test suite
  • Size of code classes
  • Reliability of test suite
  • Size of code methods
  • Gradient of testing pyramid
  • Amount of time to deploy a new release
  • Amount of time to fix a production issue
  • Customer visits compared to prior periods
  • Customer conversions compared to prior periods
Improve this answer
1
  • 1
    Re: Gradient of testing pyramid. Ideal shape of test stack depends on your situation and inverted or skewed test pyramid is not always an anti-pattern. See e.g., sqa.stackexchange.com/questions/37623/…
    – dzieciou
    Commented Apr 17, 2019 at 14:21
0

The one word answer would be no.

The reason behind is that bugs and tests can be much diverse in nature and only using numbers as metric won't suffice, and I don't know if any other metric exists which can give all the details of bugs and tests (e.g. details like root cause, effectiveness, etc).

If an organization wants to improve quality, I believe that they must follow some of the best practices(if not all), then they won't need to use such not so useful metrices.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.