27

% of escaped defects means how many defects were found by customers post a release. It has been proposed as one of the metrics to evaluate testers' performance in one of the companies I know.

I wonder whether this an objective criterion and why it might be not.

For instance, it might be not a good metrics for a tester alone. Given two teams, one that develops high quality code, and another that product very buggy code, it seems to me that in the latter chances are more bugs will remain undiscovered, even if tested by the same tester.

  • 2
    Part of me feels that is just too broad and subjective. But when I look at "good subjective bad subective" I think this is exactly the kind of "good subjective" question. Is it good? "well that depends, and here's why" is exactly the kind of subjective question we want to encourage. – corsiKa Apr 9 '18 at 15:52
  • @corsiKa Thanks for clarification. If I knew it depends I would word this question differently, e.g., "when % of..." instead of "is % of...". Fortunately, posters knew better than me answered also "when" :-) – dzieciou Apr 9 '18 at 16:02
  • Well, I think even with "when is..." it's still quite subjective - but that subjectivity leads to good answers. The question is fine as it stands, but I can see how folks might think it to be too broad and/or subjective at first glance, that's all. – corsiKa Apr 9 '18 at 16:09
  • 1
    Can you believe that this decent question with several good informative answers, answers relevant to our core audience, almost got closed by Closing Mafia? Sometimes I feel like Closing Mafia is on a secret mission to kill this forum. :-( Of course answers are opinion-based. This is engineering, art of balancing opposites, there are no strict easy-to-follow rules. – Peter M. Apr 10 '18 at 3:02
  • 1
    To put it another way: Don't think first of the KPI and then ask if it is "good" - instead, spend time carefully detailing how you define what you mean by GOOD and then the KPIs will be obvious. – dwizum Apr 10 '18 at 13:55
36

In my personal opinion, I think this theory is flawed.

As you stated, there are so many variables that come into play:

  • % of escaped defects treat all bugs as equal, but they are not. Some bugs are more critical and some are merely cosmetic. If they really want to go with % of escaped defects, they had better find a way to properly and accurately weight a bug in terms of criticality.
  • This approach will encourage testers to focus on quantity instead of quality. Finding two cosmetic bugs gives a tester more merit than finding one blocker.
  • This approach will encourage testers to focus on user experience testing / UI testing as it is more likely for a customer to find a bug that is related to user experience / UI. Non-customer-facing testing may not get enough attention or effort.
  • For every testing project, its context is different. Say, a product is pushed into market without allocating much time for testing and the customers come back with lots of bugs and hence a high % escaped bug rate, is that testers' fault or is it a management issue?

I personally dislike this approach, a lot.

As Shadow has pointed out, I believe the following is a better but slower way to measure a tester's performance / contribution.

There are a few points that need to be addressed before proceeding any further:

  • We cannot catch all the bugs hence measure a tester's performance by SOLELY looking at how many bugs that have evaded into production code is not a feasible measurement.
  • For a given software project, overtime, it will become mature, e.g. the number of bugs discovered for this project will decrease over time.

So, how do we measure a tester's performance?

  • Establish an accurate record of software quality prior to a given tester joining a given project.
  • Review the criticality of the bugs this tester catches while maintaining an accurate record of software quality.
  • Overtime, after this tester joins, does software quality improve? It is hard to measure software quality quantitively, but there are a few indications that we can use as references, such as:
    • A decrease in critical bugs that evade capture, an increase in customers' happiness.
    • A decrease in bugs that appear intermittently.
    • An increase in bugs that are captured by test automation (developed by this tester if it is applicable).
    • Whether this tester can give clear and succinct bug descriptions to developers.
    • Whether this tester can prioritize testing areas and allocate time / effort properly.
    • Whether developers provide positive feedback on this tester.

In summary, there is no such thing as a boiled down number that can directly and accurately measure a tester's performance because a tester's job is complex and demands skills across multiple spectrum, which can be technical or non-technical. So, it will take a relatively long time and take multiple references into consideration before we can say if a tester is a good tester.

  • 2
    +1 absolutely YZ – Michael Durrant Apr 9 '18 at 2:23
  • 2
    Another flaw is that there is no assurance that customers will find all the bugs. So at any time there will be some unknown number of bugs which haven't been discovered. – MaxW Apr 9 '18 at 5:21
  • @MaxW, I agree with your first statement, regarding the second statement, I would say "regardless of what we do, there are always unknown number of bugs which have not been discovered" – Yu Zhang Apr 9 '18 at 5:36
  • What if this metric is turned into % of escaped Sev 1 (major bugs)? Will that address some of the issues you mentioned? – dzieciou Apr 9 '18 at 5:38
  • 2
    Trust me - I'm agreeing with you @YuZhang. But in the defence of those who would think to implement this strategy - what are the alternatives? If you could list some ideas that you think are more reasonable then this answer will be significantly more valuable. The problem is that getting the raw numbers that management so dearly crave is not as simple as they wish it to be... – Shadow Apr 9 '18 at 8:33
13

The short, simple answer: No

The slightly more nuanced answer: It can be, but you have to be very careful

The real explanation: Like most other candidates for tester performance KPIs, escaped defects is dependent on the activities of too many other people to be viable as a simple measure. Unlike something like words per minute or accuracy on a standard test for a typist, there are a ridiculous number of other factors that contribute to escaped defects, including but not limited to:

  • severity and impact of the defect(s) - one escapee that blocks customers completely is far worse than several minor and easily worked-around problems.
  • complexity of the application and module(s) - the more complex the software and enhancements, the more likely that something will be missed. Using percentage of escaped defects penalizes the testers working with the complex enhancement.
  • location of the defects - depending on the nature of the code, it can be much easier to find defects in some types of software than others. Testers should not be penalized because they are working on software that is more prone to subtle, hard-to-find defects.
  • testing time - it's a truism that no non-trivial software can be completely tested (and nobody would want to). If the testing team lacks a reasonable amount of time to test a feature due to late delivery and immovable deadlines, that feature will be released with more bugs. The testers are not at fault for this.
  • developers - some developers write more bugs than others. Testers don't control whose code they test, and someone testing work with a lot of bugs is likely to miss a higher percentage than someone testing work with relatively few issues, not least because of the higher likelihood of masking bugs.
  • design/requirements - even in the most agile of environments it's not uncommon for customers to report enhancement requests as bugs because the software does not do what they expected it to do (or does what they expected but not what they actually wanted). It's normal for customers to get exactly what they asked for, and what they watched be built and saw numerous demonstrations of, only to find when they try to use it that it doesn't actually meet their needs for reasons nobody was aware of during the development process. This is because software development (and testing) is a wicked problem. There is no "correct" answer and no best way to do anything.
  • staff turnover - when working with a complex application, a new developer (no matter how experienced) will make more mistakes. A tester who is not aware that there are one or more new developers working with the software has no idea that they will need to test more carefully. Even if they are aware, there is still a higher likelihood of an unusual scenario or unexpected interaction than with code written by developers who are familiar with the software.

I'd suggest you take a look at the answers to some of these questions for more information on the dangers of KPIs in general:

What is a good KPI for software QA?

How do I set specific (%) QA targets on reduction in bugs for the year?

How to assess QA employee performance?

Testers Performance Indicators / Metrics

  • Isn't the problem with masked bugs easily avoided by re-testing the software after bug fixes? – Dmitry Grigoryev Apr 9 '18 at 14:50
  • 1
    @DmitryGrigoryev - Sure, if you have unlimited time. The more cycles of fixing then finding something that was masked by the bugs fixed in the previous cycle, the more likely it is there will be more that you didn't get to find. – Kate Paulk Apr 9 '18 at 14:52
  • @DmitryGrigoryev there absolutely isn't a single standard threshold of acceptable bug count. To give the obvious examples, It's rather worse to have bugs in the flight control software for the next big airliner than it is to have bugs in the next facebook. – Leliel Apr 10 '18 at 4:57
  • @Leliel Can we compare apples with apples? A project manager which hires facebook testing team to test flight control software should indeed be sent to prison for negligence, even if the airliner doesn't crash. – Dmitry Grigoryev Apr 10 '18 at 7:32
5

We have several excellent answers, like Yu Zhang, Kate Paulk, o.m.

Let me add my 2 cents:

As consultants say "be careful what you measure - you WILL get it".

There are several discussions on workplace.SE about how using wrong metrics can skew cooperation between teams and make teamwork impossible.

Count of severe bugs in PROD can (and should) be just one of the metrics used to measure QA performance. Other are:

  • time delay on feedbacks (timely response),
  • time bugs spend testing (we need to pass them fast),
  • following the process and documenting it
  • customer satisfaction with the product

and many more, many not easily measurable. Quality is a team sport. If management will try to establish adversarial relationship between testers and any other teams (developers, analyst/engineer, sysadmins, devops, customer etc), they deserve the disaster which will inevitably follow.

"No bugs in prod" is possible and easy - if it takes infinite time to test everything (and retest after any bug was fixed) and we don't need customer's money so we never ship. Otherwise it is a business decision to deem release "good enough" and satisfactory for the paying customers.

Of course if we test more we would found more bugs. Will customers wait, or switch to a competitor with buggier code, but more desirable features?

That's why it is a business decision to stop testing and release. Of course testers want more time to test, if bugs in PROD is the only metrics they need to follow. So if business decides to override testers and release, why testers should be responsible?

Testers provide management with the information of quality status of current release, and management decides to release when risk of severe bugs in prod is lowered to acceptable level.

  • (1) if you measure tester performance by the time bugs spend testing, you will get bug reports like "SW crashes during test #42" instead of "function X doesn't validate output buffer's size". (2) A business decision to stop testing and release works much better when the manager tells the test team upfront what test coverage is required. If the manager changes his mind and decides to release now then the % of escaped defects for that project indeed doesn't measure testers performance. – Dmitry Grigoryev Apr 10 '18 at 7:49
  • @DmitryGrigoryev - I agree with you completely re (1) and (2), that's why I said "be careful what you measure - you WILL get it". Both (1) and (2) are standard management problems. I will go even farther in (2): manager should trust QA team to do the job they were hired for. – Peter M. Apr 10 '18 at 13:57
4

Whenever you measure performance, you are defining good or bad performance.

Sure, a professional wants to deliver professional work according to his or her standards, but a professional also wants approval from the management, bonuses, promotions, not-being-fired, etc.

No bugs in a production system is a nice goal to aspire to but also completely impossible. A good tester and especially a good test manager understands the "completely impossible" part and advises management on cost-effective testing levels and processes. What management really should want is an acceptable level of bugs and acceptable confidence about the quality of the the system for an acceptable total cost of development and testing.

  • Anything which counts number of bugs and not impact is questionable. Do you want to encourage the dev/test team to fix a dozen minor UI problems or a big hidden cause for mystery crashes?
  • Testing must always consider the cost of the test process, the impact of testing on the cost of the entire development process, and the guesstimated cost of bugs in a production system. The best KPI has to be a ratio.
  • Many really critical bugs are about fundamental misunderstandings of the specification. As the proverb goes, "build what I need, not what I want and certainly not what I told you to build." When such bugs come up, it is always "is this a bug fix or a specification change?" and the answer will be almost impossible to find. Building KPIs around such a thing encourages fingerpointing. (Unless the spec is part of the contract, then it becomes "working as designed is an euphemism for 'doesn't work'."
  • +1. No bugs in prod is possible and easy - if it takes infinite time to test everything (and retest after any bug was fixed) and we don't need customer's money so we never ship. Otherwise it is a business decision to deem release "good enough" and satisfactory for the paying customers. Of course if we test more we would found more bugs. Will customers wait, or switch to a competitor with buggier code, but more desirable features? That's why it is a business decision to stop testing and release. – Peter M. Apr 9 '18 at 17:18
  • @PeterMasiar The fact that it's impossible to find and fix all bugs doesn't change the fact that a tester which finds 90% of bugs is better than a tester which only finds 50%. And the bug impact is only known after the bug is found, so it doesn't seem fair to reward a tester who found a high-impact bug by chance. I'd rather hire a guy which can design a test suite with 100% code coverage than a guy which manages to find a couple of high-impact bugs without a systematic approach to it. – Dmitry Grigoryev Apr 10 '18 at 7:36
  • @DmitryGrigoryev, I'd hire the guy who understands that 100% statement coverage is an obsessive tick and that branch/condition coverage of the business logic is key. – o.m. Apr 10 '18 at 15:51
  • @o.m. First, 100% branch coverage results in 100% statement coverage. And second, if testing 100% of the code is "obsessive", then why not remove seemingly unneeded the code which is not to be tested? – Dmitry Grigoryev Apr 11 '18 at 8:43
  • @DmitryGrigoryev, the reverse isn't true, and that "100% coverage" (often without qualifier) shows up in plenty of documents by people who want to impress management with their diligence. Regarding untested code, that can be something like "if (variable == null) { throw new Exception(); }" -- I've written lines like that when there was no way to get a nullpointer at the time, but when I knew that the class was supposed to grow in the following weeks. Just in case a future call to a private method was faulty. – o.m. Apr 11 '18 at 15:38
3

It is not an objective crtieria as evidenced by the fact that we have to define it and there are different opinions.

Generally you would to avoid these kind of measure for measuring quality assurance personnel.

Reasons not to use this approach:

  • The numbers can be gamed and over time frequently are.
  • This measure quantity not quality
  • Only UI bugs are focused on
  • Application code quality often affects bug incident rates including subtle and hidden ones

Instead I would focus on things like

  • Value that testing has proven with bugs found by the tests
  • Value that the team feels the tester is adding in preventing bugs reaching production
  • Testing initiative taken in new areas to add value in testing
  • Knowledge and relationships with other team members and teams
  • Domain knowledge and ability to judge what sort of tests to use
  • 2
    "measure quantity not quality" doesn't make much sense to me, considering that KPIs have to be quantitative. – Dmitry Grigoryev Apr 9 '18 at 14:34
  • 1
    Number of Bugs found is certainly quantitive but is an example of measuring quantity (of bugs) not quality (how much they affect users and revenue). – Michael Durrant Apr 9 '18 at 22:01
  • One sensible approach would be to make testers hunt down all bugs (then forego fixing minor ones which have little impact), at which point the impact of a bug becomes irrelevant for tester evaluation. Otherwise, I don't see a problem with introducing bug impact as a weight factor in KPI calculation. – Dmitry Grigoryev Apr 10 '18 at 7:23
3

Capturing DEFECT LEAKAGE metrics is important but how it is captured; is more important

First of all, it should not be captured to assess the performance of testers only.

Why is it important to capture this metric?

  1. It is very important to calculate defect leakage in order to assess the overall quality of the product delivered by the team (Developers, testers and other stakeholders).
  2. There might be some defects leaked because of code merge mistakes
  3. Some requirements might have been captured incorrectly and were pointed out as defects during UAT

For more reasons, please refer to the Reasons for leaked defects

There will always be some defects reported by customer. I am in complete agreement. But, what if there are blunders from the project team? It is important to evaluate this metric and take steps in order to avoid; at least a few mistakes that have been made by the project team. In the example, quoted in the question; the RCA of client reported bugs should be conducted using 5 whys or fishbone so that all the major causes can be found. Even if the analysis is not done then how would an organization take steps to avoid such instances in future? How will the processes mature? And from my experience, I can assure you that a serious analysis would reflect on actual issues but on the other hand, no analysis or casual analysis would put everything on testers. Because, the easiest thing to say is that defects were leaked and testers were responsible for testing. So, they are to be blamed.

I want to quote here that in one of our projects the defect leakage was too high. So, it started with the analysis of client reported defects. And after the RCA was completed, it was concluded that the reason for high defect leakage was "understaffed testing team". And this finding was shared with all the projects so that they can learn from it and avoid such instances in future.

For example, in one of the companies that I worked, the permitted limit for defect leakage was 7% and defect leakage was calculated as follow:

Defect Leakage = (Num of client reported defects/100 PDs of estimated effort)X100

I can understand why the denominator was not "Total number of defects found".

This was an organization level standard which was gradually reduced to 4% in 2 years after putting this metric in place.

Even if the defect leakage was within the limits, we still use to go ahead and review all the client reported defects and do an RCA. I think that this was important otherwise our organization would have never been able to reduce the target from 7 to 4%.

Moreover,we also use to filter the client reported defects because:

  1. Most of the times, while logging a bug, users did not care to assign proper Priority and Serverity.

  2. And sometimes ended up reporting requirements and enhancements as defects. I mean the functionalities that users think as in scope but was never a part of requirements shared with us.

However, we had another metric which was "Defect Removal Efficiency" and it was calculated as follow:

Defect removal efficiency = [(Total Internal Defects)/(Total Internal+client reported defects)] X100

Where

Internal Defects=Doc review defects, code review defects, defects found during internal testing External Defects=Internal defects + client reported defects

2

% of escaped defects can be a fair KPI with a couple of restrictions:

  1. You define which classes of defects are included in the metric

For example, you could ask UI testers to verify that each control element has a desired effect on application behaviour, without testing all side-effects or combinations. Then, if your testers miss a button that's constantly unclickable, you can hold that defect against them, but not a button that becomes unclickable in a specific scenario. If you want to reliably catch that second case, you have to agree which scenarios should be tested as well.

  1. You give your testers enough time to catch all such defects

This is pretty simple. Either you tell your testers what kinds of defects they should be catching, and they tell you how much time they need, or you tell your testers when the deadline is, and they tell you what kinds of defects they will be able to cover.

These restrictions are necessary to make the metric attainable in the first place. If you impose a metric that your team cannot realistically attain, it will be either ignored or gamed.

As a final remark, remember that the metrics that work best are the metrics that your team buys into. Before such metrics are introduced, you should present them to the team and ask whether they make any sense.

1

It's a relevant metric, but it can skew performance indication if it's relied upon to be the main metric. Same defects can have multiple symptoms. QC which reports only on the symptoms would produce multiple reports of the same bugs which manifest immediately and overlook many of the bugs which manifest themselves rarely. Which would make the percentage of perceived escaped defects much smaller.

Low-probability high customer-impact (rather than high-customer impact) defects would be nothing but noise in such a metric, but they could spell out disaster for the business.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.