Are there any metrics to determine if a "software defect" that happens in a Production environment is really a bug/defect - and then it's worth engaging 2nd level or "on-call" support?

In particular, I'm searching for literature about this.

Thanks in advance.

  • 1
    I'm not sure what exactly you're asking for. If there is a defect in production, then it lends to reason that it is a defect. This doesn't seem like it would be metric driven, but triage driven. The "software defect" could be due to environment issues, an actual product defect, or a user issue. Training your support engineers in root cause analysis is your best bet to make sure it ends up going into the correct bucket.
    – Sam Woods
    Commented Apr 14, 2014 at 17:50
  • I'm talking mostly of defects introduced by misconfigurations (applicative ones) of the Production system. We have teams that do "promotion" or "catalog" configuration in Production, and often they introduce a misconfiguration/"bug". We are the Production team (on call), and we are engaged by the 1st line support. Which may be a metric to engage us? I mean, how many "examples" of errors should the 1st line support report before we are called? One? Two? Is there literature on this?
    – friol
    Commented Apr 14, 2014 at 17:56
  • 1
    wont it all depend on how many customers are affected and how costly the 'bug' is? How about a 'how much is this issue costing us' metric? Commented Apr 14, 2014 at 18:10
  • @PhilKirkham: You can assume that the bug fix "cost" is constant (you have to call the "catalog" team and make them fix the issue). Real problem is this (that's what my question is about): when the first occurrency of the issue is discovered, you don't really know how many customers are affected (and you don't even know if the occurrency is due to "bad data" or if it's a real bug/misconfiguration). So, how many cases of the issue should we wait for before 2nd level teams are engaged?
    – friol
    Commented Apr 14, 2014 at 18:54

3 Answers 3


It depends.

I know this sounds frivolous, but there really is no standard for when a customer-reported issue should be escalated to a bug.

Some of the things it depends on are:

  • How well-trained the first-line support people are. Really good first-line support who know the application and can guide a customer/user to provide reproduction steps and track down most of the misconfiguration or misunderstanding issues mean almost everything they escalate is an actual software issue.
  • Whether the problem needs a programming fix or not. Sometimes in the case of corrupted data it's necessary to create a small utility application to clean things up. Here it generally doesn't matter whether the problem was caused by an actual bug or by a customer incorrectly terminating a long-running process: they just want their data corrected and a wise organization will do this for them (usually with a strong warning about terminating processes in spite of the big flashing warnings signs telling them not to do this).
  • Whether the customer support engineers can fix the problem without involving programming. Depending on your organization, support engineers can do things like create database scripts to correct a problem.
  • Whether the problem description matches something the test team has already reported but was not fixed for some reason. It's quite common for issues reported by the test team to be set aside for later if the perceived impact or risk is sufficiently low. Sometimes this assessment is incorrect because nobody knew there were customers using that peculiar combination of configuration flags (My personal record on the "nobody would ever do that" stakes is two days between releasing something with a problem "nobody would ever do" and a customer doing exactly that).
  • The impact of the problem on the customer. If the problem impacts financial data or data required for regulatory purposes, the tendency will be to escalate immediately due to the high risk/impact involved.
  • Which customer is reporting the problem. Support staff know who the biggest customers are and will be more likely to escalate problems reported by large customers, customers with a reputation for phoning the CEO to complain about the software, and so forth.
  • Thanks for your answer, but let's say our scenario is much more simple. Normally, if an issue is reported to our 1st level team, chances for it to be a real bug are high (no "restart your pc" or "network problem" issues, that is). Also, issues impact the final customer, so each case for a single problem is a loss of money.
    – friol
    Commented Apr 14, 2014 at 20:49
  • 2
    Friol, if it is a high chance that it will be a real bug, and the tier 1 support does not have the skill set to determine that (root cause analysis) by themselves, then it sounds like you should immediately escalate every occurence to tier 2. If you're trying to keep those escalations to a minimum, train your tier 1 staff to better be able to identify root cause.
    – Sam Woods
    Commented Apr 15, 2014 at 0:06

Our team has a set of guidelines to determine the severity of a production issue. The severity drives the level of communication/escalation. Of course, the guidelines are only guidelines and don't replace human judgement.

The factors in the guidelines: - How many customers are impacted? (few, many, most/all) - Severity of the problem? (single feature unavailable, hard outage, potential data damage)

I can't put the full set of guidelines here, but you get the gist. With each level of severity, there is a timeline associated with it (SLA). If the SLA isn't met, the incident bumps a level.

You might want to search "incident management classification" for more examples.


We separate "feedback" from "bug". We found than many so-called "bugs" reported by feedback are not bugs but code working as required (but not obvious for user). And of course you may have multiple feedbacks for same bug (and often you need to do some research to link it to existing bug). And quite often feedback reports Heisenbug which is hard to reproduce and/or study. Adding all feedbacks to bug database would bloat it. So we have two-level process: (1) evaluating feedbacks (with possible creating a bug from feedback, or deciding that it is caused by existing bug) (2) evaluating bugs and assigning priority and severity

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.