5

In one of my interviews, I was asked this question What to do when defect is found in production but not during the QA phase?

I answered it this way:

  1. Check for impact on the system. And if is of high severity and high priority, call for a hot fix.
  2. Do a retrospective meeting and find the root cause for the defect and also make sure that we have test scenario's and test cases ready for regression.
  3. Perform a quick regression on QA Environment for the hot fix and make sure that build is stable and no new defects introduced because of the defect.
  4. QA gives approval for moving build to PROD Environment and performs regression testing and signoff for the release.

I am not sure whether this is the approach that we need to follow. Can someone help me with best approaches need to follow in this kind of situations.

4

The process we follow where I work when a defect is reported in production might give you some perspective:

  1. Reproduce the problem. If possible, we try to reproduce on production. Failing that, we'll use the staging environment which is a mirror of production except when we have a new deployment staged there prior to release.
  2. Analyze the problem. First we determine whether the problem is actually a defect. In a complex application it could be a misunderstanding or thinking that intended behavior isn't what is supposed to happen.
  3. How old is it?. If it is a defect, we investigate to determine how long that defect has been around. We've found that after major deployments our customers are much more sensitive to the application and can pick up issues that have been around for years as a result.
  4. What is the impact?. It's rare for us to accidentally let a show-stopper defect through, but it does happen.
  5. Prioritize the problem. Prioritization generally takes into account how long it's been in the system, how badly it impacts affected customers, and what proportion of the customer base it hits.
  6. Determine the fix. Depending on the issue and how it's prioritized, this can be a quick hack to make things work with a more comprehensive fix scheduled for later, or it can be a proper correction to the problem.
  7. Schedule the fix. Again, what the schedule is depends on the nature of the problem. We have a very small team (4 developers and 1 tester supporting and enhancing multi-million LOC legacy applications and trying to build something more modern to replace them at the same time) so the demands on our time mean we can't always put a fix in quickly.
  8. Code, Test, & Deploy the fix.
  9. Root cause analysis. Depending on the scheduling, this could come before step 8, but it generally doesn't happen until we've at least prioritized the issue and communicated workarounds to the customers if any exist.

Depending on the severity of the issue, the whole process can be completed in under an hour, or it can be months before the defect is scheduled for a fix - or anything between those extremes.

Things we do and don't do

  • We do not blame. If I feel that I should have caught a problem before release I will say so, but nobody blames anyone else.
  • We do list contributing factors. These can and do include things like complexity of the code, lack of knowledge about how the application is used, unreasonable deadlines, and so forth. These are generally things that make it more likely that mistakes will happen rather than direct causes, because sometimes a problem gets through because of a perfect storm of contributing factors rather than a direct cause.
  • We try to add precautions to prevent this kind of problem recurring.
3

I think there are two sets of things that need to happen, possibly in parallel, possibly sequentially, possibly by different people, or possibly by the same people. At a guess, the interviewer would be more interested in B below than A, as A is going to be heavily dependent on the culture/company in question.

A) Deal with the immediate problem, which is to say, the defect found in production. Your steps 1, part of 2, 3, and 4 above cover that.

B) Root cause analysis, which may feed back into A. Specifically, you need to understand why the problem wasn't found in QA. If it's something simple, like missing a testcase, you add the testcase, verify that the testcase can find the problem, and then once you have the fix, rerun regression and deploy. (You also need to understand why the testcase was missing in the first place, and address that, along with any other test cases that might be missing for the same reason.) However, it's also possible that the reason the problem wasn't found was because it couldn't be found in the test environment, for reasons of scale, or configuration, or something else. Then you need to do a cost/benefit analysis: what would it take (time, money, skills, long-term maintenance etc) to update/replicate/whatever the test environment so that it could have found the problem, and what are the benefits? Are there classes of problems you aren't finding that you could find now? Are they ones that you need to be worried about, long term? For example, is the production environment growing much more quickly than the test environment is?

2
  1. 1st of all, Stop the blame game when defect is found in production
  2. Address the issue and try to fix it through config or how to apply workaround, in the mean time trigger R&D with required logs and traces.
  3. Let R&D investigate and support them on their queries.
  4. Analyse its impact with on production and provide a date for fix. At R&D lab
  5. Reproduce at Lab.
  6. After fix execute P1 cases
  7. Execute Regression Tests, preferably Automation
  8. Run Sanity on environment similar to Production
  9. Once fit for deployment release patch/sw
  10. Then Retrospect and make RCA with output as Action point-How to ensure similar issue does not appear again
  11. Ofcourse add the fuctional case in Automation suite.
  • Does R&D mean Research and Development? What's SW? Or RCA? – trashpanda Feb 26 at 14:37
  • @trashpanda - based on context I'm going to say R&D means Research & Development, SW means software, and RCA means Root Cause Analysis – Kate Paulk Feb 26 at 18:56
  • SW: In this context its Software or Patch or Software fix R&D: Research and Development, basically in your context here, it would be Dev and test teams RCA: Root Cause Analysis. I recommend below RCA types The 5 whys Flowcharting Fishbone diagrams – Vikas Sharma Feb 27 at 6:56
1

This answer focus on the phrase "and not during the QA phase".

This indicates to me that the main concern right now is blame.

There is a quite a natural tendency to want to blame someone. Like QA. However this goes nowhere fast in terms of fixing it and also improving quality for the long term. Instead it creates - or continues - a toxic environment of blame, finger-pointing and defensive behavior. Little to none of which improves quality.

If you sense that others are playing the blame game then you should make them stop immediately. Just kidding actually - you can rarely 'make people' do anything in any way other than command and control and not caring about the future - what you should do is get together as adults and follow the steps indicated by the other great answers here. Agree that you all want to improve the future and agree that any sort of blame game is counter-productive and should be called out openly (but nicely of course). The is commonly referred to as leadership and can be done by any one at any level.

0

Every software testing services company has faced this issue once a while when defect was found on production environment but not during QA phase. Below are points that should be kept in mind.

  1. Reproduce the problem on production and testing environment.
  2. If the problem is occurring only on production environment then it may be due to configuration issue.
  3. On the other side if it is occurring on QA environment then check the impact of that issue on the application.
  4. Investigate the issue to find out that how long that defect has been around.
  5. Determine the fix of ticket and list out the areas where can put more impact.
  6. If the issue is impacting more customers then apply the fix and deploy it on QA environment.
  7. Testing team should focus on testing all the regression scenarios around the fix.
  8. If the applied fix works fine then it should be deployed to production and post release sanity should be done so that it should not occur again.
  9. Do a retrospection meeting.

Things that should be kept in mind: 1. Work as a team on priority basis and try to learn from the mistake already happened in past. 2. Analyse the factors responsible for the occurrence of the issue like it may be due to poor development, tight deadlines for feature development testing so that it should not happen again.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.