What to do when defect is found in production but not during the QA phase?

Question

In one of my interviews, I was asked this question What to do when the defect is found in production but not during the QA phase?

I answered it this way:

1. Check for impact on the system. And if is of high severity and high priority, call for a hotfix.
2. Do a retrospective meeting and find the root cause for the defect and also make sure that we have test scenario's and test cases ready for regression.
3. Perform a quick regression on QA Environment for the hotfix and make sure that build is stable and no new defects introduced because of the defect.
4. QA gives approval for moving build to PROD Environment and performs regression testing and signoff for the release.

I am not sure whether this is the approach that we need to follow. Can someone help me with the best approaches need to follow in this kind of situations?

Answer 1

The process we follow where I work when a defect is reported in production might give you some perspective:

1. Reproduce the problem. If possible, we try to reproduce on production. Failing that, we'll use the staging environment which is a mirror of production except when we have a new deployment staged there prior to release.
2. Analyze the problem. First we determine whether the problem is actually a defect. In a complex application it could be a misunderstanding or thinking that intended behavior isn't what is supposed to happen.
3. How old is it?. If it is a defect, we investigate to determine how long that defect has been around. We've found that after major deployments our customers are much more sensitive to the application and can pick up issues that have been around for years as a result.
4. What is the impact?. It's rare for us to accidentally let a show-stopper defect through, but it does happen.
5. Prioritize the problem. Prioritization generally takes into account how long it's been in the system, how badly it impacts affected customers, and what proportion of the customer base it hits.
6. Determine the fix. Depending on the issue and how it's prioritized, this can be a quick hack to make things work with a more comprehensive fix scheduled for later, or it can be a proper correction to the problem.
7. Schedule the fix. Again, what the schedule is depends on the nature of the problem. We have a very small team (4 developers and 1 tester supporting and enhancing multi-million LOC legacy applications and trying to build something more modern to replace them at the same time) so the demands on our time mean we can't always put a fix in quickly.
8. Code, Test, & Deploy the fix.
9. Root cause analysis. Depending on the scheduling, this could come before step 8, but it generally doesn't happen until we've at least prioritized the issue and communicated workarounds to the customers if any exist.

Depending on the severity of the issue, the whole process can be completed in under an hour, or it can be months before the defect is scheduled for a fix - or anything between those extremes.

Things we do and don't do

• We do not blame. If I feel that I should have caught a problem before release I will say so, but nobody blames anyone else.
• We do list contributing factors. These can and do include things like complexity of the code, lack of knowledge about how the application is used, unreasonable deadlines, and so forth. These are generally things that make it more likely that mistakes will happen rather than direct causes, because sometimes a problem gets through because of a perfect storm of contributing factors rather than a direct cause.
• We try to add precautions to prevent this kind of problem recurring.

Answer 2

I think there are two sets of things that need to happen, possibly in parallel, possibly sequentially, possibly by different people, or possibly by the same people. At a guess, the interviewer would be more interested in B below than A, as A is going to be heavily dependent on the culture/company in question.

A) Deal with the immediate problem, which is to say, the defect found in production. Your steps 1, part of 2, 3, and 4 above cover that.

B) Root cause analysis, which may feed back into A. Specifically, you need to understand why the problem wasn't found in QA. If it's something simple, like missing a testcase, you add the testcase, verify that the testcase can find the problem, and then once you have the fix, rerun regression and deploy. (You also need to understand why the testcase was missing in the first place, and address that, along with any other test cases that might be missing for the same reason.) However, it's also possible that the reason the problem wasn't found was because it couldn't be found in the test environment, for reasons of scale, or configuration, or something else. Then you need to do a cost/benefit analysis: what would it take (time, money, skills, long-term maintenance etc) to update/replicate/whatever the test environment so that it could have found the problem, and what are the benefits? Are there classes of problems you aren't finding that you could find now? Are they ones that you need to be worried about, long term? For example, is the production environment growing much more quickly than the test environment is?

Answer 3

1. 1st of all, Stop the blame game when defect is found in production
2. Address the issue and try to fix it through config or how to apply workaround, in the mean time trigger R&D with required logs and traces.
3. Let R&D investigate and support them on their queries.
4. Analyse its impact with on production and provide a date for fix. At R&D lab
5. Reproduce at Lab.
6. After fix execute P1 cases
7. Execute Regression Tests, preferably Automation
8. Run Sanity on environment similar to Production
9. Once fit for deployment release patch/sw
10. Then Retrospect and make RCA with output as Action point-How to ensure similar issue does not appear again
11. Ofcourse add the fuctional case in Automation suite.

Answer 4

1. First understand that there is no software in this world with zero defects when comes to production.
2. Stop worrying about yourself or team that bug went to the production.
3. Find what kind of impact did the bug done in the production.
4. Do a clear root cause analysis.
5. If the bug impacted huge, check whether you need to role back the release to previous one.
6. check priority and severity of the bug.
7. Have a discussion with Business analysis, development team about the bug.
8. Get a clear fix from the development team, run it in your UAT and then release it into the production.
9. Check whether any new impact of the bug has come.
10. Record the process, fixes and make it as document.
11. Inform to the client and ask apologizes.

Answer 5

This answer focus on the phrase "and not during the QA phase".

This indicates to me that the main concern right now is blame.

There is a quite a natural tendency to want to blame someone. Like QA. However this goes nowhere fast in terms of fixing it and also improving quality for the long term. Instead it creates - or continues - a toxic environment of blame, finger-pointing and defensive behavior. Little to none of which improves quality.

If you sense that others are playing the blame game then you should make them stop immediately. Just kidding actually - you can rarely 'make people' do anything in any way other than command and control and not caring about the future - what you should do is get together as adults and follow the steps indicated by the other great answers here. Agree that you all want to improve the future and agree that any sort of blame game is counter-productive and should be called out openly (but nicely of course). The is commonly referred to as leadership and can be done by any one at any level.

Answer 6

I am surprised to read that everyone is talking about tester/QA missing the test case. Most of the time, its the requirement issue. I have done the research and even the data backs it up, around 30% of the time, it's either a missing requirement or ambiguous requirement and many times, developers don't even ask questions before implementing it.

Consider QA as the police like you can't have one cop for one civilian; it's the same analogy. QA is there to keep a check and ensure that everyone in the org (from BA to developer) must have done their job, right. As far as the blame game is concerned, it depends upon an organization's culture. If it's an immature culture then it will bound to happen.

Answer 7

Every software testing services company has faced this issue once a while when defect was found on production environment but not during QA phase. Below are points that should be kept in mind.

1. Reproduce the problem on production and testing environment.
2. If the problem is occurring only on production environment then it may be due to configuration issue.
3. On the other side if it is occurring on QA environment then check the impact of that issue on the application.
4. Investigate the issue to find out that how long that defect has been around.
5. Determine the fix of ticket and list out the areas where can put more impact.
6. If the issue is impacting more customers then apply the fix and deploy it on QA environment.
7. Testing team should focus on testing all the regression scenarios around the fix.
8. If the applied fix works fine then it should be deployed to production and post release sanity should be done so that it should not occur again.
9. Do a retrospection meeting.

Things that should be kept in mind: 1. Work as a team on priority basis and try to learn from the mistake already happened in past. 2. Analyse the factors responsible for the occurrence of the issue like it may be due to poor development, tight deadlines for feature development testing so that it should not happen again.

Answer 8

Bugs are found in prod all the time. Especially when the time to market becomes a race and the focus of testing lies on the little time given rather than the application. So as testers we need to leave our ego on the door and do our best with the tools and circumstances we're given.

First, understand that testers don't own bugs, so don't take it personally. Things tend to seem obvious quite often after the fact so don't make it about you, I can't stress this enough, control your ego. A lot of people missed the bug and a lot of factors contributed to that. The place to think about that is after everything is stable and you do some sort of lessons learned from the release.

Second, get information on the bug. How does it happen and get with the developer once a fix has been made so you can understand how the bug happens and what the fix implies so you can design tests around it? Add the tests to the regression suite or if there's a test that was designed to catch the bug, revisit it and modify it accordingly.

Third, unless it is something critical, take some time to test the fix as best you can. Quick fixes and releases are some of the most common ways bugs are introduced to prod. The rush to fix an issue and push it live hinders our thinking at times. Control your emotions and focus on thinking clearly.

It's not about blaming people. It's about working together to create the best software you can. As long as the team understands that, everything will be fine.

Answer 9

Gather information: Collect as much information as possible about the defect. Identify the symptoms, the impact on users or customers, and any error messages or logs that may be relevant.

Reproduce the defect: Try to reproduce the defect in a test environment. This will help you understand the root cause of the issue and develop a solution.

Prioritize the defect: Once you have a better understanding of the issue, prioritize it based on its severity and impact on users or customers. This will help you allocate resources and determine the urgency of the fix.

Communicate with stakeholders: Keep stakeholders informed of the situation, including the status of the defect and any steps being taken to resolve it. This will help manage expectations and maintain trust.

Fix the defect: Develop and test a fix for the defect. Be sure to follow established development and testing processes to ensure the fix is of high quality.

Deploy the fix: Once the fix is ready, deploy it to production as soon as possible. Be sure to communicate any downtime or disruption to users or customers.

Review and learn: After the defect has been resolved, conduct a through analysis to identify the root cause of the issue and determine what can be done to prevent similar defects from occurring in the future. Use this information to update your development and testing processes.

Prepare the learning docs and send across the team: Based on the root cause analysis and how it was recreated that scenario. What approached was followed to test existing defect in various environment such Dev, QA and Staging environment. Also make sure testing was done including Component testing (CT), Component Integration Testing (CIT) and System Integration Testing (SIT).

How the Testing should be perform in Microservices Environment https://martinfowler.com/articles/microservice-testing/