Order/Handling of thousands of defects just before release

Question

This question has been asked to me in an interview with a really good company.Below i will provide the question in the form of our interaction(M:me & I:interviewer).Though there is no definitive ans but I need to know what idea/answer the interviewer really wanted:

I: The scenario is you and 2 others consist of the testing team. You,the lead, are the only one who can do automation, others can do manual testing only. You have nearly 10,000 bugs that have been raised and you have 4-5 weeks or less before this product is delivered. What will you do to ensure that the product is delivered in time?

M: Filter the bugs priority wise and retest them. In the meantime keep a log about what functionalities are facing more regression and thus start automating them. Similar or related bugs will be given to others for further testing.

I: Say suppose none of the bugs have been marked with any priority. What will you do?

M:I will filter with dates.In any kind of SDLC, even the agile one, the core components are developed first, if there are core bugs, they need to be fixed first.

I:(Disapprovingly)What if a very important functionality is added in a later sprint? Also how will you make use of your team-mates and your ability to automate.

M:Along with the date, as a tester I will have to know the core & important functionalities of the product till date.So keeping that in mind will find the core areas of each sprint to work on(about team mated answered the same thing as before).

I: Say suppose, the bugs haven't been marked with the timeline of each sprint. What will you do?

M:I will search the bugs list with keywords which represent the important functionalities without which the product cannot be released. I will pick the bugs from there.

I:(Again disapprovingly) With a keyword, you will get so many results, will you go through them one by one?

M:(slowly losing hope)I will just go through the title and decide.

I:Generally titles are not so explanatory, how will you handle?

M:I will start testing the product myself and search with similar bugs that I face, rather than trying to go through bugs because I need to take decision for the product delivery.

I: So you will ignore those many bugs? Stakeholders may not agree. (After this I totally lost it and just kept on blabbering and I don't remember what else was asked.Also everywhere the management/work of the 2 other manual testers was asked)

This was an interview for Sr SDET.

Answer 1

In addition to what the other answers have said, I would say the interviewer is looking for how you, as a new addition to a team, would deal with a no-win situation. Frankly, I'd suspect that- at minimum - the company has found itself in this kind of situation in the past. At worst (I freely admit that I'm cynical) something similar is going to be facing whoever gets the position.

As an interviewer, I'd want something like this from the person I was interviewing:

First, I'd be wanting to know how these bugs are organized, particularly priority, severity, and risk. I'm assuming that I'm coming into this situation and not that I've been involved from the beginning, because this kind of situation suggests that things have gone badly wrong somewhere.

If the bugs are not organized in a way that involves priority, severity, and risk, I'd want to talk to the other testers, project management, and development to determine what problems they know of that pose the biggest risk to the projected deployment date.

If there is such an organization, I'd be looking to talk to testers, project management, and development to confirm the highest risk bugs. Ideally, I'd be looking for a way to build a list of bugs that must be fixed before the product can be released. With 10,000 bugs, that list is going to take some time to create, and that's assuming there are no bugs that testers haven't been able to find because the reported bugs are hiding or blocking them.

Once I have an idea of how bad the situation is, I can decide if - in my opinion - it's possible to release the application as planned. If most of the bugs are relatively low risk, and the high risk bugs seem to be reasonably easily fixed, I'd focus my team on the high risk bugs, and work with the project manager and any other team leads to get the highest risk (high severity, most likely to occur in the field, and/or block areas of the application) bugs fixed and tested.

If I can't see a way to release the product in time, I'd start talking to the project manager and my boss to see if there's a way to do a limited beta release of solid functionality or to delay the release. Since I'm new to the position, I don't know if there are any contract requirements or other factors outside my control that could force the release date to be immovable.

I'd also make sure that after the release I got with the leaders of all the teams involved to work out how such a situation came about and what actions we could take to prevent it happening again, as well as how we can work together to bring the bug backlog down.

Note that none of this has anything to do with the SDET role. It's clear from the question that the interviewer expects an SDET to also act as a test lead - I don't think this is a particularly good thing, and frankly, I'd be wanting to know if this is something the company expects from its SDETs.

It's worth remembering that even though interviews are high-stress situations, trying to think sideways and look at the implications of the questions you're asked rather than diving in. It's hard to do because you're stressed and trying to give your best, but if you can take a little time to mentally ask yourself what the interviewer is looking for with the question, you can usually give a better answer.

Answer 2

The first thing that comes to mind is -- were these tests ever working before? If so then, don't panic. Something has changed either in the codebase or the testing framework that is probably causing groups of them to fail. Track that down and see if you can eliminate several thousand fails at a go. You're still going to need to go read over the ones passing again manually and double check them but maybe that will only take a few days.

If they never were checked before I'd still do something similar -- look for any commonality that might let you fix large groups at once.

Otherwise there is so much noise there it might cause you to miss something critical that is failing.

After that accept that you might not be able to get to everything and focus on the money maker code path. The things that have to work or the business folds. Then after you've cleared a few more of those, every other day or three look and see if there are anymore grouped failures like mentioned before and give a go at clearing a few more groups.

Note: Answering this from the point of view of an SDET -- someone that can fix the offending code base itself.

Answer 3

If interviewer was mentioning bugs and not test failure ( if its test failure refer answer by @Lewis

First of all having 10000 active bugs in a product is really big red flag .

And you should never release such a product. But if the management decision is still to release then,

The answer the interviewer was expecting would be "severity"

The team should focus on fixing high severity bugs first if there are no priorities and keep low once on hold if it's not a urgent requirement and, is not affecting the actual business logic.

And concentrate on automating smoke test initially , then start automating all regression suites

Group the bugs and see where the bug clustering happens and rigorously test that module once the fix is made.

Before release manually test all the smoke test scenarios ( Critical businesses logic)

Also , having 10000 bugs can result in defect masking where these bugs are masking some critical bugs within the product.

So once the fix is made, more rigorous testing should be made around the modules to dig for more critical bugs

so if i was in the interview , i would answer like:

1. 10000 bugs in any project would be a huge red flag , it shows there was no proper bug fix process and estimation in place. I would surely worry about defect clustering and defect masking , meaning there are possibility that most of the bugs are concentrated on a single module , and this much bugs may be masking any other critical bugs which will only be identified after fixing and retesting the module rigorously. And will recommend to push the release date further due to this reason.

Mean while as development team is busy with fixing the bugs , we will start automating the smoke test use cases and bug use cases . Once the fix arrives we would assign the retesting tasks to the manual testers and we our self do rigorous adhoc testing on the module to find any masked critical bugs.

2. If there is no priority then it would be idle to revisit critical or high severity bugs first , and also to investigate bug life time and understand why bugs where not fixed for so long to help in improving the overall process in future.

About the low severity bugs we need to make a team decision on the timeline and on the release decision of whether to release the first version with these bugs but still documenting the same and workarounds where required. Also provide the next release date for the possible fix if possible.

So being a senior QA you should put forward your strong opinion to stay "NO" when you see red flags . Do not be too flexible

Answer 4

The other answers here are good if the point of the question is to give a concrete answer.

However, a lot of interviewers ask vague questions without a specific answer because they want to know how you think or to understand if you are making assumptions about the question. They want you to ask clarifying questions to them to get specifics. This helps guide your answer.

The scenario is you and 2 others consist of the testing team. You,the lead, are the only one who can do automation, others can do manual testing only. You have nearly 10,000 bugs that have been raised and you have 4-5 weeks or less before this product is delivered. What will you do to ensure that the product is delivered in time?

Some questions to ask:

• How experienced are the manual qa testers?
• Are the manual testers experienced in this project? Or are they new to the project as well?
• Do all 10,000 need to be fixed before the delivery date?
• Is there a bug tracker software the teams uses? If so, what?
• How are the known bugs tracked? Do they have a priority, severity listed? Are they grouped/tagged by feature?
• Are there any automated tests currently in use for software? If so, how many unit tests, integration tests, UI tests? Or, do I need to create all automated tests/framework from scratch in the 4-5 week timeframe?
• How much testing are the developers responsible for? Are they creating unit/integration tests?
• Are the 10,000 bugs UI bugs? Or a mix of bugs that can be tested using unit tests, integration tests, UI tests?
• What devices need to be used for testing?
• What is the level of quality we need to reach to satisfy users and stakeholders? How do stakeholders perceive quality?
• How do stakeholders determine a successful project launch?
• What is the teams definition of done?
• Will the team have time after project release to fix bugs? Or are we moving on to the next project? How much time will we have if we do get time?
• Is the team using Agile SDLC or Waterfall SDLC?

There's an endless number of questions you can ask to get clarification you need to provide a well thought out answer.

And, from the detailed conversation above, the interviewer kept prompting for specifics on how to include the manual testers in your plan. This gives you a big hint of what the interviewer is looking for: they don't want you to take on the full burden of testing this project yourself; they want to know as a Senior level SDET/QA Engineer how you mentor/lead a team of junior level testers.

Keep in mind, interviews shouldn't be an interrogation where you just answer their questions. Interviews should be a conversation where you can ask anything that helps to clarify their questions.