What if the software is so buggy it can't really be tested at all?

Question

I know the best bet in this situation is for the testers to go through the process of reporting blocking-type problems initially show up, with the focus being on critical bugs. Since, this type of problem can high severely affect on schedules and indicates deeper problems in the software development process.

This situation will occur due to insufficient unit testing or insufficient integration testing, poor design, improper build or release procedures, etc. These are the major reasons.

What are the other steps which we have to follow as a QA if the software is so buggy?

Answer 1

Continue testing in this situation would be counter-productive and may lead to a "deadlock" - when the reported issues would depend on each other recursively.

• make sure the management is aware of the situation
• sit down with the development team - determine the most critical and problematic areas and the step-by-step process to get out of the testing "deadlock"
• follow the improved/fixed areas of the software in the "breadth-first" manner - from most critical to less critical
• do the root-cause analysis - what led to this kind of situation, what can be improved to not let this happen again - was it a design/architecture issue, poor-documented software requirements, lack of different types of testing, communication issues etc.

Answer 2

Ideally, suspend the testing. And it's time to refer the Test Plan document.

This is the time when one should look at the Testing suspension and resumption criteria section in the Test Plan document.

I can refer to the following three points that are generally mentioned under testing suspension criteria and this scenario looks like falling under point 1 and/or 2:

1. A bug is found which requires fixing before further testing is possible, or which renders further test cases obsolete.

2. The developed software is not functioning well enough for the tests to be meaningful.

3. Problems arise with test environment or access and/or setup is incomplete and not available.

As mentioned, the software can't really be tested because of critical bugs. So, the testing team should raise a red flag along with data (a list of well organized bugs found so far) in order to present their case.

And testing should be resumed once the conditions for the resumption criteria have met:

Resumption Criteria - Testing can be resumed only when

1. Situation due to which testing was suspended has improved and it is feasible to go ahead with the testing activities.

Answer 3

What are the other steps which we have to follow as a QA if the software is so buggy?

This question reminded me of an interview question:

• During one software security interview, one candidate was asked: "How would you fix a zero-day security bug?" This candidate replied, "I would start from fixing 100-day bugs."

100-day bugs is a made up term for older security bug comparing to zero-day bugs.

When a piece of software has accumulated too many bugs, it is almost certain that some bugs are more fundamental than the rest, e.g. some bugs have results of other bugs. In other words, mistakes made earlier lead to more mistakes later.

Apart from Aalok and alecxe have stated, is it possible for you to categorize the bugs so that you can start from the ones that most likely to have compounding effect?

Fixing them first and you may notice other bugs will be fixed in the process as well.

Answer 4

If you are a developer...

1. Refactor code every time you touch it.

2. Know the application's behavior before refactoring.

3. Refactor as if you have to write unit tests (if you know how to write testable code).

4. Include estimates, do change impact analysis, and let your manager know about it.

If you are a tester...

1. Talk to the developer if they can batch bug fixes.

2. Know the application behavior and prepare test cases.

3. Don't blame developers for new bugs appearing because of new fixes. Be patient.

If you are a/the technical lead...

1. This could be a nightmare if not taken care of soon.

2. Talk to the developer(s) about estimates.

3. Motivate the developer when code must change for the defect to be fixed. Change the code to be testable with automated tests.

4. Appreciate automated tests, even if it increases development time.

Eventually, all of you are responsible for working or non-working software. So be patient and keep making it better every time you change it to fix something.

Anyways, I'm in the middle of doing something similar so sharing experience.

Answer 5

Often this situation happens to combine with a high pressure from management to get the product out, and they may not be receptive to QA telling them that it is unreleasable.

To make the best out of a bad situation, it is useful to find and follow through the fix for the bugs that most annoy the users. This is easier if the software is used internally in the company or in some close partner companies, but one can try to seek out a few example users anywhere.

It is surprising how well people can adapt to very buggy software when they have no choice. Just a few bug fixes can make a marked improvement even in a lousy piece of software.

Answer 6

The answer is Acceptance Tests. Your software should meet a baseline set of functionality (usually determined by the PO, but could be dev/QA combo) to be considered "done", and thus ready for QA.

Answer 7

In this situation the first thing I do is report the observed bugs, and label them in some way as blocking further testing.

Exactly how the labeling is done depends on the tool and the circumstances, but I make sure each bug that blocks further testing is clearly flagged as blocking further testing. For instance, a bug that makes it impossible to save data clearly prevents any further actions with that data.

Once I've flagged my testing as blocked, I notify my lead and project manager. They need to know that I can't progress further because of a situation not in my control. At that point it's their responsibility to make sure the blocking bugs are fixed so I can get back to work on the software.

Once my lead and PM have been notified, I work on something else until the blocking issues are fixed. There's never a shortage of things to do, so having something to work on isn't a problem.

In my opinion the most important step is making sure your lead and project manager know you are unable to continue testing and why.

Answer 8

An automated test is also an application that has a set of requirements, a design, and a delivery date.

Use the time wisely to continue developing the test application, because if the SW team do fix their bugs just one day before delivery, it will still be expected that you can run your tests and give the ok. Keep everyone regularly appraised of the test result status - not just x% of tests passed but also a breakdown of the application under test into sub-components.

In that way, everyone knows where the hotspots in the project are. Whilst there is little value for the SW team in running tests that rely on other code working, it may help you to improve your test application.

Answer 9

The best way to handle these type of situation is to reject the Build with all the blocker and critical issues list. Apart from this, send email to the developer by including all the senior QA persons of the project so that everyone should be aware of the situation. For handling the things in good manner set up a meeting with Developer and Business team so that loop holes of the buggy software can be discussed in the better way. Almost every qa company follow these type of processes to handle buggy softwares.

Answer 10

When there are more defects with severity 1 and 2, this situation occurs.

As far as I have experienced, there are atleast three projects I faced the same problem and I followed systematic approach as below:

Basic and fundamental Pre-requisite:

Have you got all the test scenarios listed?

-If your answer is YES, then the work is simple (follow step 1). If not, please focus on deriving clear list of TEST SCENARIOS with tools such as Pairwise Testing (Orthogonal Array)

Step 1:

Can we focus on performing REGRESSION TESTS if possible within the next test cycle?

-Answering YES to this question will help in uncovering complete list of Sev 1 and 2 defects atleast (not the entire defects since full regression tests may not be feasible)

Shortcuts: Try to implement tools such as CypressIO to automate those tests hence you can extend the same framework to Unit Test (on the left side) and for regression testing/UAT (on right side of testing cycle)

Step 2:

Knowing when to stop testing is crucial. Figure this out in the Test Plan section 'Suspention Criteria'-do you have a percentage of pass rate accepted and mentioned in Test Plan?

If this percentage has been provided already,its great, else it has to be amended to make it like atleast 90% of the tests should pass in order to release the software (for example)

If you dont have a test plan , better to write one by following IEEE template at: https://jmpovedar.files.wordpress.com/2014/03/ieee-829.pdf