4

I'm hoping to open an informed discussion as to the best practices when dealing with defect management. As a tester, if I'm testing a website and there are visual defects, however the test case I am testing against does not quantify the user-interface as part of the test.

One option is that:

As a quality assurance tester we would lose integrity by ignoring defects we know won't be raised by our test cases, simply because they aren't specified to be tested or part of the requirements.

Another option is:

Said element/defect is not within the scope, therefore irrelevant to our testing, henceforth the ticket should be passed rather than fail as it wasn't specified.


My team and I are currently discussing this right now with a real world issue, I'd really love to hear feedback of other cases and best practices.

2
  • 1
    Is there not a happy middle ground, where you pass that specific test case but raise another one to track the issue you've found? This doesn't seem like a binary choice, you have other options.
    – jonrsharpe
    Jun 26, 2019 at 12:39
  • 1
    I would consider assumptions for the test case and report them as warnings rather than errors if they fail. If I decided not to use those test cases as out of scope, I would add a qualifier to the the introduction of the test report, highlighting that the UI faults had been observed but were not detailed as they were out of scope. This choice primarily an issue of resource management; how much time is spent doing that. Jun 26, 2019 at 12:45

8 Answers 8

4

Don't ignore the defects. Do document them in some manner.

From there though, a lot of it is up to you and how your organization operates and how late in the development life cycle you're discovering these defects. The most important QA deliverable in my experience is an accurate description of how well the application under test meets expectations. When I've been in similar situations to the one you described we've normally broken out defects that don't fit into the existing test suite into bug tickets. We'd communicate these bugs (in addition to the usual test suite/results) to business and engineering stakeholders. This helps to inform the decision to fix the defect immediately, fix eventually, or ignore entirely.

You also may want to add a test case for this defect if or when it's resolved!

1

A test shouldn't fail for an unrelated issue (i.e. not-included step).

If you do come across something else though, then you need to find an applicable test that covers it and then fail that. However I agree it's possible you may find something that wouldn't be included in any test, so you may have to write a new very specific test for it.

e.g. Imagine you save a record and it saves successfully, but then a message pops up for a couple of seconds "DEBUG1 - Saved as {GUID} to {Table}", then it moves to the correct next page.

You'd immediately imagine that it's been added by a developer to get info themselves when checking something. It's also definitely a defect.

But any tests you have for that area are very unlikely to specifically mention that it should have an absence of messages between the change in states for Saved -> Next screen.

What happens next will depend on the company and practices, as if you're TDD you may need the test written for the dev to then work against. If you're BDD you may log a new defect and then be aware you'll also need to write a specific testcase for it after the work comes back as complete.

However you function, defects should always be reported somewhere.

1

The thumb rule is that we should not ignore any defect/observation found during the testing. In case it is unrelated to the said test case, we should find one which it is related to in the scope. Rather than qualifying a testcase, the focus should be on qualifying the test cycle.

If you are unable to find any related testcase in the scope, options can be multiple depending on your responsibilities during the tests:

  • Create a new test case
  • Update the test case to include the scenario
  • Document the observations and share with the stakeholders

Also, interested thing would be to find the instances of such incidents where “Said element/defect is not within the scope, therefore irrelevant to our testing”, in case of multiple instances the scope of the test should be revisited. Are we testing the right set of features or some modification is needed in the scope itself?

1

Option 1 would mean holding up your current work, which not only jams up your team, but hurts your users by delaying the bug fixes, enhancements, or new features that would make their lives easier.

Option 2 is just as bad. Ignoring a defect is my #1 NEVER list. Doing so just guarantees your product and users will suffer in the future. Failing to address it increases the risk that it’s impact will become more severe.

So, what do you do?

Refuse to choose. To achieve both objectives:

1). Pause work on your test case (let’s call it TC01.”)

2). Create a new ticket that describes the defects you found. We will call it “D01.”

Within in you will include all the usual things such as:

  • Environment (OS, browser, etc)
  • Description of problem
  • The change/fix you are requesting

And then you will also add:

  • “LINKED TO TC01” (along with a few words on that test case you were working on when you found it.

3). Return to your test case TC001.

  • “LINKED TO D01” (along with a few words saying that you ran into it, determined it was not within scope to block this current project, and created the new ticket).

Ideally, you should actually hyperlink the two tickets/cases directly to one another. Many testing software programs do this for you automatically.

4). Finish testing your test case. Once it’s complete, close as successful.


By taking the 5 minutes to document the defect and correctly pass the test case you are giving your company and your users the best outcome.

Happy testing!

0

My first instinct is to imagine the following.

  • Defect invalidates your test case - Then you're Blocked. Pass or Fail doesn't exist
  • Defect is irrelevant to your test case - Pass and Fail still exist.

As a safety measure, for the second case, in any reporting you always flag that the defect exists

0

As others have said, you need to find a way to report the defect.

The purpose of testing is to make sure your customers get the best possible product. If there are visual issues that could make a page/screen a poor experience for the users, it needs addressed.

My first thought, based on our tracking system, is to log a new defect & tag it as 'found while testing'. That indicates (at least as we've defined it) as something unrelated that we tripped across while doing some other testing or work.

Your team needs to determine what works best for your internal process to handle these types of issues. What is "right" for you might not be for a different team.

0

Simply: NO. Do not fail a test case, if it is not connected with a bug.

Explanation: If you can pass through the test case then there is no reason to fail it. If the bug you raise is not specified by the test case, just leave the bug without relation to the test case or you can fail another test case which contains that. This is very time-consuming to find it... so your managers should accept, that some bugs are not connected to the test case. This could mean, that you do not have 100% coverage of the functionality by the test case, so be careful. If the 100% coverage is in play, it could be done by creating a new test case for each unlinked bug... I personally do not like that, the same as 100% is not reachable.

Report every bug you find during the testing. This is the purpose why you test. Test cases are just a tool of how to do it.

0

Our general goal is to make sure that the given application works as expected, and defects certainly ensure that we don't meet our goal. Hence, the defect can't be ignored and must be reported/tracked. Going by the practices of good quality assurance services, you can consider these points. If the defect is part of the test case flow, then test should be marked as fail.
Else if the defect isn't part of the test case flow, then the test should be marked as pass. And another test should be created which has the defect as part of the flow, which will obviously fail.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.