2

When I find a defect, how can I know if it is a defect or it is an environmental issue?

Usually, the application under test interacts with other systems, so, for example, I will be testing on one system and the other team will be working on another system. We have to communicate between then, so there will be times that we find an issue but then can't determine if this is an issue with the application or an external system has failed?

  • 3
    The title (and tag) of this post does not match the question asked. The question is nog limited to automated testing, it applies to bugs found in all sorts of software testing. – Patrick Feb 7 '19 at 7:39
  • 1
    I feel like this could be a great question but is a little confused and poorly tagged. – trashpanda Feb 7 '19 at 14:54
  • I tried cleaning this up; I'm not sure if it matches the intent of the question or not. – Kevin McKenzie Feb 7 '19 at 17:35
  • Is the question really "How do I avoid false test failures due to environmental issues/make my automation environment stable?" or "How do I determine the source of a defect?" – ernie Feb 7 '19 at 17:44
2

There is no simple answer to this, and no generic solution.

A few heuristic I have used over the years are:

  • Lion in the desert algorithm- assuming that the results are reproducible, remove parts of the environment until you can no longer see the problem. Repeat until you isolate the source. Possible problems: removing parts of the environment can still cause a real bug not to reproduce.

  • Logging- collect all the logs from the system, align the timestamps and try to build the flow of events and deduce the problem. Possible problems: you may not have enough logging turned on or implemented

  • Repeat on a different environment- as the title says, simply repeat on a different environment, different starting conditions, different configuration etc. until you can deduce what the problem is. Possible problems: this method should be combined with other methods since the lack of a bug doesn't necessarily explains the source for it.

  • Use the debug version- it's a subset of the logging idea, debug versions produce more information helping to debug problems, for example better stack traces. Possible problems: debug versions are not easy to get, and sometimes the change in build causes bugs not to reproduce.

The bottom line ? there is no bottom line, some cases will be very easy to filter out but many will be next to impossible to debug.

0

So, where to start . . . and some of this may well be things that fall into the class of "it would be nice to do that, but I can't."

First, there seems to be some confusion over what you're testing. If you're testing something that needs to interact with the outside world, and you want to focus your testing on that something, you need to do one of two things: either replace the outside world with mocked servers that only return the data you want to return, or you need to control the servers the system under test is talking to. If you can't do either of those, and more to the point in this case, the server being communicated is changing as well, you aren't just testing "your" software, you're testing the system. System failures are failures that will need to be debugged.

Second, if the software under test is misbehaving in some way, and you can't figure out why, that's a problem in and of itself. Put yourself in the place of the end user; if you, as a tester, can't figure out what's causing a problem, a normal user won't be able to either. As Rsf says, you probably need more/better logging.

And, finally, you'd originally put a tag in here about test automation. The environment you describe, where you don't control a lot of the pieces, isn't a good fit for automated testing. By it's nature, you really need to control all parts of the system when doing automated testing, so you can be sure that the automated testing is seeing the behavior you intend it to test.

0

First I make sure I can reproduce it

If I can't I need to consider it to be due to either environment, state or intermittency. I am then down to logs and event viewers to try and trace events and find cause. Past occurances can be helpful in spotting the specific pattern that is causing the error.

Once I can reproduce it

  • I start varying input and workflows to see how common and consistent the error is.
  • I ask the developer if they had the issue.
  • I try different browsers and devices (if UI tests) to see if they are a factor.
  • I look for the bug in different development and test environments.
  • I try boundary testing to see under what conditions the error occurs.
  • I use browser dev tools and network diagnostics to understand what programs are running where
  • I use different test data to see what data produces the error

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.