When I find a defect, how can I know if it is a defect or it is an environmental issue?
Usually, the application under test interacts with other systems, so, for example, I will be testing on one system and the other team will be working on another system. We have to communicate between then, so there will be times that we find an issue but then can't determine if this is an issue with the application or an external system has failed?
There is no simple answer to this, and no generic solution.
A few heuristic I have used over the years are:
Lion in the desert algorithm- assuming that the results are reproducible, remove parts of the environment until you can no longer see the problem. Repeat until you isolate the source. Possible problems: removing parts of the environment can still cause a real bug not to reproduce.
Logging- collect all the logs from the system, align the timestamps and try to build the flow of events and deduce the problem. Possible problems: you may not have enough logging turned on or implemented
Repeat on a different environment- as the title says, simply repeat on a different environment, different starting conditions, different configuration etc. until you can deduce what the problem is. Possible problems: this method should be combined with other methods since the lack of a bug doesn't necessarily explains the source for it.
Use the debug version- it's a subset of the logging idea, debug versions produce more information helping to debug problems, for example better stack traces. Possible problems: debug versions are not easy to get, and sometimes the change in build causes bugs not to reproduce.
The bottom line ? there is no bottom line, some cases will be very easy to filter out but many will be next to impossible to debug.
So, where to start . . . and some of this may well be things that fall into the class of "it would be nice to do that, but I can't."
First, there seems to be some confusion over what you're testing. If you're testing something that needs to interact with the outside world, and you want to focus your testing on that something, you need to do one of two things: either replace the outside world with mocked servers that only return the data you want to return, or you need to control the servers the system under test is talking to. If you can't do either of those, and more to the point in this case, the server being communicated is changing as well, you aren't just testing "your" software, you're testing the system. System failures are failures that will need to be debugged.
Second, if the software under test is misbehaving in some way, and you can't figure out why, that's a problem in and of itself. Put yourself in the place of the end user; if you, as a tester, can't figure out what's causing a problem, a normal user won't be able to either. As Rsf says, you probably need more/better logging.
And, finally, you'd originally put a tag in here about test automation. The environment you describe, where you don't control a lot of the pieces, isn't a good fit for automated testing. By it's nature, you really need to control all parts of the system when doing automated testing, so you can be sure that the automated testing is seeing the behavior you intend it to test.
If I can't I need to consider it to be due to either environment, state or intermittency. I am then down to logs and event viewers to try and trace events and find cause. Past occurances can be helpful in spotting the specific pattern that is causing the error.
