I have seen this recommendation many times that tear down is supposed to clear test data which is created during tests. This is supposed to avoid interference with any future tests. For example email-id which is used during test could be used during next run of test as well.
I am not comfortable with this approach as in the wake of test failures tool generated test data might come handy to debug the error.
Curious to know how other teams have been handling this situation.
I agree Tarun, and for exactly the reasons you stated. Leaving all test data in place lets you efficiently analyze and debug your test results.
If you want a clean test environment for each test execution, better to do the "tear down" task immediately before you create your test data for the current run.
Not cleaning up on tear down works as long as you are 100% sure that every test cleans up before setup correctly and always will (e.g., its setup won't get outdated). If you are the only tester and the only person using your environment - or, if you have a good 'clean the environment & restore to defaults' script / function that all testers use before test or which the harness runs automatically - then I think it can work.
However, most software does not want to stop and sit after a failure to maintain a pristine failure state with all of the data from the error still in its original form. Most software projects need to use test resources efficiently, and that means cleaning up and moving on to the next test. To have a debug-able failure, that means you need to save state anyways before moving on. Copy the error files and config files, back up the DB, dump machine information on CPU / RAM / etc. into log files, and so on, and put it all in one place so its easy for the tester investigating the issue to find everything s/he needs. Since most testers will want to save state before the test ends anyways, there's not a lot of advantages to avoiding a full cleanup during tear down.*
Also, if you don't clean up during tear down, you will need to maintain old tests when more configuration options are made available, so they set those new options correctly. This can be avoided by setting the machine to a 'default state' before running tests using a single function that is always called before tests, so you just update that one function - but if you are choosing not to cleanup after tests, test setup maintainability is something you likely want to plan for. If you don't have a process for ensuring test setup remains maintainable (and the feasibility of this kind of planning varies between companies), then cleaning up is a good idea.
I know in my company, I can't rely on other test developers to clean up everything before starting tests - largely because my company is planning to make huge changes to QA in the future, and I can't know for sure that the other testers using my harness will be skilled, or will understand the system well enough to think of all the things they will need to clean up. I don't get to develop policy, I can only try and influence it. I could let other test developers suffer for their bad habits, but I know that could rebound on me politically. So, I always clean up to the best of my ability.
There's also the case where the failure disrupts things so much that the cleanup code can't run. In this case, I personally like to have the failure to cleanup reported with the tear down for the actual test that caused the problem, rather than reported as a failure of the setup of the next test to run. If nothing else, it's easier to show to devs and management and explain as one failed test, not two.
*Not a lot of advantages to avoiding cleanup during teardown, but there are some: Maybe cleanup is expensive, and doing it during both setup and teardown is too time-consuming - cleanup during setup is definitely more important. Maybe you haven't had time to develop the 'save everything' infrastructure so you do want things to stop and sit, rather than having the run continue. Maybe you think the cases where a tester on your product doesn't clean up before testing are so rare that you would rather save the development time needed for cleaning up and trust that this piece of QA culture doesn't change over time in your company.
I don't believe by 'clear' it is meant 'delete'. It simply means you must ensure that one run does not spill into a second run. You may want to consider dumping the database before wiping it, or simply creating a new database for each run.
As long as there's a clear wall of separation and two runs on the same input yield the same set of data, there shouldn't be a problem.
If you run your automation tests in their own environment you can get around this by taking a backup, or snapshot, if your tests fail so you have the data error collected. Of course you'll also need to copy any log files or any other collection as well, I notice this is always neglected and sometimes what seemed to be a data error is something else entirely and then I have to dig through log files to find the one I want - if it hasn't been overwritten already.
Sometimes I ran automation tests in our Test Environment, but we structured our tests so we didn't need to clear the data - in some instances that was more difficult than was required for the test. Since we needed the production data and it was time consuming to install in Test, and we continually ran tests, we worked around it. Your mileage may vary.
To me the concept of clear goes back to when I wanted to do database compares before and after tests and I knew what state I wanted things to be in, this was the way to be sure the data was correct and the test passed. So it's good to clear, when you can and it affords you value, but you should carefully define what you mean by clear.
You want your tests to be repeatable; if you find a bug, someone else may want to reproduce it by running the same test under the same conditions. It is up to you to decide how to meet that goal.
There will always be times when the tearDown method fails or does not get a chance to run, e.g. when someone kills a test by hand, so I do not want to depend on the tearDown method to clean everything up. Instead, I only use a tearDown method to release any resources that subsequent tests might need.
If my test needs run under a specific set of conditions, I write my setup method to establish those conditions. If establishing the conditions is too heavy-weight a process, my setup method may simply check whether the conditions are true and signal an error if something is awry.
To answer your question: Yes, and No.
For bug testing, we use 2 different db. Usually I start with a "clean" database (usually "demo" db), but sometimes I use the development db which is full of all sorts of stress test type data. There is no general rule on this, and the devs always use the stress test db when coding.
During installation testing, about 98% of the testing is done using the stress test type data. This is because the majority of our customers will be converting their db to the new build.
It is an immutable law in test automation that you start with a known data state. Do not ever rely on tear down or the results of another test. Clean your state and establish your baseline data before the test runs. This is the only way to have reliable tests and has the advantage of leaving your complex data in a known state for debugging as the first responder so knowingly alluded to.
