Should automated tests rely on each other or be kept separate?

Question

Currently, I am in charge of developing a blackbox automated testing framework using C# and Selenium web driver and using this framework to automate our current manual regression suite. The system under test is an ASP.NET web application.

Each test may rely on a complex aggregate of data. For example, to test the creation of an 'activity', a 'client' must be present with a 'service' against which the 'activity' is to be created.

How should data required for such a test be managed? Should the test assume it to be present? In this case, the test would simply create the activity. Or, should the test create the requisite data? In this case, the test would create the client, create the service, then create the activity.

We're thinking of the following potential solutions and are debating the merits of each:

1. db is blank, test creates data as needed, db is wiped after each test

• pro: test and data are together, tests can not affect each other

• con: long run time, more complex tests

2. db is blank, fixture creates data needed for all tests, db is wiped after each fixture

• pro: test and data are close

• con: tests within a fixture can effect each other

3. db is populated with requisite data, tests assume requisite data is present

• pro: faster run time

• con: test and data are separate. Tests more likely to affect each other

Which, if any of these approaches is considered best practice? If none of the above, what approach is considered best practice?

Answer 1

I have worked in each of these scenarios:

1. DB is cleared after each test.
   • Pro: No artifacts left over so test are a little easier to write.
   • Con: The DB restore can take the longest if you do something, then roll it back repeatedly. You also need to ensure the rollback is successful.
2. DB is cleared after each test cycle.
   • Pro: No artifacts left over so test are a little easier to write.
   • Con: You also need to ensure the rollback is successful.
3. The tests check for existence of data and creates as necessary.
   • Pro: The test do not rely on the DB state.
   • Con: Takes more time to write tests. You also need to ensure the rollback is successful.

My Conclusion:

Option 1 and 2 are easier to develop and will likely execute quicker than Option 3. However, the test created for 1 and 2 should not be ran on production (by changing your Base Address, for example). Option 3 takes a bit more work upfront and may take longer to run loops and create data, but that should be a moot point. Does it matter that your automated tests take 5 minutes instead or 3? Not really when you compare to doing anything manually.

Answer 2

I don´t know if I got your special case right but in general it is better to keep test cases seperate in sense of having as less dependencies as possible between them. Thus for example it is easier to execute each test case standalone if you wish or need to. If you have a lot of dependencies you will alway have to run a lot of stuff even if you just want to check one single test case. To minimize dependencies you need to define clear pre and post conditions for each test case. You could also consider defing pre/post conditions for different levels - e.g. test suite vs. test case.

Besides that I think there is a lot of other things to consider regarding test cases dependencies e.g. robustness and maintenance of test cases, reporting and so on. Imagine an automated reporting which creates a Jira task for each failed test case. If you have strong dependencies between test cases such an automated reporting could create a lot of mess in your Jira.

P.S. hehe .. you´ve edited your post while I wrote my answer. So I want to add - plz clarify What is the difference between your second and third point? And .. tests that affect each other --> no good ;-)