How to isolate test cases from each other?

Question

Our system under test is a GUI application. It appears beneficial to have test cases isolated, i.e. their execution does not depend on each other. For example, a later test case should not rely on objects created in a prior test case. By isolating them, there are no implicit dependencies and it becomes easier to maintain them (rearrange them, rewrite them, etc.).

The following options to implement isolation come to my mind:

• objects created during a test case are deleted explicitly at the end of it; global settings that were changed are restored to their defaults explicitly
• restart the SUT and reset it to its defaults by deleting the application's data store (e.g. directory in Windows' %appdata%)
• if your application allows the import of settings, import a defined settings file (e.g. the defaults after installation) at the beginning of a test case

Are there other options? Do you know of any literature? Or is isolation not so important, or even undesireable?

Answer 1

For most applications, its state is store in the database, so, simply restarting the database and putting it in the desired state is ok.

For relational DBs, using procedures and calling a specific script which does the work described before is easy. The point is to study which tables must be deleted and which cannot, in order to have good performance.

Hooks and pre-scenarios functions can be used for it.

If you have global settings that are changed by tests, they should be injected by the test cases, having a default if you don't need nothing in a test. This way, the SUT will expect to receive a new configuration at each run, throwing away the alterations done by the previous test.

Answer 2

You want a number of levels of testing that require different sets of data setup. Unit tests, Unit Integration Tests, System Tests, System Integration Tests, etc... You really should consider each of these as separate pieces of testing that build upon each other - i.e. start with simple isolated testing that is passed before going onto more complex interconnected testing.

For tests involving database backends, you have a few options, depending on the size of the database and the time it takes to setup the initial database

• Separate database instance per test. This really only works when the database has minimal data to perform the tests and the database is small. Often just in the case of Unit Tests.
• Single database instance for all tests. For this you need to control the data in your test database very strictly and have no data artefacts from one test that affect another test. This being the case, you need to have set-up scripts to create your test data, then execute your tests (either through a UI or on the database directly), then importantly, perform a teardown of your test data, to leave the database in the same state that it was in before running the tests.

All tests need to be atomic and repeatable, otherwise you cannot be sure that your tests will pass when they should.

Answer 3

This is what we call as pre-conditions and post-conditions of a test case.

In JUnit, every test case is marked with @Test annotation. Pre-conditions for each @Test are defined separately and marked with @Before annotation. Similarly, post-conditions of each @Test are defined separately and marked with @After annotation.

You can achieve similar isolation by creating a different action set/operations for the pre-conditions and post-conditions.

Few things to remember when defining pre & post conditions:

1. Pre-conditions should initialize all objects or connections and not Tests.
2. Post-conditions should terminate and dump all objects or connections.
3. Tests should either refer or modify objects and connections, but never initialize or close them.