3

Our system under test is a GUI application. It appears beneficial to have test cases isolated, i.e. their execution does not depend on each other. For example, a later test case should not rely on objects created in a prior test case. By isolating them, there are no implicit dependencies and it becomes easier to maintain them (rearrange them, rewrite them, etc.).

The following options to implement isolation come to my mind:

  • objects created during a test case are deleted explicitly at the end of it; global settings that were changed are restored to their defaults explicitly
  • restart the SUT and reset it to its defaults by deleting the application's data store (e.g. directory in Windows' %appdata%)
  • if your application allows the import of settings, import a defined settings file (e.g. the defaults after installation) at the beginning of a test case

Are there other options? Do you know of any literature? Or is isolation not so important, or even undesireable?

  • As a side remark: some test cases might very well depend on each other, if it does not get out of control. However, having one huge list of test cases where each one changes some global settings during the execution, and test cases rely on that implicitly, appears naturally undesireable: because you do not know in what state the SUT is when adding further test cases, and rearranging cases is almost impossible. – Richard Kiefer May 4 '17 at 8:22
  • 1
    Most test frameworks have hooks for this functionality built in, allowing for setup and teardown. I don't think there's a one size fits all solution . . . sometimes you want clean slate, other times you want state. – ernie May 4 '17 at 17:43
  • Thanks @ernie. Still, what are best practices to achieve a clean state? – Richard Kiefer May 5 '17 at 7:23
  • I don't think there are any particular best practices, more just doing what works for your problem. Setup/teardown methods in your tests if that's sufficient, spin up/spin down instances/VMs that are snapshotted, bring up DBs that are in known states, etc. – ernie May 5 '17 at 18:24
1

For most applications, its state is store in the database, so, simply restarting the database and putting it in the desired state is ok.

For relational DBs, using procedures and calling a specific script which does the work described before is easy. The point is to study which tables must be deleted and which cannot, in order to have good performance.

Hooks and pre-scenarios functions can be used for it.

If you have global settings that are changed by tests, they should be injected by the test cases, having a default if you don't need nothing in a test. This way, the SUT will expect to receive a new configuration at each run, throwing away the alterations done by the previous test.

1

You want a number of levels of testing that require different sets of data setup. Unit tests, Unit Integration Tests, System Tests, System Integration Tests, etc... You really should consider each of these as separate pieces of testing that build upon each other - i.e. start with simple isolated testing that is passed before going onto more complex interconnected testing.

For tests involving database backends, you have a few options, depending on the size of the database and the time it takes to setup the initial database

  • Separate database instance per test. This really only works when the database has minimal data to perform the tests and the database is small. Often just in the case of Unit Tests.
  • Single database instance for all tests. For this you need to control the data in your test database very strictly and have no data artefacts from one test that affect another test. This being the case, you need to have set-up scripts to create your test data, then execute your tests (either through a UI or on the database directly), then importantly, perform a teardown of your test data, to leave the database in the same state that it was in before running the tests.

All tests need to be atomic and repeatable, otherwise you cannot be sure that your tests will pass when they should.

1

This is what we call as pre-conditions and post-conditions of a test case.

In JUnit, every test case is marked with @Test annotation. Pre-conditions for each @Test are defined separately and marked with @Before annotation. Similarly, post-conditions of each @Test are defined separately and marked with @After annotation.

You can achieve similar isolation by creating a different action set/operations for the pre-conditions and post-conditions.

Few things to remember when defining pre & post conditions:

  1. Pre-conditions should initialize all objects or connections and not Tests.
  2. Post-conditions should terminate and dump all objects or connections.
  3. Tests should either refer or modify objects and connections, but never initialize or close them.
  • If I understand you right, pre- and post-conditions initialize or dump objects, but they do not validate a certain state? Guessing the function from their name, I had assumed they work like assertions. For example, the SUT's use case is to manage accounts, and each of my test cases assumes that no account exists besides those which I will create during the Arrange-section. Do your pre-conditions check that assumption explicitly? – Richard Kiefer May 13 '17 at 16:06
  • Yes, existence/non-existence of test data or environment settings or system configurations for each test should be validated in the pre-conditions explicitly and be reset to default values or state in post-conditions. – Zeeshan Siddiqui May 13 '17 at 22:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.