We are currently in a similar situation, except not with Selenium, but with CodedUI. I think for the concept the technology does not really matter. We have defined a critical path with functional locations that are most critical to our application. This because we have nearly no automated test coverage and want to start with the most important parts first, this to give fast feedback to our developers.
We start the testrun with a minimal dataset, just the bare minimum to start the application. Afterwards we create users, give them rights, enter maintenance data and walk through the whole workflow of our application in an automated test chain. The whole chain now takes around 30 minutes to execute and is growing rapidly.
To be able to start a part of the test chain standalone we cut the critical path into 100+ smaller tests. Each test Arranges the data, Acts the function and Asserts that it was succesful. Then it saves its database state to a backup file which is named after its class + methodname and the database version number. (If the database version is increased all these backups are invalided and deleted.) The arrange function of the next test in the chain depends on the previous test and starts with a function DependsOn(Class.testMethod). The DependsOn() function, checks if a database backup exists and if so restores it and brings the application in the expected state. If it does not exist it executes the previous test in the chain and restores the backup file created by that test. Its possible that a test started later in the chain, calls a lot of other tests.
Our single test flow
- Load backup of application state of previous test
- Login and bring the application on the correct page
- Arrange some more data
- Act test steps
- Assert outcome
- Logout/close application
- Save application state to backup file with unique TestMethod name
It could be important to login, navigate and logout the application. This way you are sure the start of the test is in the expected state.
Now it is possible to re-run tests in the end of the chain very fast, because they just need to restore the previous state and act its steps. This beats running the test for 25 minutes just to see it fail due to a human error in your test code... :) This makes test debugging easier.
For test development I would suggest to be able to run the chain on a virtual machine and in find a easy way to get the application states to your development machine, so that you do not have to wait for appending new tests to the chain.
Reasons not todo this:
- Tests depends on other tests. If one test in the beginning of the chains fails, the whole chain fails. This could be a problem if the reasons it fails is a low priority or if you do not run your tests on each checkin and fixing it costs a lot of time. In this "failed" state other developers might be breaking functionality further up the chain and you are not aware of.
- Saving and restoring the application state costs time and is a bit overhead.
- It makes the test framework more complex and its not always clear on which tests to depends. Certainly for newcomers to the test suite.
- We have forks half way in our chain which have started to live their own life.
- Changing tests in the beginning of the chain can be dangerous, later tests can fail, because of changed background data. This could be time consuming in researching tests fails.
Why are we doing this then???
The main reason is because its currently very hard to generate the test data from code. We want all test-data to be generated, because we don't want to maintain test databases for each test, nor do we want to duplicate database queries from the original code. Also we don't want to repeat all the setup steps for each test run, because this would make the individual test run very slow.
Our future and preferred solution
Slowly we are working on Classes with which both the tests and the application can create states and data. Preventing code duplication and updating tests after the code was changed. We have special TestSet classes which call the application classes and models to generate the state we want and provide the application state we expect. Eventually we will not need our chain anymore hopefully. But refactoring and restructuring all the background data is very time consuming in a 10+ year old application with millions of lines of code, also this process is dangerous without test coverage. Slowly we are moving out of this Legacy situation.
You want to be able to run each test in isolation, because as the test suite grows you want to be able to run them in parellel to speed up the testrun. Also you want to run the whole suit per checkin, so it becomes less time consuming to figure out what broke something. Even better you want to run the tests before the checkin is checked-in, with a pre-commit hook, to prevent the main branch to contain a broke state.
