4

I have a particular end to end test scenario which am trying to automate using Selenium Webdriver. If I automate the whole scenario into one test case, the test case is going to take a minimum of 25 minutes to complete which I don't think would be ideal. The scenario is somewhat like the following:

  1. Create a new competition.
  2. Add teams to Competition.
  3. Generate matches using automatic Fixture generation tool.
  4. Enter Results to matches.
  5. Publish Results.

I am thinking of breaking it down into multiple test cases, each testing a specific functionality but still interconnected. So the first test case will verify creation of a competition. The second test case will add teams to competitions created in the first test case And so on...

Is it possible to do?

I am sure automated test specialists have already faced this sort of issue and already designed a solution.

  • 1
    It's certainly possible - whether it's desirable is a different argument. If this data is stored in a database, is it possible to use predefined databases to reduce your data dependency issues? – Kate Paulk Jul 1 '15 at 11:32
4

We are currently in a similar situation, except not with Selenium, but with CodedUI. I think for the concept the technology does not really matter. We have defined a critical path with functional locations that are most critical to our application. This because we have nearly no automated test coverage and want to start with the most important parts first, this to give fast feedback to our developers.

We start the testrun with a minimal dataset, just the bare minimum to start the application. Afterwards we create users, give them rights, enter maintenance data and walk through the whole workflow of our application in an automated test chain. The whole chain now takes around 30 minutes to execute and is growing rapidly.

To be able to start a part of the test chain standalone we cut the critical path into 100+ smaller tests. Each test Arranges the data, Acts the function and Asserts that it was succesful. Then it saves its database state to a backup file which is named after its class + methodname and the database version number. (If the database version is increased all these backups are invalided and deleted.) The arrange function of the next test in the chain depends on the previous test and starts with a function DependsOn(Class.testMethod). The DependsOn() function, checks if a database backup exists and if so restores it and brings the application in the expected state. If it does not exist it executes the previous test in the chain and restores the backup file created by that test. Its possible that a test started later in the chain, calls a lot of other tests.

Our single test flow

  • Load backup of application state of previous test
  • Login and bring the application on the correct page
  • Arrange some more data
  • Act test steps
  • Assert outcome
  • Logout/close application
  • Save application state to backup file with unique TestMethod name

It could be important to login, navigate and logout the application. This way you are sure the start of the test is in the expected state.

Now it is possible to re-run tests in the end of the chain very fast, because they just need to restore the previous state and act its steps. This beats running the test for 25 minutes just to see it fail due to a human error in your test code... :) This makes test debugging easier.

For test development I would suggest to be able to run the chain on a virtual machine and in find a easy way to get the application states to your development machine, so that you do not have to wait for appending new tests to the chain.

Reasons not todo this:

  • Tests depends on other tests. If one test in the beginning of the chains fails, the whole chain fails. This could be a problem if the reasons it fails is a low priority or if you do not run your tests on each checkin and fixing it costs a lot of time. In this "failed" state other developers might be breaking functionality further up the chain and you are not aware of.
  • Saving and restoring the application state costs time and is a bit overhead.
  • It makes the test framework more complex and its not always clear on which tests to depends. Certainly for newcomers to the test suite.
  • We have forks half way in our chain which have started to live their own life.
  • Changing tests in the beginning of the chain can be dangerous, later tests can fail, because of changed background data. This could be time consuming in researching tests fails.

Why are we doing this then???

The main reason is because its currently very hard to generate the test data from code. We want all test-data to be generated, because we don't want to maintain test databases for each test, nor do we want to duplicate database queries from the original code. Also we don't want to repeat all the setup steps for each test run, because this would make the individual test run very slow.

Our future and preferred solution

Slowly we are working on Classes with which both the tests and the application can create states and data. Preventing code duplication and updating tests after the code was changed. We have special TestSet classes which call the application classes and models to generate the state we want and provide the application state we expect. Eventually we will not need our chain anymore hopefully. But refactoring and restructuring all the background data is very time consuming in a 10+ year old application with millions of lines of code, also this process is dangerous without test coverage. Slowly we are moving out of this Legacy situation.

You want to be able to run each test in isolation, because as the test suite grows you want to be able to run them in parellel to speed up the testrun. Also you want to run the whole suit per checkin, so it becomes less time consuming to figure out what broke something. Even better you want to run the tests before the checkin is checked-in, with a pre-commit hook, to prevent the main branch to contain a broke state.

  • 1
    This is an excellent explanation of the reasons to use dependent tests, AND the reasons not to do so. +1 (I'd + more if I could) – Kate Paulk Jul 2 '15 at 11:36
0

As you are using Selenium webdriver for automation, u can use cucumber framework approach to breakdown the test cases. A simple Gherkin format is used to breakdown the test cases, and also helpful to reuse the code number of times.

http://www.toolsqa.com/cucumber/first-cucumber-selenium-java-test/

Above link gives the cucumber format UI.

I am sure. You can break the tests and minimize the time in scripting.

  • I think his question is about test dependency. The second test is data depended on the first if you split them up. How does Gherkin solve this? I think it doesn't. – Niels van Reijmersdal Jul 1 '15 at 11:00
0

I'm using selenium and C# via visual studio, running the tests via Nunit. If run in their entirety my suite of tests would take over 12 hours to complete (I'm testing chrome, firefox and a single IE version in the automated suite). My devs have assisted by providing the basic framework including the database calls and functions to alter the web.config, snapshot data etc. My tests are structured in a single solution but grouped into projects and sub-divided to cover specific functions. The crucial thing when testing in this way is data design to make sure I keep the data sets as small as possible. This means that some of my tests have quite a bit of 'setup' in the Arrange section but this can be minimised by using multiple data sets.

The advantage is that I don't have to run the entire suite of tests and can concentrate on specific areas.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.