Are your UI automated tests dependent or ordered?

Question

While writing UI automated tests, I've come across situations where test dependency looked like a must. For example, say I have two tests, where the second is dependent on the success of the first. I can add dependency (using TestNG) on the first test so that the second is executed only when the first succeeds.

But most programmers (I suppose xUnit users) support independent tests. That is, instead of being dependent on the first test, the second would need to bring the system to the right state before executing. To achieve this, the second test might invoke some common methods which would do the same operations as that of the first.

The problem I see with this approach is execution duplication. The first test is executed. Then the "independent" second test is executed (or it could be executed before the first, as there is no order of execution) which would essentially do the same things as the first test in order to bring the system back to a state where the second can be executed. So, this would take more time over all.

Do you write independent tests? If so, how do you cope up with execution time?

Answer 1

I break my automation up into 2 seperate types of classes. I have 1 class that I use for testing the actual requirements. These tests are independant of any others and broken up into stories. These also double as my test evidence in case of projects that require industry compliance (SOX, PCI, etc). I can't say that I'm overly worried about the execution time of these.

The second class is in my regression suite. Although none of them require a seperate test case to be run before any of them, some of them are deep enough into the web application that they can not be run without the same steps as a previous test (for example, I can't test anything behind a login page if the login functionality does not work). There will be some duplication in these as well, however, it's minimal. Although I don't use the same "test" again inside of it, I do use some of the same functionality while getting it to the required state.

The execution time can be extensive, however, if coded for minimum duplication (ie: happy path to get the system into the correct state), this can be reduced to a minimum.

Answer 2

You may as well use the dependent tests. The main reason to keep them independent would be to prevent one failure from hiding a second failure. If the second test is truly running the first test before executing its actual test scenario, then you don't get that benefit anyway.

Answer 3

With open source tools, and virtulisation, scale-out of automated testing is now a lot cheaper so I use that to get around the "takes more execution time" problem.

The next problem is now I am running the same tests twice, how can I be more effective. What I would probably do is try and come up with a scenario where part A of your test now use the same functionality, but test different things or code paths in that space, using different test data for example.

Now that I have more time, and I can apply differerent data, I can make more effective use of automation to cover more and more test scanarios.

I then repeat the cycle and continue to buid more and more tests, as there is always another scenario that you can think of and add to the mix.

Answer 4

Failures during the actual test are not the only problem with dependent test cases. You can have a failure during the checking phase that can cause a problem with your dependent testcase, for example. You could also have a sporadic problem that only affects one out of twenty cases for the first test case. If your cases were independent, you would have a 95% chance of still successfully collecting the second result even when the first test case failed. I would also expect independent tests to be easier to repeat multiple times in a row - such as a tester would do when trying to reproduce a sporadic bug - although that might depend on your test framework(s). These arguments might not be adequate where very high amounts of execution time are at stake, but they are additional factors to consider.

Sometimes good product design and testability features can help with execution time, especially when verifying basic page elements (e.g., Button X calls function Y), where the process of getting there and the state along the way is less important than the final step. In web apps, for example, it can be helpful to just navigate to pages via the address bar instead of via the UI, and good design can make that possible in more situations. However, you will still have some end-to-end tests.

The basic approaches I see at that point are prioritization, and parallelization. Prioritization is just running the test cases most likely to find the greatest number of higher-priority bugs frequently, and running the rest of the tests only occasionally. Cases that should be prioritized are those that are most likely to be impacted by current code changes, and those that cover the most ground for their execution time. Parallelization means having multiple machines running the test cases at the same time, and is becoming more possible with virtual machines (which don't work for all programs, but are a great option for many apps using GUIs) and cloud services like Amazon Web Services. In general, having more machines working is usually cheaper than having testers put tons of effort into super-efficient tests (especially when those efforts could be put into finding new bugs instead!).

Answer 5

Although I follow the dependent approach provided by TestNG for my Selenium tests as Tarun mentioned, I have never come across any situation where one failure would hide another as Bruce pointed out.

I think that it might never happen, because every case that I make dependent on another one will always require the previous test case to successfully run so as to start the second one. Else there would hardly be any chance for dependence between test cases or test steps.

For example, we have two tests, say test T1 which is a login test and T2 which is something to be done after logging in.

Let's consider T2 is dependent on T1.

Here while I configure my tests to run through TestNG config file, I will call both tests T1 and T2 separately as two tests, i.e., I need

1. login test
2. Something that test after logged in

So, I find my dependent tests very tester friendly and fast enough.

Note: correct me if i am wrong

Answer 6

I write independent tests so that I have the option of running them in isolation.

If a test depends on another test, it may be because the antecedent test sets something up that the dependent test requires. Some test frameworks (e.g. JUnit) let specify that the specified setup login should run exactly once for a set of tests, no matter how many tests are in the set. You may be able to refactor your tests to leverage that feature.

If the setup process is lengthy, you might want a way for the setup process to persist a "setup finished successfully" flag that your tests can interrogate. For example, if your setup involves populating a database in a certain way, you might want to add a "setup complete" table that contains a row only if the setup was successful.

Answer 7

Yes I also do write tests that are independent of each other since it is the basic principle for testing. Since I use selenium for testing, I usually re-factor my code so that at-least the duplication of code is reduced.

Say for example, I have 2 tests to validate

a) Login a request

b) Update a request.

Now since b is dependent on a, I usually write the code in a function and call it from a and b both.

Will surely take time, but there can be ways where-in you can execute tests in parallel. In selenium I do it via Selenium Grid.

This is helpful when u just want to test one functionality.

Hope this helps!!!!