Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

While writing UI automated tests, I've come across situations where test dependency looked like a must. For example, say I have two tests, where the second is dependent on the success of the first. I can add dependency (using TestNG) on the first test so that the second is executed only when the first succeeds.

But most programmers (I suppose xUnit users) support independent tests. That is, instead of being dependent on the first test, the second would need to bring the system to the right state before executing. To achieve this, the second test might invoke some common methods which would do the same operations as that of the first.

The problem I see with this approach is execution duplication. The first test is executed. Then the "independent" second test is executed (or it could be executed before the first, as there is no order of execution) which would essentially do the same things as the first test in order to bring the system back to a state where the second can be executed. So, this would take more time over all.

Do you write independent tests? If so, how do you cope up with execution time?

share|improve this question
add comment

7 Answers 7

up vote 5 down vote accepted

I break my automation up into 2 seperate types of classes. I have 1 class that I use for testing the actual requirements. These tests are independant of any others and broken up into stories. These also double as my test evidence in case of projects that require industry compliance (SOX, PCI, etc). I can't say that I'm overly worried about the execution time of these.

The second class is in my regression suite. Although none of them require a seperate test case to be run before any of them, some of them are deep enough into the web application that they can not be run without the same steps as a previous test (for example, I can't test anything behind a login page if the login functionality does not work). There will be some duplication in these as well, however, it's minimal. Although I don't use the same "test" again inside of it, I do use some of the same functionality while getting it to the required state.

The execution time can be extensive, however, if coded for minimum duplication (ie: happy path to get the system into the correct state), this can be reduced to a minimum.

share|improve this answer
    
So there is a test method for a scenario, then there is another test method for same scenario with so called happy path to attain desired state for another test. This it self looks duplication to me. Am I reading it wrong? –  Tarun May 4 '11 at 6:07
1  
The happy path would be the shortest series of actions in order to get the system to the required state. There is certainly some duplication, but, as I said, keep it to a minimum. When possible, break up the individual actions into methods that aren't in any test itself, but instead called from the test methods allowing more re-use and increase maintainability. When these methods are called by "test A", more methods may be used for additional logging or checks within that test. Hope that makes sense. –  Lyndon Vrooman May 4 '11 at 6:34
    
It does indeed.. –  Tarun May 4 '11 at 6:38
add comment

Yes I also do write tests that are independent of each other since it is the basic principle for testing. Since I use selenium for testing, I usually re-factor my code so that at-least the duplication of code is reduced.

Say for e.g, i have 2 tests to validate a) Login a request b) Update a request. Now since b is dependent of a, I usually write the code in a function and call it from a and b both. Will surely take time, but there can be ways where-in you can execute tests in parallel, in selenium I do it via Selenium Grid. This is helpful when u just want to test one functionality. Hope this helps!!!!

share|improve this answer
    
"basic principle for testing" this statement scares me :-| and how would you approach it if you did not have grid sort of thing at your disposal –  Tarun May 4 '11 at 5:58
    
Selenium grid is just a facilitator for running tests in parallel. Without it, we may still need to run tests serially. –  Ashish May 5 '11 at 16:08
add comment

You may as well use the dependent tests. The main reason to keep them independent would be to prevent one failure from hiding a second failure. If the second test is truly running the first test before executing its actual test scenario, then you don't get that benefit anyway.

share|improve this answer
add comment

Even i do follow the dependent approach provided by TestNG for my Selenium tests as Tarun mentioned. But I never came across any situations where one failure would hide another as Bruce pointed out . I think that it might never happens because every cases that i make dependent on another one will always require the previous test case to successfully run so as to start the second one, else there would hardly be any chance for dependence between test cases or test steps. For example we have two tests, say test T1 which is a login test and T2 which is something to be done after logging in . Let T2 is dependent on T1. Here while I configure my tests to run through TestNG config file , I will call both test T1 and T2 separately as two tests . Ie I need 1. login test and 2. something tat test after logged in So I find my dependent tests very tester friendly and fast enough.

Note: correct me if i am wrong

share|improve this answer
add comment

With open source tools, and virtulisation, scale-out of automated testing is now a lot cheaper so I use that to get around the "takes more execution time" problem.

The next problem is now I am running the same tests twice, how can I be more effective. What I would probably do is try and come up with a scenario where part A of your test now use the same functionality, but test different things or code paths in that space, using different test data for example.

Now that I have more time, and I can apply differerent data, I can make more effective use of automation to cover more and more test scanarios.

I then repeat the cycle and continue to buid more and more tests, as there is always another scenario that you can think of and add to the mix.

share|improve this answer
    
Bruce speaking about using virtulization to scale out the automation, how do u exactly approach it ? Donno whether this question is out of context in this thread, if so please open a new thread and elaborate on this. –  Rakesh Prabhakaran May 10 '11 at 2:42
    
There are three different models I have used, the best one has a central server and web services that the clients call asking for the next test to run, and then pushing results back to the sever. The second is to use something like TFS test agents and pushing the work to the clients that way. The third, low tech solution is to build lists of tests per machine and manualy kick off each machine individuallym then collect and combine the results. The client was mainly WatiN, which has a zero touch install or license fees. –  Bruce McLeod May 10 '11 at 2:58
add comment

I write independent tests so that I have the option of running them in isolation.

If a test depends on another test, it may be because the antecedent test sets something up that the dependent test requires. Some test frameworks (e.g. JUnit) let specify that the specified setup login should run exactly once for a set of tests, no matter how many tests are in the set. You may be able to refactor your tests to leverage that feature.

If the setup process is lengthy, you might want a way for the setup process to persist a "setup finished successfully" flag that your tests can interrogate. For example, if your setup involves populating a database in a certain way, you might want to add a "setup complete" table that contains a row only if the setup was successful.

share|improve this answer
add comment

Failures during the actual test are not the only problem with dependent test cases. You can have a failure during the checking phase that can cause a problem with your dependent testcase, for example. You could also have a sporadic problem that only affects one out of twenty cases for the first test case. If your cases were independent, you would have a 95% chance of still successfully collecting the second result even when the first test case failed. I would also expect independent tests to be easier to repeat multiple times in a row - such as a tester would do when trying to reproduce a sporadic bug - although that might depend on your test framework(s). These arguments might not be adequate where very high amounts of execution time are at stake, but they are additional factors to consider.

Sometimes good product design and testability features can help with execution time, especially when verifying basic page elements (e.g., Button X calls function Y), where the process of getting there and the state along the way is less important than the final step. In web apps, for example, it can be helpful to just navigate to pages via the address bar instead of via the UI, and good design can make that possible in more situations. However, you will still have some end-to-end tests.

The basic approaches I see at that point are prioritization, and parallelization. Prioritization is just running the test cases most likely to find the greatest number of higher-priority bugs frequently, and running the rest of the tests only occasionally. Cases that should be prioritized are those that are most likely to be impacted by current code changes, and those that cover the most ground for their execution time. Parallelization means having multiple machines running the test cases at the same time, and is becoming more possible with virtual machines (which don't work for all programs, but are a great option for many apps using GUIs) and cloud services like Amazon Web Services. In general, having more machines working is usually cheaper than having testers put tons of effort into super-efficient tests (especially when those efforts could be put into finding new bugs instead!).

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.