14

I have around 200 test classes for my web application at the moment. I use Selenium2/Webdriver and the tests are written in java, with TestNG used as my framework.

When I run each test by itself, the test will pass 100% of the time.

When I clump any number of tests together into a suite, running the suite will yield random results to my tests. Some will fail one run, pass another, fail another but in a different line of code, and so on.

Is there a known way to ensure the stability of automated tests when run in a TestNG suite?

EDIT: The failures are speed related. Some WebElements will time-out and cannot be found while other times I can see the automation moving more quickly.

Our Setup macro, for example, will fail randomly. It consists of entering a user/pass, clicking a button, selecting a role from a list by clicking on it, and then clicking the continue button. The button becomes enabled once a role is selected. Sometimes the test will try to push the button before it has been enabled. I have not implemented a waitForElement because there are no issues when the test is ran independently. That's just one example.

9
  • This is similar to the question I asked, although I run webdriver C# and NUnit: sqa.stackexchange.com/q/3581/18
    – MichaelF
    Oct 3, 2012 at 19:26
  • Ah, I agree it is definitely similar. My main concern is that the goal is to eventually integrate the tests into our nightly build, and if a test fails, the build will not be made. For that reason, it is critical we have 100% pass rates for tests that should pass
    – squeemish
    Oct 3, 2012 at 19:33
  • 1
    I feel your pain, eventually I want to run this framework with my builds as well and have scripts set that can run various permutations but I don't want build failures because of a sporadic test failure
    – MichaelF
    Oct 3, 2012 at 19:57
  • 9
    You should definitely implement waitForElement any time you're interacting with the first element after a page load or even after any javascript execution changes the DOM. Personally, my abstraction layer automatically waits for an element to exist on EVERY interaction with an element. Timing issues can be hard to anticipate and can be difficult to track down and this removes it from the equation.
    – Sam Woods
    Oct 3, 2012 at 22:09
  • Good advice. I strayed from using that since it would sometimes increase the time a test took to run. I will probably go and add those in
    – squeemish
    Oct 4, 2012 at 12:48

3 Answers 3

15

I suspect this is not a TestNG issue. I would start by trying to produce a minimal set of tests that, when run together, cause failures. After that, I would explore these possibilities:

  • Inter-test interaction changes test-application interaction: It is possible that interaction between your tests changes how your test interactions with your application. A way to check this is to add logging your tests, and then compare the logged data when a test is in isolation to the log data when the test is run with other tests.

  • Inter-test interaction violates test's preconditions: Each test assumes some preconditions. If the preconditions are not met, the test fails. Some preconditions will be explicit (preconditions the test checks for beforehand or sets up on its own); others will be implicit (preconditions the test assumes or that the author forgot to think about). Usually it is the implicit ones that give you problems.

  • Inter-test interaction changes how your test processes its result: It is possible that interaction between your tests causes bugs in how a test analyzes its interactions with the web application.

  • Inter-test interaction uncovers application bugs: Of course, it is always possible that you found an actual application bug.

  • Inter-test interaction causes resource problems: It is possible that running many tests causes you to consume too much of a resource.

1
  • Resource problems may be it. The speed at which the tests are run varies from each execution. For each test we start and then kill a Firefox driver, so I do not think anything inter-test would cause this
    – squeemish
    Oct 3, 2012 at 19:35
2

Ended up being a TestNG issue. Interleaving methods are a known issue. Breaking the tests into suites and running all suites from one XML alleviates most of the issues.

Problems can also arise if tests that have a 'dependsOnMethods' dependency have the same method name in multiple classes. That can cause the method in Class B to fire if the dependent method in Class A passed.

0

Generally, test failures in batch happens when tests leave/or find the application in an unexpected state

This may manifest in multiple ways. Sometimes its data, sometimes its cache issues, sometimes UI objects state and many more...

The way we generally resolve these issues is to ensure application base state as the initial step in the test in the form of assertions so that we can fail the test right away if the application is in an undesired state as per the pre-condition of the current test case.

For example, if a previous test cancels an active order, which will be used in the next test so in the next test, as a first step we will assert that the order which going to be used is in an active state.

So what we will do is something like we fire an SQL query to pick an active order every time that test runs with, so in any case, we get the desired data for the test dynamically without any side effects of the previous test on the application data.

Or if there is no active order data anymore after the previous test run, the current test will fail right away with 'NO DATA FOUND' message to clarify the failure for easy test analysis.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.