Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have around 200 test classes for my web application at the moment. I use Selenium2/Webdriver and the tests are written in java, with TestNG used as my framework.

When I run each test by itself, the test will pass 100% of the time.

When I clump any number of tests together into a suite, running the suite will yield random results to my tests. Some will fail one run, pass another, fail another but in a different line of code, and so on.

Is there a known way to ensure the stability of automated tests when run in a TestNG suite?

EDIT: The failures are speed related. Some WebElements will time-out and cannot be found while other times I can see the automation moving more quickly.

Our Setup macro, for example, will fail randomly. It consists of entering a user/pass, clicking a button, selecting a role from a list by clicking on it, and then clicking the continue button. The button becomes enabled once a role is selected. Sometimes the test will try to push the button before it has been enabled. I have not implemented a waitForElement because there are no issues when the test is ran independently. That's just one example.

share|improve this question
This is similar to the question I asked, although I run webdriver C# and NUnit: – MichaelF Oct 3 '12 at 19:26
Ah, I agree it is definitely similar. My main concern is that the goal is to eventually integrate the tests into our nightly build, and if a test fails, the build will not be made. For that reason, it is critical we have 100% pass rates for tests that should pass – squeemish Oct 3 '12 at 19:33
I feel your pain, eventually I want to run this framework with my builds as well and have scripts set that can run various permutations but I don't want build failures because of a sporadic test failure – MichaelF Oct 3 '12 at 19:57
You should definitely implement waitForElement any time you're interacting with the first element after a page load or even after any javascript execution changes the DOM. Personally, my abstraction layer automatically waits for an element to exist on EVERY interaction with an element. Timing issues can be hard to anticipate and can be difficult to track down and this removes it from the equation. – Sam Woods Oct 3 '12 at 22:09
Good advice. I strayed from using that since it would sometimes increase the time a test took to run. I will probably go and add those in – squeemish Oct 4 '12 at 12:48
up vote 11 down vote accepted

I suspect this is not a TestNG issue. I would start by trying to produce a minimal set of tests that, when run together, cause failures. After that, I would explore these possibilities:

  • Inter-test interaction changes test-application interaction: It is possible that interaction between your tests changes how your test interactions with your application. A way to check this is to add logging your tests, and then compare the logged data when a test is in isolation to the log data when the test is run with other tests.

  • Inter-test interaction violates test's preconditions: Each test assumes some preconditions. If the preconditions are not met, the test fails. Some preconditions will be explicit (preconditions the test checks for beforehand or sets up on its own); others will be implicit (preconditions the test assumes or that the author forgot to think about). Usually it is the implicit ones that give you problems.

  • Inter-test interaction changes how your test processes its result: It is possible that interaction between your tests causes bugs in how a test analyzes its interactions with the web application.

  • Inter-test interaction uncovers application bugs: Of course, it is always possible that you found an actual application bug.

  • Inter-test interaction causes resource problems: It is possible that running many tests causes you to consume too much of a resource.

share|improve this answer
Resource problems may be it. The speed at which the tests are run varies from each execution. For each test we start and then kill a Firefox driver, so I do not think anything inter-test would cause this – squeemish Oct 3 '12 at 19:35

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.