How to make my tests more robust in WebDriver using JUnit (code included)

Question

I'm hoping someone can advise me on some best practices to make my tests more robust.

The problem I'm having is that every so often my tests will fail at the point of WebDriver waiting for an expected condition, e.g. new WebDriverWait(chrome, 10).until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("ul.componentList.j-componentList"))); and I know the element is in the DOM as it will pass more often that not.

So if I was to run this overnight, it could fail when it would pass 9 times out of 10. But the point Im making is that my code seems a bit weak in places and not as robust as it could be. Where am I going wrong?

I've pasted a small snippet of my code to ask where I can make it better so that WebDriver waits and finds the elements before timing out.

You might notice I have my tests in alphabetical order prefixed with a,b,c, etc. Im using the new annotation type FixMethodOrder, just in case anyone asks why I have them alphabetical.

Any constructive advice is most welcome.

@BeforeClass
public static void launchBrowser(){
    String currentDir = System.getProperty("user.dir");
    String chromeDriverLocation = currentDir + "/../../tools/chromedriver/chromedriver.exe";
    System.setProperty("webdriver.chrome.driver", chromeDriverLocation);

    chrome = new ChromeDriver();
}

@Test
public void aLogin(){
    chrome.manage().window().maximize();
    chrome.navigate().to("http://privateweb.com/thisisfordemopurposes");

    new WebDriverWait(chrome, 10).until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("input[type='text'][id='usernameInput']")));

    WebElement usernameField = chrome.findElement(By.cssSelector("input[type='text'][id='usernameInput']"));
    usernameField.sendKeys("fake");

    WebElement passwordField = chrome.findElement(By.cssSelector("input[id='passwordInput']"));
    passwordField.sendKeys("fake");

    chrome.findElement(By.cssSelector("input#login")).click();
}

@Test
public void bSelectBlankProject(){
    new WebDriverWait(chrome, 10).until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//*[@id=\"templateGrid\"]/li[2]/img[1]")));

    WebElement item1 = chrome.findElement(By.xpath("//*[@id=\"templateGrid\"]/li[2]/img[1]"));
    WebElement item2 = chrome.findElement(By.xpath("//*[@id=\"templateGrid\"]/li[2]/img[2]"));
    WebElement item3 = chrome.findElement(By.xpath("//*[@id=\"templateGrid\"]/li[2]/header/span"));
    WebElement item4 = chrome.findElement(By.xpath("//*[@id=\"templateGrid\"]/li[2]"));

    Actions click = new Actions(chrome);
    click.moveToElement(item1).moveToElement(item2).moveToElement(item3).moveToElement(item4).click().build().perform();

    System.out.println("Blank Project has been selected");
}

@Test
public void cDragCloseButtonOnToTheCanvas(){

    new WebDriverWait(chrome, 10).until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("ul.componentList.j-componentList")));

    WebElement listContainerClose = chrome.findElement(By.xpath("//*[@id=\"componentsDiv\"]/div/div/div"));
    WebElement componentListClose = chrome.findElement(By.xpath("//*[@id=\"componentsDiv\"]/div/div/div/ul"));
    WebElement closeButton = chrome.findElement(By.xpath("//*[@id=\"componentsDiv\"]/div/div/div/ul/li[2]"));
    WebElement componentThumb = chrome.findElement(By.xpath("//*[@id=\"componentsDiv\"]/div/div/div/ul/li[2]/div[1]"));
    WebElement imageHolderCloseButton = chrome.findElement(By.xpath("//*[@id=\"componentsDiv\"]/div/div/div/ul/li[2]/div[1]/div"));
    WebElement componentNameCloseButton = chrome.findElement(By.xpath("//*[@id=\"componentsDiv\"]/div/div/div/ul/li[2]/div[2]"));

    WebElement canvas = chrome.findElement(By.cssSelector("div#page-c3"));

    Actions dragAndDrop = new Actions(chrome);

    dragAndDrop.clickAndHold(closeButton)
            .moveToElement(listContainerClose)
            .moveToElement(componentListClose)
            .moveToElement(componentThumb)
            .moveToElement(imageHolderCloseButton)
            .moveToElement(componentNameCloseButton)
            .moveToElement(canvas)
            .release().perform();

    WebElement draggableCloseButton = chrome.findElement(By.cssSelector("div.t-componentImg-A.component.closeButtonComponent.draggableComponent.ui-draggable"));

    Assert.assertEquals("closeButton", draggableCloseButton.getAttribute("data-type"));
}
 @Test
public void dRenamingCloseButtonComponent(){
    WebElement name = chrome.findElement(By.cssSelector("input.t-textInput-A[name=\"name\"]"));

    new WebDriverWait(chrome, 5).until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("input.t-textInput-A[name=\"name\"]")));

    String selectAll = Keys.chord(Keys.CONTROL, "a");

    name.sendKeys(selectAll);
    name.clear();
    name.sendKeys("Close me");
    Assert.assertEquals("Close me", name.getAttribute("value"));
    System.out.println("Renamed the close button to something else");
} 
@Test
public void eMoveComponent(){
    String selectAll = Keys.chord(Keys.CONTROL, "a");

    new WebDriverWait(chrome, 5).until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("input.t-textInput-A.number[name=\"x\"]")));

    WebElement xAxis = chrome.findElement(By.cssSelector("input.t-textInput-A.number[name=\"x\"]"));
    xAxis.sendKeys(selectAll);
    xAxis.clear();
    xAxis.sendKeys("240");
    Assert.assertEquals("240",xAxis.getAttribute("value") );

    new WebDriverWait(chrome, 5).until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("input.t-textInput-A.number[name=\"y\"]")));

    WebElement yAxis = chrome.findElement(By.cssSelector("input.t-textInput-A.number[name=\"y\"]"));
    yAxis.sendKeys(selectAll);
    yAxis.clear();
    yAxis.sendKeys("10");
    Assert.assertEquals("10", yAxis.getAttribute("value"));
}

EDIT 1: Comments in relation to @ErkiM answer.

I have just run one of my classes and received a failed test.

@Test
public void jGoToPage(){

    new WebDriverWait(chrome, 5).until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("select.t-select-A[name=\"page\"]")));
    List<WebElement> option = chrome.findElements(By.cssSelector("select.t-select-A[name=\"page\"]"));

    option.get(0).click();
    option.get(1).click();
}

The stacktrace I receive is as follows:

org.openqa.selenium.TimeoutException: Timed out after 5 seconds waiting for visibility of element located by By.selector: select.t-select-A[name="page"]
Build info: version: '2.34.0', revision: '11cd0ef93615408e0b6b3bfa28defe125906461a', time: '2013-08-06 11:43:14'
System info: os.name: 'Windows 7', os.arch: 'amd64', os.version: '6.1', java.version: '1.7.0_45'
Driver info: driver.version: unknown
at org.openqa.selenium.support.ui.FluentWait.timeoutException(FluentWait.java:259)
at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:228)
at AdBuilderComponents.CloseButtonPropertiesPanel.jGoToPage(CloseButtonPropertiesPanel.java:308)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:202)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: org.openqa.selenium.NoSuchElementException: no such element
(Session info: chrome=31.0.1650.63)
(Driver info: chromedriver=2.6.232923,platform=Windows NT 6.1 SP1 x86_64)  (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 2.07 seconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
Build info: version: '2.34.0', revision: '11cd0ef93615408e0b6b3bfa28defe125906461a', time: '2013-08-06 11:43:14'
System info: os.name: 'Windows 7', os.arch: 'amd64', os.version: '6.1', java.version: '1.7.0_45'
Session ID: c211c5ddfe42b768b3bf697b80ce1508
Driver info: org.openqa.selenium.chrome.ChromeDriver

Answer 1

Hm. Did I understand correctly that you are indeed receiving a timeoutexception? Could you provide any stacktrace?

"I know the element is in the DOM as it will pass more often that not.

So if I was to run this overnight, it could fail when it would pass 9 times out of 10."

This statement is not good enough. The best you can say is that you can be sure, that the element was there for at least 9 times out of ten.

So whats happening in visibilityOfElementLocated:

  /**
   * An expectation for checking that an element is present on the DOM of a page
   * and visible. Visibility means that the element is not only displayed but
   * also has a height and width that is greater than 0.
   *
   * @param locator used to find the element
   * @return the WebElement once it is located and visible
   */
  public static ExpectedCondition<WebElement> visibilityOfElementLocated(
      final By locator) {
    return new ExpectedCondition<WebElement>() {
      @Override
      public WebElement apply(WebDriver driver) {
        try {
          return elementIfVisible(findElement(locator, driver));
        } catch (StaleElementReferenceException e) {
          return null;
        }
      }
     // skipped some sht
  }

  /**
   * @return the given element if it is visible and has non-zero size, otherwise
   *         null.
   */
  private static WebElement elementIfVisible(WebElement element) {
    return element.isDisplayed() ? element : null;
  }

So to sum it up, this code checks if an element is present in DOM and its height and width are > 0, if so, it will return the WebElement, otherwise null. It is straightforward, if you are getting a timeout, you can be pretty sure the element is not in the DOM.

Anyway, what is happening before this:

ExpectedConditions.visibilityOfElementLocated(By.cssSelector("ul.componentList.j-componentList")));

some databinding? other stuff that could perhaps cause the staleness of the webelement? Try this piece of code:

new FluentWait<WebDriver>(driver)
        .withTimeout(60, TimeUnit.SECONDS)
        .pollingEvery(1, TimeUnit.SECONDS)
        .ignoring(NoSuchElementException.class)
        .ignoring(StaleElementReferenceException.class)
        .until(new Function<WebDriver, Boolean>() {
            @NotNull
            @Override
            public Boolean apply(WebDriver webDriver) {
                WebElement element = driver.findElement(yourElement);
                return element != null && element.isDisplayed();
            }
        });

This is definetly overly verbose, but i wrote it all out to make it more understandable. If it doesn't help you, i suggest adding more logging to better understand what is going on there. Perhaps 10 sec is not enough some times? Add longer timeout? Perhaps get a screenshot if the test is failing?

As you are using JUnit to execute your test and you were also talking about general robustness, perhaps implement a TestWatcher? Something like:

public class TestRules extends TestWatcher {

    @Override
    protected void failed(Throwable e, Description description) {
        // This will be called whenever a test fails.
        // Take screenshot, log error, go hot wild.
    }

And in your test class simply call it:

public class testClass{

@Rule
public TestRules testRules = new TestRules();

@Test
public void doTestSomething() throws Exception{
    // If the test fails for any reason, it will be caught by testrules.
}

---

EDIT: Also, as robustness was mentioned - when you run into maintenance nightmare, look into this article: http://code.google.com/p/selenium/wiki/PageObjects

EDIT2: I looked thru the sample code again and i have been there. this spaghetti code will be a maintenance nightmare in no time. Seriously consider any test architecture.

Answer 2

I advice you to keep your tests simple and they will be more robust.
You should specialize your test:

• UI components (Verify the display of your page)
  
• UI actions (Verify button are working as expected for example)
  
• Workflows
  

More you are verifying things in your test and more the portability it fails will increased.
Don't hesitate to use @BeforeClass and @BeforeTest to keep your code more clear. After your test don't forget to use @AfterClass or @AfterTest if you need to close a database connection or driver.

You can use Selenium IDE to generate your JUnit code if you want to see others ways to make your test.

Answer 3

public void waitForElementPresent(By locator) {
    try{
        WebDriverWait wait=new WebDriverWait(driver,30);
        wait.until(ExpectedConditions.visibilityOfElementLocated(locator));

    }catch(TimeoutException timeout){
        for (int i = 0; i <= 30; i++) {
            try {
                boolean exists=false;
                driver.manage().timeouts().implicitlyWait(2, TimeUnit.SECONDS);
                driver.findElement(locator);
                exists=true;
                if (exists) {
                    break;
                } 
                driver.manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);
            } catch (NoSuchElementException e) {
                // TODO Auto-generated catch block
                if(i==30){
                    throw e;
                }
            }
        }
    }
}

I hope this helps...

Answer 4

Some of the issues that is causing these tests to be more fragile:

1. Use the Page Object Model
2. Each test should be stand-alone (yours requires the previous test to pass).
3. Test only 1 action per test

The primary issue I am seeing here are with the first 2 mentioned. By not using the Page Object Model it is difficult to determine what exactly is happening and you will constantly run into these issues as you are maintaining the tests. By requiring the 2nd test to succeed for the 3rd test to work you've essentially just made 1 large test. Break down each test into completely self-reliant tests. This will be much easier to do with the Page Object Model.

Answer 5

I had exactly the same issue some time ago. I was waiting for content of gxt window to be loaded and many times I failed because of timeout exception. Normally window was loading content about max 5 sec, but with WebDriver even 30 minutes was not enough. What I found was that timeout happened because of WebDriver was causing DOM not to be properly loaded (freezing). As a workaround I have found that simple window re-size is enough. Probably your problem was already solved, but i put this to others that can face the same issue in the future.

Below there is method code to make your code cleaner.

I'm always putting commonly used methods in some special class to not duplicate code lines.

public static void browserShake(WebDriver driver, int width, int height) {
    driver.manage().window().setSize(new Dimension(width+50, height));
    driver.manage().window().setSize(new Dimension(width, height));
}

Answer 6

You can move to testng and use dependsOnMethods. You will have to change few things in your code to make it compatible.

It makes sure your tests in the same class are executed in the good order. In the following example "secondTest" will start only when "firstTest" is done.

@Test
public void firstTest() {
    //code
}
@Test (dependsOnMethods = { "firstTest" })
public void secondTest() {
    //code
}