Dependent methods in the same class run out of order only when included in a “testng suite”; running class independently respects the dependency

Question

I have a class with dependent methods using the TestNG annotation

dependsOnMethods

The test runs fine 100% of the time if I simply run it as a TestNG Test from the package.

When I include the test in a TestNG Suite, the methods run out of order. Yes, I am using:

<test name="Test" preserve-order="true">

in my .xml file.

Every other test in the suite respects the method ordering and run without issue. Is there any known information on why this might be occurring?

Test code:

@Test(groups={ "Administration"})
public class RoleCrudTest extends AbstractIntegrationTest
{
protected static SeleniumActionHelper action;

    @Test
    public void inactiveRole() throws Exception
    {
        SeleniumHelper helper = new SeleniumHelper();
        action = new SeleniumActionHelper(driver);

        helper.login();

        String roleUrl = navigateToUrl("role/roles.xhtml");
        driver.get(roleUrl);

        assertEquals("Role:", findElementBySelector("span.portletButtonHeader").getText());

        WebElement roleName = findElementById("roleName");
        assertFalse(roleName.isEnabled());

        WebElement deptId = findElementById("deptid");
        assertFalse(deptId.isEnabled());
    }

    @Test(dependsOnMethods = "inactiveRole")
    public void createRole() throws Exception
    {
        WebElement addButton = findElementById("add");
        addButton.click();

        waitUntilAjaxRequestCompletes();

        WebElement roleName = findElementById("roleName");
        roleName.click();
        roleName.sendKeys("AAAAAAAA");

        WebElement deptId = findElementByXpath("(//button[@type='button'])[3]");
        deptId.click();

        WebElement dept = findElementByXpath("//div[@id='department_panel']/ul/li[2]");
        dept.click();

        WebElement checkbox = findElementByXpath("//li[@id='privileges:1']/div/span/div/div");
        checkbox.click();

        Thread.sleep(1000);

        WebElement save = findElementById("save");
        save.click();

        assertEquals("Role saved successfully", findElementBySelector("div.ui-growl-message > p").getText());
    }

    @Test(dependsOnMethods = "createRole")
    public void editUndo() throws Exception
    {
        Thread.sleep(1000);

        WebElement tableSort = findElementByXpath("//th[@id='tableSection:rolesListWrapped:j_idt85']/div/span[2]");
        tableSort.click();

        Thread.sleep(1000);

        WebElement createdRole = findElementByXpath("//tbody[@id='tableSection:rolesListWrapped_data']/tr[1]/td/div");
        createdRole.click();

        Thread.sleep(1000);

        WebElement roleName = findElementById("roleName");
        roleName.click();
        roleName.clear();
        roleName.sendKeys("edited");

        WebElement deptId = findElementByXpath("(//button[@type='button'])[3]");
        deptId.click();

        WebElement dept = findElementByXpath("//div[@id='department_panel']/ul/li[3]");
        dept.click();

        WebElement checkbox = findElementByXpath("//li[@id='privileges:1']/div/span/div/div/span");
        checkbox.click();

        WebElement checkbox2 = findElementByXpath("//li[@id='privileges:0']/div/span/div/div");
        checkbox2.click();

        Thread.sleep(1000);

        WebElement undo = findElementById("cancel");
        undo.click();

        String text = findElementById("roleName").getAttribute("value");
        String oldtext = "AAAAAAAA";

        assertTrue(text.equals(oldtext));
    }

    @Test(dependsOnMethods = "editUndo")
    public void editRole() throws Exception
    {
        Thread.sleep(1000);

        WebElement createdRole = findElementByXpath("//tbody[@id='tableSection:rolesListWrapped_data']/tr[1]/td/div");
        createdRole.click();

        Thread.sleep(1000);

        WebElement roleName = findElementById("roleName");
        roleName.click();
        roleName.clear();
        roleName.sendKeys("AAAAAAAAedited");

        WebElement deptId = findElementByXpath("(//button[@type='button'])[3]");
        deptId.click();

        WebElement dept = findElementByXpath("//div[@id='department_panel']/ul/li[3]");
        dept.click();

        WebElement checkbox = findElementByXpath("//li[@id='privileges:1']/div/span/div/div/span");
        checkbox.click();

        WebElement checkbox2 = findElementByXpath("//li[@id='privileges:0']/div/span/div/div");
        checkbox2.click();

        Thread.sleep(1000);

        WebElement save = findElementById("save");
        save.click();

        assertEquals("Role saved successfully", findElementBySelector("div.ui-growl-message > p").getText());
    }

    @Test(dependsOnMethods = "editRole")
    public void deleteRole() throws Exception
    {
        Thread.sleep(1000);

        WebElement deleteButton = findElementById("tableSection:delete");
        deleteButton.click();

        WebElement deleteConfirm = findElementById("confirmDelete:yes");
        deleteConfirm.click();

        Thread.sleep(500);

        assertEquals("Role deleted successfully", findElementBySelector("div.ui-growl-message > p").getText());

        waitUntilAjaxRequestCompletes();
    }
}

And here is the entire TestNG XML file (please note that this is a simple one with only 3 classes, the behavior is the same if I run an XML with 20 classes). No matter the amount of classes or ordering, the roleCrud test always runs the methods out of order, every other class runs fine:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite" parallel="none">
  <test name="Test" preserve-order="true">
    <classes>
      <class name="com.x.c4i.view.security.RoleCreateErrorCheckTest"/>
      <class name="com.x.c4i.view.security.GroupCrudTest"/>
      <class name="com.x.c4i.view.security.RoleCrudTest"/>
    </classes>
  </test> <!-- Test -->
</suite> <!-- Suite -->

Answer 1

Per the resolution of the GitHub issue yurloc filed, it appears that using group-by-instances will prevent tests from being interleaved when using dependsOnMethods.