2

How to automatically rerun failed TestNG tests on Jenkins and update the final results to pass if retries passes?

Description: I have two tests that ran. Test1 passed, but Test2 failed then on retry it passed.

Actual Results: Tests run: 1, Failures: 1, Errors: 0, Skipped: 0.

Expected Results; Tests run: 2, Failures: 0, Errors: 0, Skipped: 0

or

Tests run: 2, Failures: 0, Errors: 0, Skipped: 1 (Skipped for retry tests)

    public class TestRetry implements IRetryAnalyzer {
    private static int MAX_RETRY_COUNT = 1; 
    //Only works for value of 1; Anything more than that causes 
    infinite retries for the same test even when it passes.

    AtomicInteger count = new AtomicInteger(MAX_RETRY_COUNT);

    public boolean isRetryAvailable() {
        return (count.intValue() > 0);
    }

    public boolean retry(ITestResult result) {
        boolean retry = false;
        if (isRetryAvailable()) {
            Reporter.log("Going to retry test case: " + result.getMethod() + ", " + (MAX_RETRY_COUNT - count.intValue() + 1) + " out of " + MAX_RETRY_COUNT, true);
            retry = true;
            count.decrementAndGet();
        }
        return retry;
    }
}

  public class TestListenerSupport extends TestListenerAdapter {
  TestUtilities testUtil = new TestUtilities();

@Override
public void onTestStart(ITestResult result){
    //Handle events before test start
    //Obtain specific annotation info from test methods
    Stable stable = 
  result.getMethod().getConstructorOrMethod().getMethod()
  .getAnnotation(Stable.class);
   Maturing maturing = 
  result.getMethod().getConstructorOrMethod().getMethod()
  .getAnnotation(Maturing.class);
   Preflight preFlight = 
   result.getMethod().getConstructorOrMethod().getMethod()
   .getAnnotation(Preflight.class);
      if (stable != null){
           Reporter.log("Annotation is set to STABLE! for '" + 
    result.getMethod().getMethodName() + "'.", true);
      }
      if (maturing != null){
           Reporter.log("Annotation is set to MATURING! for '" + 
    result.getMethod().getMethodName() + "'.", true);
     }
      if (preFlight != null){
           Reporter.log("Annotation is set to PREFLIGHT! for '" + 
    result.getMethod().getMethodName() + "'.", true);
    }
}


  @Override
  public void onTestFailure(ITestResult result){

      if (result.getMethod().getRetryAnalyzer() != null) {
            TestRetry retryAnalyzer = 
     (TestRetry)result.getMethod().getRetryAnalyzer();

            if(retryAnalyzer.isRetryAvailable()) {
                result.setStatus(ITestResult.SKIP);
            } else {
                result.setStatus(ITestResult.FAILURE);
            }

            Reporter.setCurrentTestResult(result);
        }
}

@Override
public void onTestSkipped(ITestResult result){

    Reporter.log("onTestSkipped invoked! " + result.getThrowable().toString(),true);
    Reporter.log("onTestSkipped Exception Message: " + result.getThrowable().getMessage(),true);
    Reporter.log("onTestSkipped Exception Cause: " + result.getThrowable().getCause(),true);
    Reporter.log("onTestSkipped Exception StackTrace: " + result.getThrowable().getStackTrace(), true);
    AppiumDriver driver = getDriverInstance(result);
    if(driver!=null){
        TestUtilities testUtil = new TestUtilities();
        try {
            testUtil.getScreenShot(driver, result.getMethod().getMethodName() + "_Skipped");
        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        Reporter.log("Killing appium driver instance",true);
        driver.quit();
    }
    else
        Reporter.log("onTestSkipped - null driver detected", true);
}

@Override
public void onTestSuccess(ITestResult result){
    Map map = new HashMap();
    //get the testcase IDs from input xml file used for creating a test run in TestRail
    Map<String,String> test_case_id = result.getMethod().getXmlTest().getAllParameters();
    String test_name = result.getMethod().getMethodName();
    testUtil.writeResult("1", test_case_id.get(test_name));

}
1
  • I am not sure this is a good practice in testing. If your test does not pass from the first time then there is some issue in your app-under-test or with environment. This should not be addressed on test level.
    – Alexey R.
    Aug 13, 2018 at 14:02

2 Answers 2

1

TestNG provides the IRetryAnalyzer interface, which you can implement like this:

public class MyRetryAnalyzer implements IRetryAnalyzer {

    private AtomicInteger retries = new AtomicInteger(3);

    @Override
    public boolean retry(ITestResult result) {
        return retries.getAndDecrement() > 0;
    }

}

Afterwards, you can use it as follows:

@Test(retryAnalyzer = MyRetryAnalyzer.class)
public void myTest(){
    // ...
}

However, flaky tests are evil—and so is the excessive use of retry rules. Have a look at this related question:

How to deal with flaky tests that have intermittent failures?

1
  • I did something similiar, but the retries that are passed are still counting towards a build failure and the jenkins marked it red instead of green. I would like the retries to be considered pass and the build to pass and to have jenkins mark it green. Let me post my code above in original post. Thanks. Aug 13, 2018 at 15:54
1

I adjusted my code with what Nick mentioned in here https://stackoverflow.com/questions/11622365/testng-how-to-only-output-one-pass-fail-result-when-test-is-rerun

and it worked.

Thanks.

public class RetryAnalyzer implements IRetryAnalyzer {
private int count = 0;
private int maxCount = 2;

@Override
public boolean retry(ITestResult result) {
if (!result.isSuccess()) {
  if (count < maxCount) {
    count++;
    result.setStatus(ITestResult.SUCCESS);
    String message = Thread.currentThread().getName() +
    ": Error in " + result.getName() + " Retrying "
        + (maxCount + 1 - count) + " more times";
    System.out.println(message);
    Reporter.log(message);
    return true;
  } else {
    result.setStatus(ITestResult.FAILURE);
  }
}
return false;
}
}
1
  • Would you mind accepting your own answer? Then others know that this problem has been solved.
    – beatngu13
    Aug 15, 2018 at 5:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.