3

Can I force a Jenkins job to report a successful TestNG test even when the suite has failures?

I have a test suite with 100 tests and I want to report a success in Jenkins if there are 3 or fewer errors. What is the best way to do this? Here are my ideas so far:

  1. Use the Jenkins "Groovy post build plugin" to force the button green after the build step already reported yellow.

  2. Run the tests in a Jenkins pre-build step and then use the TestNG XmlResult class to load the test results in the regular build step, count the fails in the result, and then return a 0 or 1 exit code?

  3. Use the Jenkins Build-Flow plugin to ask all 100 tests to re-run if 1 or more fails.

I am not sure of the best way to do this and I am wondering if anyone has a better idea.

My workaround for this for now is to divide my test suites into 3 separate suites that run in a fixed order: first, a point-suite with 1 test, then a smoke-suite with a handful of tests, then the full regression as long as the smoke-suite passed.

Improve this question
2
  • 1
    Why do you want this? Typically a failed test means a problem with the product under test or a problem with the test suite. Either way, you would want to be alerted to this, wouldn't you?
    – Yamikuronue
    Jul 25, 2014 at 16:32
  • I would experiment with TestNG listeners, trying to manipulate test results in case tests are failing.
    – dzieciou
    Nov 18, 2015 at 6:07

2 Answers 2

Reset to default
1

You can make use of maven-failsafe-plugin instead sure-fire to do so just put below code in your pom.xml and comment out surefire plugin from pom.xml along with dependency.

what will happen like if your testng Suite is having more than one Test with testcases or classes and if any testcase will fail from any of the Test then also ur test execution will continue untill all the testcases are executed that are present in ur testng Suite and Build will be marked as unstable instead of failed.

only one thing you have to take care is u run it as "mvn verify"

Below is pom.xml snippet -

    <plugin>  
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.11</version>
    <configuration>
<suiteXmlFiles>
<suiteXmlFile>testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
    <executions>
      <execution>
        <goals>
          <goal>integration-test</goal>
          <goal>verify</goal>
        </goals>
      </execution>
    </executions>
  </plugin>
Improve this answer
2
  • 1
    This will not report a success in the case of 3, 2, or 1 failing tests. It will report "unstable".
    – Yamikuronue
    Jul 25, 2014 at 16:30
  • Yes it will report build as unstable but it is useful in case you are having multiple groups for execution so even though your case will fail but your test execution will be continue for all the cases.
    – Karan Thakur
    Jul 28, 2014 at 17:40
1

You might want to take a look at TestNG Jenkins plugin. Although it does not currently fix your problem, you will find a group of people facing the same situation as you discussing an issue. You might chime in and help develop/test this fix of the plugin.

Improve this answer
1
  • Yes, I have seen that before but I get the impression that after all the work he did, it won't ever actually be implemented/merged. One complication that arises from this kind of thing is the question of how you report a threshold of failed tests on a "results report". I understand why Cedric, on the TestNG team might think this isn't really doable.
    – djangofan
    Jun 16, 2014 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.