Can I force a Jenkins job to report a successful TestNG test even when the suite has failures?

I have a test suite with 100 tests and I want to report a success in Jenkins if there are 3 or fewer errors. What is the best way to do this? Here are my ideas so far:

  1. Use the Jenkins "Groovy post build plugin" to force the button green after the build step already reported yellow.

  2. Run the tests in a Jenkins pre-build step and then use the TestNG XmlResult class to load the test results in the regular build step, count the fails in the result, and then return a 0 or 1 exit code?

  3. Use the Jenkins Build-Flow plugin to ask all 100 tests to re-run if 1 or more fails.

I am not sure of the best way to do this and I am wondering if anyone has a better idea.

My workaround for this for now is to divide my test suites into 3 separate suites that run in a fixed order: first, a point-suite with 1 test, then a smoke-suite with a handful of tests, then the full regression as long as the smoke-suite passed.

  • 1
    Why do you want this? Typically a failed test means a problem with the product under test or a problem with the test suite. Either way, you would want to be alerted to this, wouldn't you? – Yamikuronue Jul 25 '14 at 16:32
  • I would experiment with TestNG listeners, trying to manipulate test results in case tests are failing. – dzieciou Nov 18 '15 at 6:07

You can make use of maven-failsafe-plugin instead sure-fire to do so just put below code in your pom.xml and comment out surefire plugin from pom.xml along with dependency.

what will happen like if your testng Suite is having more than one Test with testcases or classes and if any testcase will fail from any of the Test then also ur test execution will continue untill all the testcases are executed that are present in ur testng Suite and Build will be marked as unstable instead of failed.

only one thing you have to take care is u run it as "mvn verify"

Below is pom.xml snippet -

| improve this answer | |
  • 1
    This will not report a success in the case of 3, 2, or 1 failing tests. It will report "unstable". – Yamikuronue Jul 25 '14 at 16:30
  • Yes it will report build as unstable but it is useful in case you are having multiple groups for execution so even though your case will fail but your test execution will be continue for all the cases. – Karan Thakur Jul 28 '14 at 17:40

You might want to take a look at TestNG Jenkins plugin. Although it does not currently fix your problem, you will find a group of people facing the same situation as you discussing an issue. You might chime in and help develop/test this fix of the plugin.

| improve this answer | |
  • Yes, I have seen that before but I get the impression that after all the work he did, it won't ever actually be implemented/merged. One complication that arises from this kind of thing is the question of how you report a threshold of failed tests on a "results report". I understand why Cedric, on the TestNG team might think this isn't really doable. – djangofan Jun 16 '14 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.