Can I force a Jenkins job to report a successful TestNG test even when the suite has failures?
I have a test suite with 100 tests and I want to report a success in Jenkins if there are 3 or fewer errors. What is the best way to do this? Here are my ideas so far:
Use the Jenkins "Groovy post build plugin" to force the button green after the build step already reported yellow.
Run the tests in a Jenkins pre-build step and then use the TestNG XmlResult class to load the test results in the regular build step, count the fails in the result, and then return a 0 or 1 exit code?
Use the Jenkins Build-Flow plugin to ask all 100 tests to re-run if 1 or more fails.
I am not sure of the best way to do this and I am wondering if anyone has a better idea.
My workaround for this for now is to divide my test suites into 3 separate suites that run in a fixed order: first, a point-suite with 1 test, then a smoke-suite with a handful of tests, then the full regression as long as the smoke-suite passed.