2

Number one rule with continuous integration (CI) server is to have build always green and when a test fails fix the system under test as quickly as possible.

However, we have currently end-to-end and integration tests on CI server that fails because of known bugs. We are aware of those bugs but they are minor and they have less priority to be fixed immediately. Consequently, build is no longer green and if new bug will be introduced we will have no such a good feedback on that.

Our first solution is to disable such failing tests, keep in the test code a link to a bug (JIRA issues) and keep in JIRA a link back to the test. This obviously have some shortcomings such as the need to track and remember about such disabled tests. Also, disabled tests will no discover other bugs.

What would be your approach for such a situation?

Improve this question
4
  • I think that's a good, practical solution. I would do the same thing.
    – user246
    Commented Aug 7, 2014 at 12:32
  • 1
    And what would you do if you have one test where only one of 15 soft assertions fails? Disabling whole test is like throwing out the baby with the bath water.
    – dzieciou
    Commented Aug 7, 2014 at 13:02
  • @dzieciou I did the same. But I am curious about other solutions. Nice question.
    – Twaldigas
    Commented Aug 7, 2014 at 13:03
  • 1
    In that case, I would refactor the baby out of the bath water.
    – user246
    Commented Aug 7, 2014 at 13:22

3 Answers 3

Reset to default
3

Do not disable the tests, which fail because of known bugs.

I would suggest to group the failing tests and run them in another test suite. You might use a group annotation on the tests for that, maybe "knownFailures" or so. The group annotation is better then a "link to issue" thing. Group annotations are supported by various test tools. Configure your build matrix to allow the failure of this test suite.

Also exclude the group of tests (forming the failing suite) from your main test suite. That keeps the main suite green.

You can then work on both suites in your matrix. The main will switch between green and red, while the "knownFailures" stays red - BUT here the number of failing tests will reduce, if you fix some of the known bugs. ,)

And you may introduce an automated check: for tests with the @group knownFailures, which are green. That helps you to remove the @group annotion, when they go from red to green.

Improve this answer
1

Our approach to the situation in short is to tolerate the situation that the build is not completely green. We track which parts of the software fail. If we consider the issue as a bug and plan to fix it in the future, having a red light in that feature is important information to us.

We have categorized features in a requirement tree which status we monitor. So if there is a red light, it is on one branch of the tree so we know in what area there are shortcomings. That is a fast way to understand the quality of the whole system.

Technically we have Jenkins CI which sends automated testing data to Meliora Testlab where we monitor the testing and requirements status and store our issues. For us this works great as we have good understanding on quality and trough issues we always have a track of things to be done.

Improve this answer
0
0

What we do is tag the known-to-fail tests as non-critical, and then tell Jenkins to only consider critical tests when determining whether the build succeeded or failed. This way you still run the tests and can still see failing tests in the final report, but they don't break the build.

This is made easier by the fact our testing framework has support for critical/non-critical built-in, so it might be more difficult if your testing framework doesn't support any sort of tagging feature.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.