Skip to main content
Tweeted twitter.com/#!/StackSQA/status/497370669169274883
added 6 characters in body
Source Link
dzieciou
  • 10.5k
  • 9
  • 48
  • 102

Number one rule with continuous integration (CI) server is to have build always green and when a test fails fix the system under test as quickly as possible.

However, we have currently end-to-end and integration tests on CI server that fails because of known bugs. We are aware of those bugs but they are minor and they have less priority to be fixed immediately. Consequently, build is no longer green and if new bug will be introduced we will have no such a good feedback on that.

Our first solution is to disable such failing tests, keep in the test code a link to a bug (JIRA issues) and keep in JIRA a link back to the bugtest. This obviously have some shortcomings such as the need to track and remember about such disabled tests. Also, disabled tests will no discover other bugs.

What would be your approach for such a situation?

Number one rule with continuous integration (CI) server is to have build always green and when a test fails fix the system under test as quickly as possible.

However, we have currently end-to-end and integration tests on CI server that fails because of known bugs. We are aware of those bugs but they are minor and they have less priority to be fixed immediately. Consequently, build is no longer green and if new bug will be introduced we will have no such a good feedback on that.

Our first solution is to disable such failing tests, keep in the test code a link to a bug (JIRA issues) and keep in JIRA a link to the bug. This obviously have some shortcomings such as the need to track and remember about such disabled tests. Also, disabled tests will no discover other bugs.

What would be your approach for such a situation?

Number one rule with continuous integration (CI) server is to have build always green and when a test fails fix the system under test as quickly as possible.

However, we have currently end-to-end and integration tests on CI server that fails because of known bugs. We are aware of those bugs but they are minor and they have less priority to be fixed immediately. Consequently, build is no longer green and if new bug will be introduced we will have no such a good feedback on that.

Our first solution is to disable such failing tests, keep in the test code a link to a bug (JIRA issues) and keep in JIRA a link back to the test. This obviously have some shortcomings such as the need to track and remember about such disabled tests. Also, disabled tests will no discover other bugs.

What would be your approach for such a situation?

Source Link
dzieciou
  • 10.5k
  • 9
  • 48
  • 102

Disabling failing tests in CI if they discover known bugs

Number one rule with continuous integration (CI) server is to have build always green and when a test fails fix the system under test as quickly as possible.

However, we have currently end-to-end and integration tests on CI server that fails because of known bugs. We are aware of those bugs but they are minor and they have less priority to be fixed immediately. Consequently, build is no longer green and if new bug will be introduced we will have no such a good feedback on that.

Our first solution is to disable such failing tests, keep in the test code a link to a bug (JIRA issues) and keep in JIRA a link to the bug. This obviously have some shortcomings such as the need to track and remember about such disabled tests. Also, disabled tests will no discover other bugs.

What would be your approach for such a situation?