How to deal with a test failing but with a fix in the works?

Question

I have automated tests running on our Continuous Integration machine.

I have a test failing because of a commit, but this fail has been considered minor, and some more urgent work was scheduled for this iteration.

However, if this test is failing, all other commits will fail until we fix this particular low-prio bug.

What is the best approach to this situation? Do I change the test for now so it would PASS and then when the bug is fixed it will fail, so we then change the test again?

Any other idea?

Answer 1

I would personally...

1. Log the work to be completed at some point in the future into your ticketing system of choice. This may be a subtask under the development work to fix the application feature itself.
2. Update the test to be skipped during automated test execution until it passes again (if waiting on development) or until it can be fixed (if waiting on QA/automation).
3. Add a comment to the test case code explaining why the test is set to "skip" and relevant context (including ticket numbers).

Answer 2

General idea is to skip, disable or otherwise exclude the test case from the run. That way your build will pass but you will have to remember to get skipped case back into the test suit once it's fixed so it will be included in all future builds.

Some frameworks offer a special marker/status: expected to fail. Cases marked as expected to fail will not be treated as failed but in the report you'll see that those are failing "as expected".

In my opinion that is the best approach. See for example https://docs.pytest.org/en/latest/skipping.html#xfail-mark-test-functions-as-expected-to-fail