I'm writing test cases for a simple open-source utility using Perl's Test::More
module.
There are two bugs in the utility that, due to the way they interact, cause one of the test cases to pass. (I discovered the two bugs because there are other test cases where the two bugs don't cancel each other out.) If either bug is fixed (but not both), suddenly that test case starts failing. How should I annotate the test to deal with this situation?
Normally when there's a bug I simply mark the test as an expected failure with a reference to the bug report. In this case, I can't mark it as an expected failure because the test itself is not failing.
My default plan is to leave the test marked as expected to pass and just add a comment next to the test case explaining the situation. There are a few reasons why I don't like this:
- The passing test case gives a false sense of proper behavior.
- When a developer fixes one of the two bugs they might be surprised to see a newly failing test case. The comment next to the test case should make it easy to investigate the situation and change the test from expected pass to expected fail, but I still prefer to avoid surprises.
- When a commit intended to fix a bug causes a test to start failing, others will be suspicious of the bug fix (and the associated change to mark the test case as expected to fail).
- Suppose commit #1 fixes one of the bugs and marks the test case as expected to fail. Commit #2 fixes the other bug and marks the test case as expected to pass. Now commit #2 can't be cherry-picked to a release branch without either cherry-picking commit #1 or re-marking the test case as expected to fail. Cherry-picking commit #1 might be considered too risky for the release branch, but re-marking the test case increases the likelihood of introducing a merge conflict resolution error.
The Test::More
module (and the Test Anything Protocol) don't support marking a test as "accidental pass", but even if they did I don't know if that would be the right way to deal with this case.
How should I mark the test case to reduce the risk of future problems?