Should I use “the test fails if” instead of “the test succeeds if”?

Question

My project inherited a document describing test procedures. It describes the expected results in terms of "the test fails if".

Over time we found that this way to describe results is not sufficient and added an adjacent "the test success if" section, but the "fails" section stayed as it is quite useful.

When I asked the original writer why he used this, he didn't remember. Googling and reading some certification documents, I found few examples of this negative phrasing. Anyone knows the source of this way of writing expected results?

Answer 1

I'm afraid I don't know the origin of this way of phrasing and I've not seen it before, however I was surprised to see that the answers here consider 'fails if' equivalent to 'passes if', as I quite strongly consider the two to be very different from each-other and also my opinion is that 'fails if' is slightly superior to 'passes if' for manual tests anyway - if I think about an automated test it seems only able to assert a 'passes if' or a 'don't fail if'.

Taking the example from glowcoder's answer, a requirement for a search box that is prompted to the user. In a test step that tests this condition, we could say 'the test fails if the search box is not prompted to the user' or 'the test passes if the search box is prompted to the user'. They seem equivalent - both do not fail if the search box is prompted to the user but consider the following results:

Actual Result: The search box is prompted to the user but there's an error on the page

• 'fails if' pass or fail: undetermined
• 'passes if' pass or fail: pass

Actual Result: The search box is prompted to the user but the label for Search is spelt 'Saerch'

• 'fails if' pass or fail: undetermined
• 'passes if' pass or fail: pass

Actual Result: The search box is prompted to the user but it takes more than ten seconds for the search box to load

• 'fails if' pass or fail: undetermined
• 'passes if' pass or fail: pass

Hopefully there's not much in it, as any manual test will hopefully be run by someone with the ability to identify that although the pass criteria are met, if errors occur when the search box loads, then the step cannot really be considered to be passing. It just feels to me that 'passes if' may have an increased tendency to encourage confirmation bias in the tester, in that they only need to look for one pass condition to be able to move onto the next step with confidence.

Personally, I don't use either terminology in my test scripts tending as I do to use a more dispassionate and less leading terminology like 'There will be a search box that is prompted to the user'. I have to able to safely assume that the testers I work with are able to look around the software under test and read far more than I could ever write into a test step. In my opinion there are always many more possible failure conditions than there are easily identifiable passing ones.

Answer 2

It's the matter of conventions in the team, no more. Expected result "test fails if the login form is absent" is identical to "test succeeds if login form is present".

The test cases can be incomplete, but it does not matter what convention was used, if tester forgot to add a check for the presence of a search field.

Answer 3

The requirements documentation should give a clear description of what is expected. That should always be written in such a way that anyone can tell if the software is successful at meeting that requirement.

Given that, I don't think it matters either way which one you choose. I don't think you need to include both, but you do need to define in the test what determines a pass or fail. For us, we have a test step description, a set of expected results after executing the step, and then a pass/fail checkbox. If the expected results are met, then the test passes. If not, then the test fails. The expected results can be written to include positive and negative. The test step could be "Navigate to the URL for the login page." The expected results could then be "The login page is present. No errors are encountered in software startup." The test is not just successful if the page comes up, but the page must come up with no other adverse effects.

Answer 4

For sure, I don't think you would want both on the same test. Your requirement will be either exclusive (more failing conditions than passing conditions, so you'd want to say "passes if...") or inclusive (more passing conditions than failing conditions, so you'd want to say "fails if..."). If you had both, that would indicate a more complicated requirement such as "passes if the search box is prompted to the user, but fails if the color scheme doesn't match the rest of the dialogs." That would make me want to make two tests out of it, one that the prompt comes up, and another that the prompts color is correct.

I think a system-wide standard of "defining tests in terms of fails if..." and not allowing the "passes if..." methodology would cause needlessly verbose (and obfuscated) test cases. The same would be true if passes were permitted but fails were not.

The only requirement I would put on the phrasing of the documentation would be that it effectively maps the test case to the requirement and either explains what the reader needs to know to interpret the test, or points them to further information where they can get their explanation. Both the passes if and fails if approaches are appropriate in their own circumstances, and should be used when they are the correct choice.