Multiple Assertions and Continue on Assertion Failures in Functional Test an Anti-pattern?

Question

According to the unit test gurus, more than one or two assertions about different things in a unit test is an anti-pattern: https://stackoverflow.com/questions/4732827/continuing-in-pythons-unittest-when-an-assertion-fails#4733120

I generally agree with this answer.

But what about integration/functional tests, such as Selenium automated browser tests? Let's say there's a bunch of setup to get the application in the state where assertions can be made. Then the page must change state again after the assertions to take it to another state where more assertions are made?

Better to have all that setup in a fixture and run multiple tests, doing each assertion in a separate test method? This is cleaner, more focused, and easier to figure out assertion failures, but a bit pedantic.

Then what about strategies to continue on assertions failures, which I find useless since if an earlier assertion failed, it makes later assertions questionable. xunit frameworks don't generally allow this natively, while functional test frameworks like Robot Framework do. Is it OK to do this in your opinion?

What is generally considered end-to-end automation best-practice in these areas by the pros?

Answer 1

So when targeting Unit tests it's usually a pass/fail. When targeting a web application for UI tests there is a varying amount of results and not just a pass/fail at a test level.

If using something like Selenium on a unit test framework I recommend 3 things that have great benefit if done properly.

1. Develop your own result output where you can define specifics that when reading results it actually provides value like "all controls are present but the login credentials produce an invalid login message" instead of just "test failed...do it all again manually". This takes some structuring and work but provides great benefit in the long run as you get more and more tests.
2. Develop your own error handling layer that will skip steps that fail and continue on when desired and kill the test on others where desired. This would also log results so that you can see errors as well as test steps.
3. Common wrapper for all test steps that will utilize 1 & 2 consistently for each step. This ensures more control over what is reported and what is ignored in your results.