0

According to the unit test gurus, more than one or two assertions about different things in a unit test is an anti-pattern: https://stackoverflow.com/questions/4732827/continuing-in-pythons-unittest-when-an-assertion-fails#4733120

I generally agree with this answer.

But what about integration/functional tests, such as Selenium automated browser tests? Let's say there's a bunch of setup to get the application in the state where assertions can be made. Then the page must change state again after the assertions to take it to another state where more assertions are made?

Better to have all that setup in a fixture and run multiple tests, doing each assertion in a separate test method? This is cleaner, more focused, and easier to figure out assertion failures, but a bit pedantic.

Then what about strategies to continue on assertions failures, which I find useless since if an earlier assertion failed, it makes later assertions questionable. xunit frameworks don't generally allow this natively, while functional test frameworks like Robot Framework do. Is it OK to do this in your opinion?

What is generally considered end-to-end automation best-practice in these areas by the pros?

8
  • IMHO, I prefer your first option. One thing I find it useful to stop as soon as the assertion fails is that there is less test result to investigate the error. In Selenium context, you will be running multiple browsers on different platforms, the less test result to read, the better to get it diagnosed and fixed.
    – chuan
    Aug 1, 2014 at 8:26
  • So you would rather have repetitive setup than make multiple assertions in the same session? That's what I try to stress to my QA members, since you can always multiprocess.
    – Aaron
    Aug 1, 2014 at 13:50
  • In my opinion, given the expense of setting up an integration test, you should test as much as possible that makes sense inside of your test scenario. For example, with a press release I would test that I found what looks like a press release (and error if I don't), then I would verify all the little parts of a press release (title, date, teaser, teaser link, etc) and spit out a report of everything that errors. What I wouldn't do is attempt to test the header or footer or anything unrelated to a press release, but while I have a press release there, test as much as you can.
    – Jeff Price
    Aug 1, 2014 at 15:59
  • So if you didn't find a press report you would stop executing? In other words does the concept of continue on failure for assertions make sense for functional tests, but not for unit tests? Or is it that that first assertion is so important that if it fails you stop running, but if it passes you continue on subsequent assertion failures?
    – Aaron
    Aug 1, 2014 at 16:29
  • Let's take the press release example. For a function test you want to see if the "process" of adding a press release worked. 1. You have to load the form 2. You have to fill out and submit the form. 3. Visit the press release page. 4. Find the press release 5. Verify all the content entered in step 2 is there. Every step in that process is an assertion. You are asserting the add press release form loaded (and not the add blog entry). You are asserting the submit processed. You are asserting a press release saved and can be viewed. If any of these fail, it doesn't make sense to continue.
    – Jeff Price
    Aug 1, 2014 at 17:37

1 Answer 1

1

So when targeting Unit tests it's usually a pass/fail. When targeting a web application for UI tests there is a varying amount of results and not just a pass/fail at a test level.

If using something like Selenium on a unit test framework I recommend 3 things that have great benefit if done properly.

  1. Develop your own result output where you can define specifics that when reading results it actually provides value like "all controls are present but the login credentials produce an invalid login message" instead of just "test failed...do it all again manually". This takes some structuring and work but provides great benefit in the long run as you get more and more tests.
  2. Develop your own error handling layer that will skip steps that fail and continue on when desired and kill the test on others where desired. This would also log results so that you can see errors as well as test steps.
  3. Common wrapper for all test steps that will utilize 1 & 2 consistently for each step. This ensures more control over what is reported and what is ignored in your results.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.