Fail vs Error in automated tests

Question

Do you guys separate fails and errors in your automated tests report?

• fail - assert error
• error - all code errors (except from asserts)

In my opinion we should. Lets say we have such testcase scenario:

1. login -> logged in
2. Create contact (example) -> contact created

If we'd have 2 automated tests for these...:

• I test

  1. login
  2. ASSERT if logged in

• II test

  1. login
  2. create contact
  3. ASSERT if contact created

... and login will be bugged, we could simply filter test results with fails only and find out that login failed (because we have assert on that) by ignoring other ERROR types (we could have 20 of them) that occurred due to that login fail)

What's your approach/experience?

Answer 1

We started to separate fails and other errors in our reports from the beginning and see that welcomed by readers of the reports.

The fails are an outcome of a test that is somewhat expected. When seeing a pass or fail the reviewer will know that the basic logic has worked. When encountering an error it's a clear sign that the test itself has failed to operate as expected.

Users later requested that automatic screenshot logging also differs between the two (logScreenshotOnFail and logScreenshotOnError) which I would count as a proof that the differentiation makes sense and is wanted.

Answer 2

Strict methodologies allow automated test case only to be Passed or to be Failed.

There could be negative test cases, where the application returns errors, but if this is properly expected, the test will be Passed as you expect an error.

So if you tend to filter out some fails, most probably you don't have proper assertions defined.

If you have a data-driven test, you should include an "expect" for each dataset and define such assertion which compares expected and real result.

Answer 3

We separate assertion errors and code errors. The latter often means we missed some important logic to cover with assertions. So we tend not to have any exceptions (or code errors).

Thus in our practice the code errors (exceptions) usually appear in the case of some really exceptional cases like database unavailability or so, which are not related to business logic of the application under test and which we cannot impact or control.