4

Do you guys separate fails and errors in your automated tests report?

  • fail - assert error
  • error - all code errors (except from asserts)

In my opinion we should. Lets say we have such testcase scenario:

  1. login -> logged in
  2. Create contact (example) -> contact created

If we'd have 2 automated tests for these...:

  • I test

    1. login
    2. ASSERT if logged in
  • II test

    1. login
    2. create contact
    3. ASSERT if contact created

... and login will be bugged, we could simply filter test results with fails only and find out that login failed (because we have assert on that) by ignoring other ERROR types (we could have 20 of them) that occurred due to that login fail)

What's your approach/experience?

  • Are you talking about application errors or automation code errors itself? – Vishal Aggarwal Feb 18 '18 at 10:02
  • Also if it is an application error,it should be trapped in one of the assertions,as the expected states of objects will be changed somehow as an impact of the error. – Vishal Aggarwal Feb 18 '18 at 10:06
  • Example: home Page title assertion will be failed if on login there is an runtime error displayed on the page instead of Home page. – Vishal Aggarwal Feb 18 '18 at 10:09
  • @VishalAggarwal Im talking about automation code errors itself – Shinigamiyuu Feb 18 '18 at 10:31
2

We started to separate fails and other errors in our reports from the beginning and see that welcomed by readers of the reports.

The fails are an outcome of a test that is somewhat expected. When seeing a pass or fail the reviewer will know that the basic logic has worked. When encountering an error it's a clear sign that the test itself has failed to operate as expected.

Users later requested that automatic screenshot logging also differs between the two (logScreenshotOnFail and logScreenshotOnError) which I would count as a proof that the differentiation makes sense and is wanted.

  • This answer also fits my topic, but I chose @Alexey R. as it has better argumentation, sorry mate :/ – Shinigamiyuu Feb 18 '18 at 11:37
2

Strict methodologies allow automated test case only to be Passed or to be Failed.

There could be negative test cases, where the application returns errors, but if this is properly expected, the test will be Passed as you expect an error.

So if you tend to filter out some fails, most probably you don't have proper assertions defined.

If you have a data-driven test, you should include an "expect" for each dataset and define such assertion which compares expected and real result.

  • @Shinigamiyuu this attitude will lead you directly to the failing, buggy software and test results which are unreliable and not relevant. – Dee Feb 19 '18 at 12:19
  • +1. Exactly. It makes no difference what statement caused the test to fail. @Shinigamiyuu - Go ahead and disagree, it just shown your lack of experience. – Peter M. Feb 19 '18 at 13:31
  • Maybe we got eachother wrong. I want to filter FAILS as priority number 1 just to see which asserts failed (first thing to take a look, because this error will be more readable as its expected to be handled). Obviously I won't be ignoring ERROR reports, I'll just put it as second priority while reading the test results. What do you guys think? @PeterMasiar – Shinigamiyuu Feb 19 '18 at 15:32
2

We separate assertion errors and code errors. The latter often means we missed some important logic to cover with assertions. So we tend not to have any exceptions (or code errors).

Thus in our practice the code errors (exceptions) usually appear in the case of some really exceptional cases like database unavailability or so, which are not related to business logic of the application under test and which we cannot impact or control.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.