1

Sometimes, I can see the tests written like that

public void shouldDoX() throws SomeCustomException {
...

I think throwing custom exceptions is an anti-pattern because it makes analysis of the test harder: it is not clear whether test should be skipped or failed. Skipping a test means certain preconditions for a test has not been satisfied. For instance, skipping authentication test doesn't mean that authentication does not work but, for instance, that a test user does not exist in a database.

A proposed pattern is to throw either an exception subclassing AssertionError (failure) or an exception subclassing SkipException (skipping).

Do you agree?

Improve this question
6
  • 1
    I think the statement "throwing custom test exceptions is an anti-pattern" is a little too broad. It all depends on how the exceptions are handled. Specifically, for the proposed pattern you provided, why not throw CustomAssertionError1 and CustomAssertionError2 that derive from AssertionError? That way they can be handled in a general way but can also provide more detail in test reporting and debugging.
    – VanderLinden
    Apr 21, 2017 at 15:51
  • @VanderLinden You're right. I updated my question.
    – dzieciou
    Apr 21, 2017 at 15:55
  • In that case, I agree with the sentiment that custom exceptions should be thoughtfully designed and used for a specific purpose (and should probably derive from a relevant class where applicable, as you proposed).
    – VanderLinden
    Apr 21, 2017 at 16:13
  • Possibly better audience for this question would be softwareengineering.stackexchange.com Here, average questions is how to start a career in QA automation without learning any programming, how to click on hidden element or how make Se IDE work with dynamic XPath. Pretty sad, really.
    – Peter M. - stands for Monica
    Apr 21, 2017 at 16:31
  • 2
    This discussion is better suited for some meta debate. I think that because software engineering SE has also tags "selenium" and "test-automation", it will be better venue for answers which rely on software engineering skills. I do not see any way to improve signal-to-noise ratio here, because most most new posters do not and will not use even Google to look for answers. My goal as poster would be to find a venue where I can get best quality answers. My goal as SQA "elder" would be to make SQA SE valuable for most posters. Seems to be en.wikipedia.org/wiki/Eternal_September here.
    – Peter M. - stands for Monica
    Apr 21, 2017 at 18:44

1 Answer 1

Reset to default
2

Ideally I wrap the test step in a error handling mechanism that either throws the error and kills the test or it proceeds to the next step even if it fails. The test log records the failure, but the rest of the steps get executed so that the other functionality can be tested too. The step then passes a true/false depending on the nature of the step which would then either fail over and kill the test or move on to the next step anyway.

From a code standpoint it won't be able to determine if it's a serious error or a minimal one so you have to put some intelligence in the test step to go along with it in order to determine if you can continue should the assertion fail or if you need to stop all together. From a coding standpoint if it fails it's dead regardless of the failure.

In general Selenium is executed on a Unit Test framework which would make any error a failure unless that is the actual goal. If it's c# you can catch everything generically and then spit out the exact text, with java you have to catch the unique errors that are possible.

I often throw a unique error with specific text based on the framework itself when someone doesn't utilize the framework properly, but that is totally up to the automation developer. If you utilize custom errors be very consistent and consider creating an error framework so that it's universal and easy to trace back to the source issue that triggered the error.

Improve this answer
4
  • "The step then passes a true/false depending on the nature of the step which would then either fail over and kill the test or move on to the next step anyway." - I'm using a smilar approach but with soft asserts and hard asserts.
    – dzieciou
    Apr 24, 2017 at 9:00
  • Yes, for automated testing it's a good method. Standard Unit test would probably not agree as the test should be simple enough to target a single or a couple failures.
    – mutt
    Apr 24, 2017 at 15:44
  • Re: "From a code standpoint it won't be able to determine if it's a serious error or a minimal one". But you can decide which checks in your tests are preconditions (e.g. a user must be present in the system to be tested), from checks actual functionality (e.g., a user present in the system should be able to login).
    – dzieciou
    Apr 25, 2017 at 15:52
  • 1
    yep, that's my exact implication...I think we are on the same page...
    – mutt
    Apr 25, 2017 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.