Each of our end-to-end test cases checks two aspects/functionalities of message processing:
- Whether the message has been processed correctly (resulted in a correct action of the system).
- Whether information about processing been correctly logged to DB (for further reporting purposes).
We decided to verify both aspects in a single case, because end-to-end tests takes much time (this is distributed system and message traverse through a number of slower, legacy systems).
Now, when assertions related to first functionality pass, and for the second will fail, it does not matter. All test cases are marked as failed, so the message sent to the management is that 100% of test cases failed, ignoring the fact, that the latter functionality is less important to our business. This is because wustomers won't care having correct reports when processing is incorrect.
How would you solve this?
- Create separate test cases for different aspects? (more time to run the tests) OR
- Configure TestNG to create separate reports for each functionality/group of assertions? (if so, then how?)
Update: Having correct reports when processing is incorrect is useful only to us for troubleshooting. That is one another reason why we decided to combine checking both aspects in one test.