Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Each of our end-to-end test cases checks two aspects/functionalities of message processing:

  1. Whether the message has been processed correctly (resulted in a correct action of the system).
  2. Whether information about processing been correctly logged to DB (for further reporting purposes).

We decided to verify both aspects in a single case, because end-to-end tests takes much time (this is distributed system and message traverse through a number of slower, legacy systems).

Now, when assertions related to first functionality pass, and for the second will fail, it does not matter. All test cases are marked as failed, so the message sent to the management is that 100% of test cases failed, ignoring the fact, that the latter functionality is less important to our business. This is because wustomers won't care having correct reports when processing is incorrect.

How would you solve this?

  • Create separate test cases for different aspects? (more time to run the tests) OR
  • Configure TestNG to create separate reports for each functionality/group of assertions? (if so, then how?)

Update: Having correct reports when processing is incorrect is useful only to us for troubleshooting. That is one another reason why we decided to combine checking both aspects in one test.

share|improve this question
1  
Does it matter that "100% failure" is sent to management? What would they do differently if you indicated "50% failure" instead? (Sometimes these are metrics without much meaning) –  Joe Strazzere Aug 16 '13 at 15:01
    
Joe, usually our test report is more torough (metrics are accompanied with defects list and their severity). However, this time I had less time to prepare such report (and create defects), so I started to think how test framework and test design can help provide clear feedback more quickly. –  dzieciou Aug 17 '13 at 8:10

2 Answers 2

up vote 5 down vote accepted

You did not provide a lot of detail about the relationship between the logging system and the processing system, so my answer is going to be general/vague. I would try the following, in this order:

  1. Educate your management. You have a good reason for how you structured your tests. Explain what they can and cannot infer from a 100% failure rate. Explain the cost of structuring the tests in a different way.

  2. Reorganize the tests even if they need to run twice as long. You didn't actually say whether running twice as long is just an inconvenient to you or is an impediment to others in your organization as well. The next two alternatives will almost certainly be more complicated than this one; you need to decide which is more painful: the extra run time of this alternative or the additional coding time / maintenance cost of the next two alternatives.

  3. Structure your logging tests so, when the process aspect fails, the logging tests report errors rather than failures. I'm not sure this obviates the need to educate your management, but at least you will have a way to distinguish "Logging not tested" conditions from "Logging tests failed" conditions.

  4. Find a way to test logging (aspect 2) without actually doing any processing (aspect 1). You did not provide enough detail to indicate whether this is possible, and presumably if it were, you would have explored this already. Still, if the first three alternatives are not doable, this is what you are left with.

share|improve this answer
1  
Good answer, though I'd recommend thinking hard about whether option 1 is realistic. Management are often resistant to reinterpretation of the data you've given them - and it looks bad for you when you have to tell them that your data is potentially misleading. –  vincebowdren Aug 16 '13 at 13:31
    
Regarding 1st idea: Our managers are really reasonable people with much experience in both testing and development, so they ask questions about details, especially when the news are bad :-) It just takes me more time to answer those questions, so I was thinking how to improve logging those errors. –  dzieciou Aug 17 '13 at 7:57
    
I really like the 3rd idea. I think I might use hard assertions for more important functionality and soft assertions for less important one. A failing hard assertion will immediately stop test execution, while hard assertion success will let soft assertions to provide feedback on reporting functionality... This might be a way to implement your 3rd idea. I just nead to read more on TestNG/JUnit result output format. –  dzieciou Aug 17 '13 at 8:15

Wrap the whole test in either @BeforeClass method (bad solution) or in a class with some flag that indicates if test was already run or not (still not good but better)

public class MyTestClass() {
   private static boolean alreadyRun = false;
   public static void runTest() {
        if (!alreadyRun) {
           alreadyRun = true;
           //...
           //run the actual test procedure
           //...
        }
   }
}

Then create two test methods, call runTest() in each of them and log appropriate assertions. In this way you will run test procedure only once but you will end with two separate entries in TestNG report.

That would actually be the point number 4 on user246's list above.

share|improve this answer
    
Thanks for the details - I think you know more about TestNG than I do. –  user246 Aug 17 '13 at 0:43
    
+1 That's a good idea to separate test run from checking test output with assertions. It will not work for some more complex test cases with multiple steps interwoven with assertions (step 1, assertion 1, step 2, assertion 2). Grouping all assertions together after step 2 would make no sense, because step 2 may override the system status after step 1 and assertion 1 will always fail. –  dzieciou Aug 17 '13 at 7:39
    
Still, JacekM's idea made me thinking about something like for(Functionality1).verifyThat(...) were v`erifyThat` is soft assertions and results for different soft assertions are logged by the test framework on separate stacks and then reported separately... Somehow... Need to think how to implement that on TestNG. –  dzieciou Aug 17 '13 at 7:43
    
@user246, actually this is functionality (separating setup for tests in BeforeClass annotated methods from actual test cases) is common feature in many XUnit frameworks. But JacekM applied this very nicely! –  dzieciou Aug 17 '13 at 7:49

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.