I am running Data-driven tests driven from a spreadsheet which are input to TestNG test Framework. In a nutshell, the framework traverses through outlook mailbox and looks for mails based on the subject. Each row in the spreadsheet is the mail subject which is an input to the test.
String comparisonResult = GetMail.compareMails(scenarioName);
Assert.assertNotNull(comparisonResult,"Mail does not exist");
Under the hood, comparemails
method compares the subject of the mail passed as a parameter to the method and looks for that mail in the mailbox.
I am asserting for null value when the mail does not exist and reporting with an assertion message - "Mail does not exist in the mailbox." However the assertion message is not displayed after each test failure, instead is reported a the end of all test runs i.e., after all the rows of the spreadsheet are processed.
When the assertion fails, I would expect it to print the assertion message on the console immediately and process the next test from the spreadsheet.
However, it prints the message at the end of the test all at once like below, assuming there were 5 failures:
[name=null](com.dms.testsuite.Test) Time elapsed: 66.905 sec <<< FAILURE!
java.lang.AssertionError: Mail does not exist expected object to not be null
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.assertNotNull(Assert.java:404)
at com.dms.testsuite.Test.submitJob(Test.java:110)
[name=null](com.dms.testsuite.Test) Time elapsed: 83.24 sec <<< FAILURE!
java.lang.AssertionError: Mail does not exist expected object to not be null
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.assertNotNull(Assert.java:404)
at com.dms.testsuite.Test.submitJob(Test.java:110)
[name=null](com.dms.testsuite.Test) Time elapsed: 51.064 sec <<< FAILURE!
java.lang.AssertionError: Mail does not exist expected object to not be null
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.assertNotNull(Assert.java:404)
at com.dms.testsuite.Test.submitJob(Test.java:110)
[name=null](com.dms.testsuite.Test) Time elapsed: 51.735 sec <<< FAILURE!
java.lang.AssertionError: Mail does not exist expected object to not be null
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.assertNotNull(Assert.java:404)
at com.dms.testsuite.Test.submitJob(Test.java:110)
[name=null](com.dms.testsuite.Test) Time elapsed: 67.177 sec <<< FAILURE!
java.lang.AssertionError: Mail does not exist expected object to not be null
at org.testng.Assert.fail(Assert.java:94)
at org.testng.Assert.assertNotNull(Assert.java:404)
at com.dms.testsuite.Test.submitJob(Test.java:110)
and the results are reported as follows:
Failed tests:
com.dms.testsuite.Test.[name=null](com.dms.testsuite.Test)
Run 1: Test.submitJob:110 Mail does not exist expected object to not be null
Run 2: PASS
Run 3: Test.submitJob:110 Mail does not exist expected object to not be null
Run 4: PASS
Run 5: Test.submitJob:110 Mail does not exist expected object to not be null
Run 6: PASS
Run 7: Test.submitJob:110 Mail does not exist expected object to not be null
Run 8: Test.submitJob:110 Mail does not exist expected object to not be null