Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

We have about 10 test suites (QF-Test GUI Testing tool) for a java application. At the moment the testsuits are run every night with a batch-script triggered by a scheduled job. The tests are not very stable - so every day we have some failures. QF-Tests gives back the test-result in the exit code (0=OK, 1=Warning, 2=Error, etc.). I would like to rerun the tests if they fail.

Is there a tool to run the test suite and run other commands depending on the result?

Example:

  • If Exitcode=1: Run Report-Command
  • If Exitcode=2: Delete the results and rerun until 5 retries reached.
  • If Exitcode=3: ...

I know I could do this programmatically, but I wonder if there is an easier solution for this.

Thanks for any advice.

share|improve this question

2 Answers 2

I don't know any tool parsing exit codes.


Anyway, I suppose "Programmatically" (writing own script) is the best solution.

At least you know, what's happening and can fix it yourself quick.


Here's batch script as an example:

qftest.exe ...
if %ERRORLEVEL% EQU 1 ( command1; command2; etc )
if %ERRORLEVEL% EQU 2 ( command1; command2; etc )
share|improve this answer

Updated based on comments. I have not used any tool but a little custom code for below approach helped me achieve rerunning failed cases.

Through Custom Code

  • Easier option is to track the flags after test execution in a simple DB table (TestResults - TestcaseId, TestrunId, Result) - Update it after each run, or a custom xml to save the results
  • After Test run parse through flagged (failed) entries in tables / failed cases and re-run them selectively
  • Since you execute from command line, it should be easy to group / execute by flagging failed cases and retry it later
  • Also, You can run it for configured number of retries

By using TestNG, Junit you can leverage in-built features for re-running failed cases.

From TestNG blog - http://testng.org/doc/documentation-main.html#rerunning. Rerun with failed XML testng-failed.xml

Using TestNG - Another approach listed in post - http://seleniumworks.blogspot.in/2013/12/re-run-failed-tests-automatically.html

share|improve this answer
1  
Both your answers are about TestNG. I have GUI-Tests and can only run them from command line. With "Programmatically" I mean writing my own script (Bat, Groovy, etc.) and run the commands from the script. IMO your answer doesn't help me. Or did I miss something? Thanks anyway! –  Tobias Jun 17 at 11:11
    
Yea Other than TestNG, a custom Db table with test case - pass / fail status can be tracked. Custom code option suggested be a possible solution. Crux is re-run all failed cases. Custom coding / re-running failed cases would require some custom coding –  Siva Jun 17 at 13:24

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.