I’m desperately looking for a test management tool that fully supports automated tests. As this sounds simple, I still fail to do so.

We have a bunch of automated tests, which are run as part of our continuous integration pipeline. This is fine and works as wanted, but we miss the benefits of test management tools, such as

  • mapping requirements with test cases
  • getting test coverage for requirements
  • linking test cases/test results with issues – easy retesting of fixed reported issues

and so on and so on. So simply said, the good things about a test management tool.

So I have been trying to find one, but it seems that all the ones I've found support mostly manual testing and/or have only limited support of automated testing.

Duplication of test cases

We have now a bunch of test cases already defined in natural language (via http://gauge.org). I want to integrate those into the TMT and not define them again, to avoid having duplicated my test cases and maintain two (or more) copies of them.

Our test cases reside in git, and I have not found a tool to import test cases from git.

Automation results

All TMT's offer a workflow to add results in an intuitive manner (for manual tests), but I have not found a tool that supports automation, like trigger tests from the tool and retrieve the results.

All that is quite frustrating and I cannot think that it is only me fighting this. Does anyone have any experience with any tool for automated tests?

Please do not simply post links; please include details about why a particular tool helps and how this tool would help me in the problems described above.

  • What is your TMT and automated tool/framework and what is the concrete problem or error you are facing? Otherwise it's too broad... – Vishal Aggarwal Apr 9 '18 at 11:16
  • All major TMTs provide an public API to integrate it with externals as per custom requirements. – Vishal Aggarwal Apr 9 '18 at 11:19
  • @VishalAggarwal I do not have a TMT, because I am looking for one. And as I said, we use Gauge as our test framework. I also have stated the problems I have (the bold lines), namely duplication of test cases and test result handling. I know that TMTs provide an public API, but I will not pay for a tool, where I need to bench my requirements, simply to be able to use the public API – emersoncod Apr 9 '18 at 11:23
  • 1
    For transparency since this seems to be attracting close votes despite being a good, on-topic question: sqa.stackexchange.com/review/close/22899 – c32hedge Apr 9 '18 at 13:24
  • 1
    As a minor note, unless you're wanting the version histories of your tests that live in git to import into your tool, you really don't need to be able to import from git--you're looking for a tool that can import from presumably text-based files in whatever format you use. The format itself is the important detail, not that they're in git. – c32hedge Apr 9 '18 at 13:32

When doing automated tests I have always been wary of test management tools especially since the advent of BDD. Every single one I have used has caused major problems with being unreliable or inflexible, once you have to start working on tests outside it then it has failed.

My philosophy with testing is to always keep things as simple as possible and this extends to choosing tools and processes.

  1. mapping requirements

If you are using BDD well then you should have feature files from your PO that define test requirements; if these are not good then you should look in to improving this process. I find 3 amigos sessions can really help with this and give a chance to refine your requirements before bigger commitment to them is made.

  1. Test coverage

Test coverage is almost impossible to do properly with automated tests as they typically combine multiple components, coverage really only applies with unit tests where you can count from the dispatcher where the test is going. I find that where I need coverage analysis I add it to Jenkins jobs. If I need to do anything bigger around coverage I use sonarqube but it can be misleading.

  1. Result reporting

As long as you can output your results in XUnit then it's possible to get a reasonable display of historic test results. I have found that the new Jenkins blue ocean UI is an easy way to get most of this. Most CI tools have plugins that will help but wont be as constraining as a full TMT.

So I guess in summary, be careful what you wish for: it may become your prison :)

  • thanks for the comment. Regardin 1. mapping requirements. It is about traceablity to be able to automatically map a requirement to test cases. So that it is with ease to see why a test case exists and to be able to react, when a requirement changes. With 2. test coverage, I dont expect to get a % number such for unit tests, but to get a general overview how many test cases cover a requirement (or maybe even none). I agree with you regarding 3 and I will take a look at e.g. Jenkins... thanks for that – emersoncod Apr 9 '18 at 11:30
  • If you write your bdd tests carefully you can create reusable parameterised steps that can called from many tests, this way your features usually only change the parameters. your feature files can contain annotations to describe their purpose if isn't otherwise clear enough – Amias Apr 9 '18 at 13:40

I have worked with 2 examples of this - The Atlassian Suite (Jira) and VSTS (Microsoft)

With both systems I have features and stories with tickets on a board. When work starts on the ticket I make a branch, then a Pull Request, then the ticket gets merged.

All this work is connected into two ways:

  • Links from the Story to the Pull Request within the ticket management system
  • Manual linkage to code level artifacts by putting the story number / identifier at the start of all branch names and commit messages in git.

I call this system an auditors dream for compliance as it lets one link all these systems together and trace code to business change to testing.

In terms of " trigger tests from the tool and retrieve the results." I'm not quite following that but what I do know is that with the above systems you link the tickets and code pushes to automatically run the tests in your CI environment and display the results and allow this to be part of a devops pipeline that runs tests and promotes code on success.

Atlassian Suite: https://www.atlassian.com/

VSTS: https://www.visualstudio.com/team-services/

Atlassian is my personal favorite for a number of reasons I am not detailing here (strong opinions but opinions non the less).

  • + 1 for Atlassian Suite(Jira), they have very flexible ways to connect with other external tools. – Vishal Aggarwal Apr 20 '18 at 11:10

You may want to give Squash Test tool suite a closer look. There is many components :

  • Squash TM (Test Management). It offers all you are asking for a TMT, and more. There is an online demonstration to try it.
  • Squash TA (Test Automation). It is automated testing framework. It does not provide yet another testing robot, but it integrates best of the breed open source robots such as Selenium, Sahi, SoapUI, etc.) and a plugin structure to call others robots and testing tools (UFT, Ranorex, Neoload, etc.). Squash TA is based on Jenkins and use a VCS (anyone recognized by Jenkins) to store automation scripts. Somehow, you may see it as the "Linux Distribution" for automation specialists
  • Squash4Jenkins is a Jenkins plugin to push automated tests results back to Squash TM, as long as results are in JUnit format.

Inside Squash TM, you are able to declare a test case as automated and select a predefined tool and an existing script for running automated test (such as Squash TA ;-). Results are then pushed back to TM. Finally, you may get reports for your testing campaigns that includes manual and automated tests.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.