Test Management Tools and automated tests - how to integrate the two?

Question

I’m desperately looking for a test management tool that fully supports automated tests. As this sounds simple, I still fail to do so.

We have a bunch of automated tests, which are run as part of our continuous integration pipeline. This is fine and works as wanted, but we miss the benefits of test management tools, such as

• mapping requirements with test cases
• getting test coverage for requirements
• linking test cases/test results with issues – easy retesting of fixed reported issues

and so on and so on. So simply said, the good things about a test management tool.

So I have been trying to find one, but it seems that all the ones I've found support mostly manual testing and/or have only limited support of automated testing.

Duplication of test cases

We have now a bunch of test cases already defined in natural language (via http://gauge.org). I want to integrate those into the TMT and not define them again, to avoid having duplicated my test cases and maintain two (or more) copies of them.

Our test cases reside in git, and I have not found a tool to import test cases from git.

Automation results

All TMT's offer a workflow to add results in an intuitive manner (for manual tests), but I have not found a tool that supports automation, like trigger tests from the tool and retrieve the results.

All that is quite frustrating and I cannot think that it is only me fighting this. Does anyone have any experience with any tool for automated tests?

Please do not simply post links; please include details about why a particular tool helps and how this tool would help me in the problems described above.

Answer 1

When doing automated tests I have always been wary of test management tools especially since the advent of BDD. Every single one I have used has caused major problems with being unreliable or inflexible, once you have to start working on tests outside it then it has failed.

My philosophy with testing is to always keep things as simple as possible and this extends to choosing tools and processes.

1. mapping requirements

If you are using BDD well then you should have feature files from your PO that define test requirements; if these are not good then you should look in to improving this process. I find 3 amigos sessions can really help with this and give a chance to refine your requirements before bigger commitment to them is made.

2. Test coverage

Test coverage is almost impossible to do properly with automated tests as they typically combine multiple components, coverage really only applies with unit tests where you can count from the dispatcher where the test is going. I find that where I need coverage analysis I add it to Jenkins jobs. If I need to do anything bigger around coverage I use sonarqube but it can be misleading.

3. Result reporting

As long as you can output your results in XUnit then it's possible to get a reasonable display of historic test results. I have found that the new Jenkins blue ocean UI is an easy way to get most of this. Most CI tools have plugins that will help but wont be as constraining as a full TMT.

So I guess in summary, be careful what you wish for: it may become your prison :)

Answer 2

I have worked with 2 examples of this - The Atlassian Suite (Jira) and VSTS (Microsoft)

With both systems I have features and stories with tickets on a board. When work starts on the ticket I make a branch, then a Pull Request, then the ticket gets merged.

All this work is connected into two ways:

• Links from the Story to the Pull Request within the ticket management system
• Manual linkage to code level artifacts by putting the story number / identifier at the start of all branch names and commit messages in git.

I call this system an auditors dream for compliance as it lets one link all these systems together and trace code to business change to testing.

In terms of " trigger tests from the tool and retrieve the results." I'm not quite following that but what I do know is that with the above systems you link the tickets and code pushes to automatically run the tests in your CI environment and display the results and allow this to be part of a devops pipeline that runs tests and promotes code on success.

Atlassian Suite: https://www.atlassian.com/

VSTS: https://www.visualstudio.com/team-services/

Atlassian is my personal favorite for a number of reasons I am not detailing here (strong opinions but opinions non the less).

Answer 3

You may want to give Squash Test tool suite a closer look. There is many components :

• Squash TM (Test Management). It offers all you are asking for a TMT, and more. There is an online demonstration to try it.
• Squash TA (Test Automation). It is automated testing framework. It does not provide yet another testing robot, but it integrates best of the breed open source robots such as Selenium, Sahi, SoapUI, etc.) and a plugin structure to call others robots and testing tools (UFT, Ranorex, Neoload, etc.). Squash TA is based on Jenkins and use a VCS (anyone recognized by Jenkins) to store automation scripts. Somehow, you may see it as the "Linux Distribution" for automation specialists
• Squash4Jenkins is a Jenkins plugin to push automated tests results back to Squash TM, as long as results are in JUnit format.

Inside Squash TM, you are able to declare a test case as automated and select a predefined tool and an existing script for running automated test (such as Squash TA ;-). Results are then pushed back to TM. Finally, you may get reports for your testing campaigns that includes manual and automated tests.

HTH.