Zephyr vs Xray vs TestRail

Question

We are thinking of implementing a test management tool for our test cases which are currently in Excel sheets. So far we have zeroed in on these three tools. Some of the features we are looking for are:

• integration with JIRA
• ability to import existing test cases from Excel sheets
• cost effective
• ability to integrate automated tests
• quality of the JIRA plugin
• in case of Zephyr and Testrail, is it advisable to use the JIRA plugin only without using their enterprise service?

It would be good if someone could provide a Matrix Comparison for these tools based on these features.

Answer 1

We are using the Jira addin with locally running TestRail without any issues. What I like about Test Rail is the ease of use. We went from install to all QA team using it in a few weeks. Also Gurock respond very quickly to development requests.

Mark Smith

Answer 2

We use TestRail.

• Integration with JIRA. We just started using this, I've linked many tickets to my test cases. I haven't had bugs linked yet, but it seems like it will be useful for 'more information' and not a goto.
• ability to import existing test cases from Excel sheets. There is an import function. Somehow I don't see this being easy depending on how your sheets are setup.
• cost effective. We compared with Zephyr and didn't see an advantage cost wise with either. We have about 12 people on QA and about ~120 total at the company.
• ability to integrate automated tests. We have one project running an automation suite that updates TestRail. So far it's pretty nice. As long as your tests are up to date, you can get a nice little report.
• quality of the JIRA plugin. Nothing awesome, but I didn't feel like Zephyr was incredible either.
• in case of Zephyr and Testrail, is it advisable to use the JIRA plugin only without using their enterprise service? As far as TestRail goes, I would use their cloud service. However I don't have experience setting it up locally etc.

Answer 3

Dennis from TestRail here. I thought it might be useful to know why we have many teams regularly migrating from basic JIRA add-ons to TestRail and there are various reasons teams migrate to or choose TestRail.

TestRail offers the best of both worlds: a fully featured test management application that provides all the features needed for teams to be effective and productive. And at the same time we have a rich JIRA add-on integration to view linked results, test cases and reports inside JIRA. TestRail is the only application offering this kind of combination, and because of this we have a lot of teams switching to TestRail every week.

We also have many teams migrating from the various available basic JIRA add-ons, as they usually don't support even very basic features required for teams to be productive that you can find in a full test management tool (e.g. very basic things like grouping tests in sections, maintaining test orders, reusing tests easily, running tests against different platforms such as browsers/OS/platform configurations, rich automation APIs, forecast reports etc). E.g. TestRail's FastTrack interface alone helps teams save a lot of time every week:

• https://blog.gurock.com/testrail-5-1-introducing-testrail-fasttrack/

You can learn more about other features why teams switch here, so you can use this for a comparison matrix you mentioned, as well as our rich JIRA integration that supports full coverage between requirements/use cases in JIRA, test cases and test results in TestRail, as well as defects/issues in JIRA:

• http://www.gurock.com/testrail/tour/modern-test-management/
• https://blog.gurock.com/testrail-5-0-jira-test-management/

TestRail is also in the very unique position that we fully support both JIRA Server and JIRA Cloud, and TestRail was even featured for this on the Atlassian/JIRA blog:

• https://developer.atlassian.com/blog/2016/02/universal-addons-for-jira-server-cloud/

I hope this helps!

Answer 4

I'm surprised that nobody in this thread has mentioned open source solutions in this area.

I have been using commercial solutions for years, but opensource tools like Testlink are always a good solution.

Especially since even commercial solutions don't provide everything you might need, or ma has to buy it via plugins.

Testlink Tutorials https://www.guru99.com/testlink-tutorial-complete-guide.html

Testlink CI integration - Jenkins https://wiki.jenkins.io/display/JENKINS/TestLink+Plugin

Testlink Jira Integration https://community.atlassian.com/t5/Jira-questions/How-do-we-integrate-Testlink-1-9-14-with-JIRA-Agile-7/qaq-p/211955

Testlink Excel Import/Export https://www.tutorialspoint.com/testlink/testlink_import_data.htm

Testlink automated Test integration https://medium.com/@hirosht/integrate-the-testlink-with-an-automation-framework-e3b6a70d1db1

Further benefits

• Many plugins for extension
• A large community to find quick solutions to problems
• No additional costs except setup (hardware/manpower)

Answer 5

We use PractiTest as our test management tool, integrated with Jira, and we are pretty happy with the capabilities of this tool. It was relatively easy importing our test cases from a spreadsheet and the integration suits our needs. I think that the price is fair for what we get from the system. I know that PractiTest does offer integration with automation frameworks, though we didn't have the chance to use this option yet.