Which one is a better tool to run web service tests?
I should have the following flexibility:
- Parameterize my requests.
- Integrate with Maven.
- Better test management.
Stack Exchange network consists of 181 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.Visit Stack Exchange
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.Sign up to join this community
Both fit your requirements so feel free to choose the one you like more.
The only area where JMeter is better is running load tests, SoapUI has limited load testing capabilities and reporting (you have only mix, max and average response time per request) while JMeter tests can scale to run in clustered mode and you can have HTML Reporting Dashboard with a way more informative tables and charts. See Testing SOAP/REST Web Services Using JMeter article for more details.
Maybe just use plain JUnit tests? I think JMeter and SoapUI just add an extra layer of complexity. For example making it more complex to schedule the tests. Using JMeter and SoapUI might seem easier for less technical testers, but in my experience these tools are pretty awful in their design. Most functionality can easily be implemented with plain code.
Plain JUnit tests are easy to parameterize, work perfectly with Maven and should work with most test management tools. Also the product is probably already using JUnit, making it easy to integrate into your continuous testing infrastructure.
Some examples to test Rest/API web services with JUnit:
I use both tools one a frequent basis. I use SoapUi for functional testing of web services and I use JMeter for load.
What follows is merely my own opinion, so please don't shoot me down....
I really like SoapUi's GUI and it's ability to create data-driven (parameterised) tests. I like the 'out of the box' assertions for checking responses are as expected. These include postive and negative response checks. E.g. calling a services with an invalid payload returns a 500. Or may return a 200, with a meaningful error message.
I use JMeter for load testing the web services.
Why two tools? Well, when I started testing services, I firstly used SoapUi, but when it came to load testing, I found the functionality lacking. So, I simply used JMeter for load testing. I could probably do a lot more in JMeter, but I personally don't find the UI as nice as SoapUI's. Plus, I invested a reasonable amount of time in building my tests in SoapUi so not keen on ditching all that just to use one tool.
I believe SoapUi can be integrated into Maven and CI. Once you have built your tests, you certainly don't need to use the UI to just run them. Test execution can be done from the command line if you wish. Not saying JMeter doesn't offer this, I simply haven't had the need to try.
Other people on here have mentioned simply coding your web service tests. I think that is a perfectly valid approach. Plus, there are lots of other tools available for such tests. E.g. Postman.
To sum up, try the different tools and don't restrict yourself. If one tool works for you, then great. If you prefer to use a range of tools for different testing aspects, then I think that's fine too.
There really isn't a right or wrong way. Try the tools yourself and if you can achieve what you want, then you have found the right toolset.
We have detailed user reviews for both SoapUI and Jmeter on IT Central Station. You can see a comparison page here.
Using this solution, we developed tests about three times faster than we did with Java and by doing manual REST reverse engineering from product code (Java). This solution also enabled manual testers to write automated tests, which they would not have been able to do in Java. We are planning to get SoapUI tests written earlier (thus moving some test development upstream).
Working with Git needs to be improved. SoapUI 1.8.0 has this annoying habit of touching/changing many files that are unrelated to a test developer's actual change. This habit complicates working with Git, since Git thinks that the user has made hundreds of changes.
If you are also looking at other options, users on our site most often compare these solutions to ParaSoft SOA. Reviews are here.
Disclaimer: I work at IT Central Station