SoapUI vs Jmeter for webservices

Question

Which one is a better tool to run web service tests?

I should have the following flexibility:

1. Parameterize my requests.
2. Integrate with Maven.
3. Better test management.

Answer 1

Both fit your requirements so feel free to choose the one you like more.

The only area where JMeter is better is running load tests, SoapUI has limited load testing capabilities and reporting (you have only mix, max and average response time per request) while JMeter tests can scale to run in clustered mode and you can have HTML Reporting Dashboard with a way more informative tables and charts. See Testing SOAP/REST Web Services Using JMeter article for more details.

Answer 2

Maybe just use plain JUnit tests? I think JMeter and SoapUI just add an extra layer of complexity. For example making it more complex to schedule the tests. Using JMeter and SoapUI might seem easier for less technical testers, but in my experience these tools are pretty awful in their design. Most functionality can easily be implemented with plain code.

Plain JUnit tests are easy to parameterize, work perfectly with Maven and should work with most test management tools. Also the product is probably already using JUnit, making it easy to integrate into your continuous testing infrastructure.

Some examples to test Rest/API web services with JUnit:

• https://dzone.com/articles/how-test-rest-api-junit
• https://www.javacodegeeks.com/2011/10/java-restful-api-integration-testing.html

Answer 3

I use both tools one a frequent basis. I use SoapUi for functional testing of web services and I use JMeter for load.

What follows is merely my own opinion, so please don't shoot me down....

I really like SoapUi's GUI and it's ability to create data-driven (parameterised) tests. I like the 'out of the box' assertions for checking responses are as expected. These include postive and negative response checks. E.g. calling a services with an invalid payload returns a 500. Or may return a 200, with a meaningful error message.

I use JMeter for load testing the web services.

Why two tools? Well, when I started testing services, I firstly used SoapUi, but when it came to load testing, I found the functionality lacking. So, I simply used JMeter for load testing. I could probably do a lot more in JMeter, but I personally don't find the UI as nice as SoapUI's. Plus, I invested a reasonable amount of time in building my tests in SoapUi so not keen on ditching all that just to use one tool.

I believe SoapUi can be integrated into Maven and CI. Once you have built your tests, you certainly don't need to use the UI to just run them. Test execution can be done from the command line if you wish. Not saying JMeter doesn't offer this, I simply haven't had the need to try.

Other people on here have mentioned simply coding your web service tests. I think that is a perfectly valid approach. Plus, there are lots of other tools available for such tests. E.g. Postman.

To sum up, try the different tools and don't restrict yourself. If one tool works for you, then great. If you prefer to use a range of tools for different testing aspects, then I think that's fine too.

There really isn't a right or wrong way. Try the tools yourself and if you can achieve what you want, then you have found the right toolset.

Answer 4

We have detailed user reviews for both SoapUI and Jmeter on IT Central Station. You can see a comparison page here.

Jmeter review from our community:

Valuable Features:

• Extremely light weight.
• Very easy to download and setup the entire infrastructure (Controller +LGs)
• Ease of scripting
• Host of plugins which further boost its effectiveness
• Availability Non-GUI mode which consumes even lesser resources than already lightweight GUI version.

Improvements to My Organization:

• Robust scripts which could handle build changes
• Easy LG(Master & Slaves) configuration setup
• Very lightweight and low usage of resources

Room for Improvement:

• GUI tends to freeze and shutdown under more load
• Ctrl+z( Undo doesn't work) so gets very inconvenient at times
• Cannot do a mass replace( Ctrl+H) on the GUI ( can do it if script is opened via notepad++)
• Re-iteration problems during errors - Usually we have login in once only controller, Action in Loop controller and logout in once only controller. Now if an error comes when the user simply logouts due to some unhandled error then the script expects that your session would continue but if that error has made you logout then your next iterations won't go through and vice versa. Suppose you put everything in loop controller and error doesn't log you out then the script start to login when you are already logged in and tends to fail.

SoapUI user review from our community:

Valuable Features:

I found the REST Discovery and Groovy scripting features to be most valuable. Additionally, the Simple programming model enables a wide audience (in contrast to Java or JavaScript). The XML format makes it easy for scripts to analyze the test base (metrics, bulk simple changes).

Improvements to My Organization:

Using this solution, we developed tests about three times faster than we did with Java and by doing manual REST reverse engineering from product code (Java). This solution also enabled manual testers to write automated tests, which they would not have been able to do in Java. We are planning to get SoapUI tests written earlier (thus moving some test development upstream).

Room for Improvement:

Working with Git needs to be improved. SoapUI 1.8.0 has this annoying habit of touching/changing many files that are unrelated to a test developer's actual change. This habit complicates working with Git, since Git thinks that the user has made hundreds of changes.

If you are also looking at other options, users on our site most often compare these solutions to ParaSoft SOA. Reviews are here.

Disclaimer: I work at IT Central Station