RESTful JSON Web API Testing - what tools?

Question

My company is working on its API which is RESTful JSON Web API. So far, all my testing was done using Selenium WebDriver. Starting from one and more tests I manage to create some sort of framework. This API will be also part of the web portal we use.

I was given task to look for some tools that can help us automate the API tests and also help in future tests.

So far I have looked up for HP and UFT with other set of tools. Also, SoapUI NG Pro seems be a good tools for this kind of tests.

Can you guys help me out here and point out some pros and cons based on your experience. Every help will be much appreciated. Thanks!

Answer 1

ReadyAPI/SoapUI

Pros

• GUI based so non-technical members can use it fairly easily
• Assertions are built in so validations are available for the most part
• Majority of use cases can be implemented without customizations
• You can leverage functional tests for load/performance with additional LoadUI license

Cons

• Cost - yearly license fee per user, per SmartBear product
• Reusability - lots of redundancy between test cases and test suites
• Can't fix issues with product - not open source. Need to wait for support team

Jmeter

Pros

• Open source
• Can create tests using GUI easily
• Tests can be easily modified to be used for load/performance testing
• Depending on webserver, can take production traffic and replay it for testing

Cons

• Adding customizations (plugins) not as easy as it would seem
• Community - sometimes hard to find solutions to problems
• Readability - looking through tests to understand what is going on can be tedious

Another option would be to use a BDD framework such as Cucumber. If your team is already writing tests using Selenium (and I assume a programming language such as Java), you can integrate these into the Cucumber framework, and you can add API tests here too. Its fairly easy to implement (if you are comfortable writing code), its free (Java/Cucumber/whatever http package you use), and it integrates very well with CI (Jenkins plugins), providing nice reports. Community is very involved and supportive. Tests are written in plain english using Gherkin, so anyone should be able to easily define scenarios and understand what tests are doing. Lastly, lots of repeatability - you can reuse steps very easily.

Answer 2

I would recommend Apache JMeter as it is:

• free
• open source
• can be used for load testing as well
• can be used for both RESTful and SOAP web services
• provides JSON Path Extractor and JSON Path Assertion for correlation and conditionally setting pass/fail criteria
• can be integrated with Selenium WebDriver so it will be a swiss-army-knife solution for all your tests.

Answer 3

POSTMAN is the defacto standard when it comes to REST API testing. Its pretty intutive GUI makes REST API testing pretty simple even for the novice testers. The only drawback for this tool is that you would have to do visual validation of your headers, response code, response body etc. There is no automated, batch type interface for creating batch tests. If you have complex verification of the response body, this would not be the right tool. You may also also want to try https://github.com/sassoftware/unravl that would give you a good framework for automated tests. There are plenty of open source project such as RESTAssured, Jerzy etc and you can create a TestSuite. Preferably you can integrate your WebDriver suite along with the REST API Suite

Answer 4

SoapUI (or SoapUI Pro if the extra features appeal to you) would suffice. You can create a test suite, that has all the test calls you want to make under it and execute the entire test suite and save reports from it. You can use groovy scripts to automate parameterisation, etc.

From a Performance testing point of view, I've used both HP Loadrunner and JMeter to automate/performance test API calls.

Answer 5

This is coming from a .NET background but the concepts apply to any language/environment.

I write my tests in Gherkin/SpecFlow which has a good syntax for viewing what the test is at a high level.

Scenario: GET /recipes
    Given I've set my authentication token
    When I get "/recipes"
    Then only my recipes should be downloaded

Once I've made the request my assertions are pretty simple. I look for the following pieces in this order.

1. The response's status code (200, 403, 422, etc...)
2. The JSON Schema (http://json-schema.org)
3. The data of the response

I use this order because the test will break at the faster/smaller assertions first. I.e testing the status code is faster than pulling data from the db and mapping it to JSON.

The assertion syntax is deeply influenced by Frisby.js with a syntax similar to the following.

ValidateResponse(lastResponse)
    .ExpectStatus(HttpStatusCode.OK)
    .ExpectJsonSchema(JsonSchema.Recipes.Detail)
    .ExpectJson(SerializedDataAsJson());

Answer 6

Late answer here, but I'm the dev of a new open-source framework called Karate.

Karate was created out of my frustration with the existing options for automating REST/JSON API tests. I'm glad to say that Karate has comprehensive support for all kinds of HTTP, including SOAP+XML and multi-part file-uploads. One of the key features of Karate is being able to express data 'natively' in JSON and comparing 2 JSON (or XML) payloads is an easy, single line operation.

I can say a lot more, but the site has a bullet-list of features that you can review.

Do consider evaluating Karate, and it would be great to get your feature-requests and feedback on the project site.

Answer 7

Do you want to test the feature or the performance of your API? To performance you can use JMeter, that is the more famous tool to performance tests.

To tests features and HTTP methods, you can use SOAPUI that is good, but is not free, if you are acquainted with javascript you can use frisby.js or chakram.

Answer 8

Besides all tools mentioned in other answers, you might also try HttpMaster.

It has some nice features for REST API testing like global parameters, data validations, request chaining, it also includes command line interface for test automation. Its GUI is quite nice and simple to use, but you should be running Windows since HttpMaster is only supported on Microsoft OS.

Answer 9

Disclaimer: I'm using APICombo

I hope I don't misunderstand your question, you can use APICombo to do JSON or XML response validation. You can also schedule the test to run so it can check automatically for you. You can check the documentation here. They also provide some statistics, like how your API performs globally, which measures the response time. It will also notify you in case the test fails trough email, SMS, webhook or even other integrations like pushbullet, pushover, etc.

Answer 10

Try, tool called Swagger UI for manual testng - Easy to use, Nice UI, Good technical documentation around the tool.

Answer 11

Currently I am using different approach automate rest services.

Coverting Json schema / Json files to java Classes with help JSONschema2Pojo. 2.Using Spring Rest Template to make requests and get Responses as Java Object.

TestRestTemplate rest = new TestRestTemplate(); ResponseEntity response = rest.getForEntity("URL", Request.class); //From here you can get http Headers as HttpHeaders headers = p.getHeaders(); //Response body as ResponceObject resBody = response.getbody();

On top of this if We can use cucumber. we can have a lot reusability of code.

This is more Object Oriented way. I strongly disagree creating manual Json files

More information can be found at. http://www.jsonschema2pojo.org/ http://www.baeldung.com/spring-boot-testresttemplate

I am thinking creating sample git project on this.

Answer 12

This is an area that has experienced a lot of development in recent years, so I'm posting an answer with updated content for anyone coming to this question.

The classic framework for automated API test and validation is Dredd, which works by generating tests from the API specification. Dredd can be confusing at the beginning, so you may find this article useful.

These days, I'd personally recommend schemathesis. Schemathesis works by generating tons of requests with all possible values from your schemas. You can also extend schemathesis to run custom specific tests against your API.

Schemathesis uses an approach called property-based testing. In property-based testing, you use a framework that generates test data based on a number of properties which you define, then run your functions/methods against the data, and verify the result based on its properties. There're libraries for property-based testing for most languages. You can leverage these libraries to define strategies that automate the test generation process for testing and validating your APIs.