1

I'm about to script performance tests for a system with REST interface. I'm looking for a performance tool that let me prepare XHR requests dynamically, i.e. simulate what JavaScript application does. For instance, to request a CSRF token from one REST endpoint and include it in a header of a subsequent HTTP request. The tool should also handle cookie-management just like any Web browser does.

Additional nice-to-have requirements:

  • the tool should allow scripting tests in Java, as most of the team is skilled in Java

So far I have considered multiple tool but each of them has some drawbacks:

  • JMeter, apart from nice GUI, provides also an API but it seems quite rusty and cumbersome to use. Also, I'm not sure if I can define requests dynamically, based on the results of previous request. Does it?
  • Grinder is Java framework, but test cases must be scripted in Jython. This requires the team to learn Python and also introduces some incompatibility problems: I had hard time to reuse some existing Python libraries. On the other side, I needed to pay some more attention when casting Java objects into Python. Why, the hack, guys couldn't make Java framework with Java test cases?
  • Gatling has quite intuitive DSL for defining requests programmatically (almost like REST-Assured), but is in Scala. That's a cool language but again we're not proficient in it.
  • I wanted to reuse framework for functional REST tests, REST-Assured, but I've read, it introduces much overhead to the performance tests + I would need to add support for reporting myself.

Any other alternative I missed?

Improve this question
0

7 Answers 7

Reset to default
5

JMeter is a good choice. Kindly find answers on your queries below:

Also, I'm not sure if I can define requests dynamically, based on the results of previous request.

You can. Particularly for REST web services I would recommend installing Extras with Libs Set of JMeter Plugins project which provides JSON Path Extractor so you will be able to work with response

to request a CSRF token from one REST endpoint and include it in a header of a subsequent HTTP request

Absolutely the same approach, see How to Load Test CSRF-Protected Web Sites guide for more information and real-life examples

The tool should also handle cookie-management just like any Web browser does.

It is, just add HTTP Cookie Manager to your test plan

JMeter also includes JSR223 Sampler with full access to JMeter and underlying Java APIs so if you need to write custom code in your test you can put it there.

Improve this answer
1
  • Is it just me that found JMeter API (not UI) quite old-school, without using generics, modern collections and thus of low usability? I guess not as I've seen some solutions that wrap JMeter with another Java DSL just to make JMeter more usable.
    – dzieciou
    Apr 25, 2015 at 11:24
3

JMeter, Grinder and Gatling all do what you want to do out of the box. If you get into the documentation, or find walk-throughs online, you can find examples for all of them. The common term in most performance tools to pull a value from one request and store it as a variable for future requests is extraction.

JMeter is pretty extensible, and is what my team is using with a bunch of custom extensions we have added to add a few features that are missing out of the box that other tools such as Load Runner or Visual Studio Load Test have out of the box.

I would caution against doing TOO much logic to recreate what the javascript on a page does. Simple things like extracting a CSRF token and using it in a subsequent request is pretty standard, but if your javascript is for example creating a complex JSON object and sending it as the body of a post or a put, you probably don't want to use the performance tool to dynamically do that. Writing code that does that can cause your performance tests to slow down and require a lot more users in order to simulate the load that you need to be able to generate, then you would need to find a bunch of additional hardware or virtual machines to use as clients to generate your load.

A better approach would be to have a setup phase that creates a bunch of request bodies ahead of time, or even have a set of hard coded request bodies that you can use, then when the test is running it just pulls from those pre-created bodies. This allows you to scale your tests much more easily.

Improve this answer
1

I would also recommend you to try coding to generate custom requests. This would help automate functional tests, as well do some decent perf tests. I was able to generate upto 50 req/ sec

This is example code, you can customize as needed.

Improve this answer
5
  • I guess, using Apache low-level HTTP client does not bring much overhead to the overall results of the test. Have you compared what would happen if instead you were using some higher abstraction like RESTAssured or Retrofit?
    – dzieciou
    Apr 11, 2015 at 7:19
  • 1
    I am trying to learn jmeter source code, approach for performance. This was part of learning exercise to quickly develop load test. Multiple instances of this executable can easily scale / generate required load. Once I check I can answer in detail your question.
    – Siva
    Apr 11, 2015 at 8:58
  • The problem with standard implementation of CloseableHttpClient is that it is synchronous client. So if each of your threads is doing one or more requests that blocks connections for long then they will block also your threads. So even if I planned to run 200 connections concurrently and I have CPU with only 4 cores then those connections were not run with such concurrency. The problem is even bigger if you organized each task per thread in such a way each task performs multiple connections in a row. The solution would be to use non blocking HTTP client, e.g., some async client.
    – dzieciou
    Apr 28, 2015 at 19:13
  • This was a demo code which I had tested in AWS. This would vary based on the application. Yea for applicable cases we can also go for async client.
    – Siva
    May 7, 2015 at 4:49
  • Check this link slideshare.net/jweden/rest-performance-testing. Non blocking http client library would solve the scale up challenge.
    – Siva
    May 25, 2015 at 10:17
1

You didn't say the tool needs to be open-source and/or free. That would add a bunch of commercial tools to your list like Loadrunner and Rational Performance Tester

Both LR and RPT allow you to script in Java, but I believe if you "record" with loadrunner, the script would be in LR scripting language, which is C-like. Both tools have powerful libraries for capturing and reusing tokens, doing asserts on responses and managing cookies.

Improve this answer
3
  • The default LoadRunner language is not "C Like." It is ANSI C. You can convert a standard script to it's Java or VB equivalent if you are more comfortable working in those languages with the associated trade off in virtual user resource weight which comes with those languages.
    – James Pulley
    May 7, 2015 at 13:00
  • @JamesPulley Does LR have a built-in way to convert a recorded script into a Java Vuser script?
    – RaGe
    May 7, 2015 at 13:41
  • Yes, covered in product documentation and online help
    – James Pulley
    May 7, 2015 at 17:41
0

Take a look at LoadRunner from HP or Cloud Test Lite from SOASTA. You can pull your JavaScript code into either tool to use to prep your XHR requests. Both tools have an excellent track record for both REST and SOAP support as well as emulation of standard browser capability for cookies and the ability to extract data from a response to place into a subsequent request.

The native language for LoadRunner is C. You can transpose the code to Java or VB if you wish with the associated higher resource cost per virtual user. For CloudTest it is JavaScript.

Both tools are no cost at the lowest license level, up to 50 users for LoadRunner and up to 100 threads to CloudTest. Keep in mind a real browser would have more than one download thread.

Improve this answer
-1

I was looking too for a nice JAVA library for such REST testing. But finally, I discover requestjs working with nodejs. It's really easy to use. For instance,

request("http://your.url", {qs: param1: value1})
Improve this answer
4
  • But it requires my team to learn and maintain yet another programming language, right?
    – dzieciou
    Apr 10, 2015 at 14:40
  • If they don't know javascript indeed. If the project you plan to build is not big or complex and you just want a simple REST shooter, this requestjs is really straightforward.
    – tetienne
    Apr 10, 2015 at 15:40
  • 1
    RequestJS is a pretty good library for testing web requests, but it does not have any of the out of the box functionality listed in the question about running massive amounts of requests concurrently and managing user load across multiple agents that you typically need for performance testing.
    – Sam Woods
    Apr 10, 2015 at 16:22
  • I downvoted this answer because it answers the question, "What is a Javascript library for testing REST requests?", which is not the question the OP posed.
    – user246
    Apr 20, 2015 at 17:55
-2

Firsly, I work at IT Central Station, a crowdsourced platform for real user IT reviews. Interesting comparisons, if you're looking at JMeter, Gatling and other similar tools you may want to check out a comparison page where you can see real user feedback on JMeter and HP StormRunner that speak specifically about script performance tests and may be useful.

Improve this answer
2
  • Hi again! I bet you can guess what this comment will have: the same thing as the other one. Except that this one doesn't declare affiliation. (Yeah, it has to be in every post, because a lot of people will only ever see the one.) I'd love to see what these posts turn into if you are able to clean them up, otherwise I'll have to remove them. =[
    – corsiKa ♦
    Jul 27, 2015 at 22:18
  • Hi, Thanks for the feedback, added the disclaimer and made the review available without registration. Hope that helps!
    – abigail
    Jul 28, 2015 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.