12

Goal: I have an enterprise application written in java to test for performance. I need to do it separately, so I will be basically simulating a user interaction over http/https.

My findings:

  • Selenium is not suitable for load testing. Even with threading, Selenium does not seem to suit a lot of requests simultaneously.

  • Jmeter it most recommended solution, but scripting with it will add a learning curve.

  • OpenSTA seems to be left over after 2007, otherwise it is a good tool

  • Recently came across gatling. They claim to overcome shortcomings of other tools including Jmeter

Secondary requirements/scenario:

  1. I have a selenium expertise in my team and also several scripts are ready for regression tests. So possibly I'd like to re-use them.

  2. I am not willing to have a paid solution until I can get my job done perfectly with open source tools/API even if it needs more efforts on my side.

So, please suggest your solutions/tools. I still think I'll end up using Jmeter..but suggestions are needed.͏͏͏͏͏

4

5 Answers 5

13

I have had some luck using Grinder. It is Java based but you can also write your scripts in Jython or Clojure.

You said that you would like to take advantage of your team's current Selenium scripts and expertise performance testing. You did not describe your regression test scripts, but you may want to reconsider whether they are appropriate. In particular, you should consider two issues: is Selenium appropriate for performance testing and are functional tests appropriate for performance testing?

First, as you may have realized, Selenium is a very resource-intensive way to induce load on a system. Let me quote from the first question in the Selenium Grid FAQ:

Q: Would you recommend using Selenium Grid for performance/Load testing?

A: Selenium Grid is not designed for performance and load testing [...] The main reason for this is that conducting performance/load testing with real browser is a pretty bad idea as it is hard/expensive to scale the load and the actual load is very inconsistent.

[...]

To simulate 200 concurrent users for instance, you would need 200 concurrent browsers with a load testing framework based on Selenium Grid. Even if you use Firefox on Linux (so the most efficient setup) you will probably need at least 10 machines to generate that kind of load. Quite insane when JMeter/Grinder/httperf can generate the same kind of load with a single machine.

Second, there is a tension between the needs of a functional test and a performance test. A functional test focuses on correct behavior under a variety of circumstances, including some that may happen rarely but which are important to the user's experience. A performance test focuses on how the system responds under expected loads. If your functional test does not model a typical user's workflow, it does not belong in the performance test. For example, it may be important to know that a page generates a nice error message when you enter a 2-digit phone number, but it is pointless to ask what happens when 100 users do that at the same time.

Even if a functional test does model typical user behavior, it may still be inappropriate for a performance test. Functional tests focus on processing a set of inputs and checking whether the results match their expected values. If a test involves a number of steps, it may also check intermediate results for diagnostic purposes. In my experience, performance tests dispense with all that checking because it slows down the performance test. (To be more precise, there may be a few threads that check expected vs. actual results but most of the threads just try to load down the system without worrying about correct results.)

5
  • Agree, also I have had some success using Selenium to write one off performance tests (not re-using existing functional tests) for the sole purpose of measuring performance while under load.
    – Sam Woods
    Commented Dec 19, 2012 at 18:06
  • your explanation about the difference between functional test and performance test scrip is awesome!
    – Tejas
    Commented Dec 20, 2012 at 11:24
  • I drop the plan to use Selenium at all, because I'd be generating a huge load, really really huge!
    – Tejas
    Commented Dec 20, 2012 at 11:26
  • @user246, What if the functional test itself requires a lot of users? Then wouldn't Selenium fail too in such a functional test?
    – Pacerier
    Commented Sep 18, 2015 at 10:22
  • Functional tests typically exercise branches in conditional logic. If "lots of users" means multiple users each with a different role, participating in a workflow, you would presumably cover that with Selenium. If you mean "hundreds of users", you are no longer testing conditional logic. Web applications do not have "if there are less than 200 users, do this; otherwise do that" conditional logic, so you no longer have a functional test. Of course you are free to define terms differently, but there is still a difference between testing conditional logic and testing aggregate behavior.
    – user246
    Commented Sep 18, 2015 at 18:49
3

I tried my hand with Gatling, and I would rate it way way over Jmeter.

  1. Biggest advantage is you get to write some code for a load test- introduces type safely, re-usability and maintainability.

  2. Scala is not hard to learn for a java developers.

  3. You can generate higher load without actually choking your own machine.

  4. Performance of Jmeter itself decreases with higher load. This will generate incorrect results for load tests. Gatling is fairly new, but awesome.

You can refer to it from it's website. I totally recommend it and I've also made a point and convinced my management to use it

http://gatling-tool.org/

2
  • I would add also JMeter API is way too verbose, while Gatling tests are consise and much more leggible. This impacts also learning curve: Gatling seems easier for me.
    – dzieciou
    Commented Apr 5, 2016 at 6:16
  • For those looking - new gatling URL: gatling.io/docs/gatling/tutorials/installation
    – Leighton
    Commented Jun 2, 2022 at 3:03
2

jmeter

I know it sounds like I'm beating a dead horse here, but this is the most commonly used open source tool. There is documentation all over the internet about using and setting it up. Additionally, if your company is a java shop, there are probably many developers who could help your team get started.

2
  • yeah, we run a java shop. So I am considering Jmeter and gatling and now, even grinder for final selection.
    – Tejas
    Commented Dec 20, 2012 at 11:25
  • 1
    This lures me to use gatling instead of jmeter github.com/excilys/gatling/wiki/Benchmarks
    – Tejas
    Commented Dec 21, 2012 at 7:34
2

I ll suggest Jmeter for Performance...

As the name indicates Jmeter is one of the Java tools which is used to load testing client/server applications. Earlier it was used for testing Web Applications only however now-a-days its being used for other test functions. It is typically used to measure performance and to load test functional behavior of client-server applications. It can also handle FTP and JDBC requests. You must be aware of such other tools like WinRunner. You will be glad to know that as compared to WinRunner, Jmeter is easy to use due to its simple and intuitive GUI. Moreover its absolutely free and can be modified at ease as its an open source.

The important functionalities of Jmeter is that a heavy load on a server can be stimulated by using it, not on a server but also a heavy load on a network or object to test its strength under different load types. A graphical analysis of performance can also be done by using Jmeter or the behaviour of your server/script/object can also be tested under heavy concurrent load.

1

Since you are not willing to use paid tools and already agree that JMeter is highly recommended as an open source solution for performance testing. Yes it may add a learning curve and can be hectic to write the test scripts. BUT that is not really the case now!

Since Blazemeter has emerged I have been using JMeter in combination with it with a great easy and almost no scripting unless I want to change certain configuration options.

Blazemeter has an extension in Google Chrome. You can use this to record your test script and then use that in JMeter to run the tests. Blazemeter extension will records all your HTTP(s) requests as you browser through your application under test along with all the required login credentials, cookies and sessions. Once you are done recording your script you can export it as a ".jmx" file, which you can open in JMeter to run the tests. You can also make all the changes you require to your test plan in JMeter.

Not the answer you're looking for? Browse other questions tagged or ask your own question.