Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Goal: I have an enterprise application written in java to test for performance. I need to do it separately, so I will be basically simulating a user interaction over http/https.

My findings:

  • Selenium is not not suitable for load testing. Even with threading, Selenium does not seem to suit a lot of requests simultaneously.

  • Jmeter it most recommended solution, but scripting with it will add a learning curve.

  • OpenSTA seems to be left over after 2007, otherwise it is a good tool

  • Recently came across gatling. They claim to overcome shortcomings of other tools including Jmeter

Secondary requirements/scenario:

  1. I have a selenium expertise in my team and also several scripts are ready for regression tests. So possibly I'd like to re-use them.

  2. I am not willing to have a paid solution until I can get my job done perfectly with open source tools/API even if it needs more efforts on my side.

So, please suggest your solutions/tools. I still think I'll end up using Jmeter..but suggestions are needed.

share|improve this question
1  
I agree with user246's answer. You may also want to look at this: sqa.stackexchange.com/questions/3551/… –  Sam Woods Dec 19 '12 at 18:05
    
This lures me to use gatling instead of jmeter github.com/excilys/gatling/wiki/Benchmarks –  Napster Dec 21 '12 at 7:35
    
I haven't used Gatling, but from what I have read, it should meet your needs. –  Sam Woods Dec 21 '12 at 18:31
add comment

4 Answers

I tried my hand with Gatling, and I would rate it way way over Jmeter.

  1. Biggest advantage is you get to write some code for a load test- introduces type safely, re-usability and maintainability.

  2. Scala is not hard to learn for a java developers.

  3. You can generate higher load without actually choking your own machine.

  4. Performance of Jmeter itself decreases with higher load. This will generate incorrect results for load tests. Gatling is fairly new, but awesome.

You can refer to it from it's website. I totally recommend it and I've also made a point and convinced my management to use it

http://gatling-tool.org/

share|improve this answer
add comment

I ll suggest Jmeter for Performance...

As the name indicates Jmeter is one of the Java tools which is used to load testing client/server applications. Earlier it was used for testing Web Applications only however now-a-days its being used for other test functions. It is typically used to measure performance and to load test functional behavior of client-server applications. It can also handle FTP and JDBC requests. You must be aware of such other tools like WinRunner. You will be glad to know that as compared to WinRunner, Jmeter is easy to use due to its simple and intuitive GUI. Moreover its absolutely free and can be modified at ease as its an open source.

The important functionalities of Jmeter is that a heavy load on a server can be stimulated by using it, not on a server but also a heavy load on a network or object to test its strength under different load types. A graphical analysis of performance can also be done by using Jmeter or the behaviour of your server/script/object can also be tested under heavy concurrent load.

share|improve this answer
add comment

jmeter

I know it sounds like I'm beating a dead horse here, but this is the most commonly used open source tool. There is documentation all over the internet about using and setting it up. Additionally, if your company is a java shop, there are probably many developers who could help your team get started.

share|improve this answer
    
yeah, we run a java shop. So I am considering Jmeter and gatling and now, even grinder for final selection. –  Napster Dec 20 '12 at 11:25
    
This lures me to use gatling instead of jmeter github.com/excilys/gatling/wiki/Benchmarks –  Napster Dec 21 '12 at 7:34
add comment

I have had some luck using Grinder. It is Java based but you can also write your scripts in Jython or Clojure.

You said that you would like to take advantage of your team's current Selenium scripts and expertise performance testing. You did not describe your regression test scripts, but you may want to reconsider whether they are appropriate. In particular, you should consider two issues: is Selenium appropriate for performance testing and are functional tests appropriate for performance testing?

First, as you may have realized, Selenium is a very resource-intensive way to induce load on a system. Let me quote from the first question in the Selenium Grid FAQ:

Q: Would you recommend using Selenium Grid for performance/Load testing?

A: Selenium Grid is not designed for performance and load testing [...] The main reason for this is that conducting performance/load testing with real browser is a pretty bad idea as it is hard/expensive to scale the load and the actual load is very inconsistent.

[...]

To simulate 200 concurrent users for instance, you would need 200 concurrent browsers with a load testing framework based on Selenium Grid. Even if you use Firefox on Linux (so the most efficient setup) you will probably need at least 10 machines to generate that kind of load. Quite insane when JMeter/Grinder/httperf can generate the same kind of load with a single machine.

Second, there is a tension between the needs of a functional test and a performance test. A functional test focuses on correct behavior under a variety of circumstances, including some that may happen rarely but which are important to the user's experience. A performance test focuses on how the system responds under expected loads. If your function test does not model a typical user's workflow, it does not belong in the performance test. For example, it may be important to know that a page generates a nice error message when you enter a 2-digit phone number, but it is pointless to ask what happens when 100 users do that at the same time.

Even if a functional test does model typical user behavior, it may still be inappropriate for a performance test. Functional tests focus on processing a set of inputs and checking whether the results match their expected values. If a test involves a number of steps, it may also check intermediate results for diagnostic purposes. In my experience, performance tests dispense with all that checking because it slows down the performance test. (To be more precise, there may be a few threads that check expected vs. actual results but most of the threads just try to load down the system without worrying about correct results.)

share|improve this answer
    
Agree, also I have had some success using Selenium to write one off performance tests (not re-using existing functional tests) for the sole purpose of measuring performance while under load. –  Sam Woods Dec 19 '12 at 18:06
    
your explanation about the difference between functional test and performance test scrip is awesome! –  Napster Dec 20 '12 at 11:24
    
I drop the plan to use Selenium at all, because I'd be generating a huge load, really really huge! –  Napster Dec 20 '12 at 11:26
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.