Load Testing, Selenium Vs JMeter Vs other tools

Question

I have been using Selenium/Webdriver for functional testing in a grid setup. We are in the process of transitioning to a Solr based search service. I already have Solrmeter for load testing the Search engine core, load UI for load testing the Rest-ful service. We also would like to load test the entire end to end service chain, i.e

User interface -> Restful service -> Solr core.

My boss wants me to use Selenium/WebDriver for this end to end load testing so that it becomes integrated with our core testing framework.

I am skeptical about it because WebDriver does not have supporting tools for load testing. The easiest way I can think of doing it in Selenium/WebDriver is to code timestamps in my scripts at the time when the query is sent and when the results are displayed on the webpage. Then write some reporting mechanism to gather and display reports. Also I imagine selenium would be slower.

On the other hand members from the stackoverflow community have suggested using JMeter for the same which I assume is less of an hassle. Can somebody give me pros/cons of Selenium/WebDriver Vs JMeter for load testing in this scenario?

Answer 1

Similar questions have been asked a number of times. The general consensus is that UI automation tools such as Selenium are not a good choice for load testing. There are a number of reasons why, but the main one is that it is a LOT slower. You can easily spin up thousands of concurrent requests in a second in a tool like JMeter which to reproduce in Selenium would take multiples of thousands of servers all orchestrated to give you the same amount of traffic. There are tools like selenium grid or browsermob which would allow you to orchestrate those kinds of tests, but it is really inefficient compared to a tool like JMeter.

Performance testing is a little different, since measuring the time it takes for UI to load or complete some operation can be done with Selenium, however you usually execute performance testing in tandem with load testing to answer questions like: How well does X action perform when the server is under no load/medium load/high load/varying load, etc.

As far as tools that are appropriate for load testing, JMeter is a very popular free tool, but there are a number of others including one I prefer - Visual Studio Load Testing (which requires a visual studio license which you may or may not already have).

Answer 2

You could also check out Gatling, which is an open source stress tool based on akka.

On the Gatling homepage, you'll find a direct comparison of how your test specification will look in Gatling vs. in JMeter. This might help you decide between these too.

If you want to create massive amounts of load, I think there's nothing better than akka as an engine.

The more load you want to create for your tests, the better the dedicated load testing tools will work in comparison to general web testing frameworks like Selenium/Webdriver.

I think you'll very quickly reach the boundaries of what you can do with Selenium/Webdriver when it comes to load testing, or else you'll have to write a lot of code yourself for things that are built-in when using Gatling or JMeter. And in that case, you'll probably introduce a few of your own concurrency bugs into the test code if you're not extremely careful: Concurrency is always hard.

Answer 3

I want to add that is it possible to use Selenium for performance testing. Example: you have application with Web-UI and you want to measure UI reaction with one user but dealing with huge database. In this case JMeter is a bad choice, because it can't process Ajax requests. I've worked with such situation on my past project, and for such volume testing we used Selenium. In other cases JMeter is better.

Answer 4

(disclaimer: I'm the Gatling project lead dev)

It currently seems unefficient to use Webdriver for load tests, even with a huge grid.

If you want to reduce the cost of maintaining functional tests AND load tests, you can build a system where you obtain your load tests base from your functional tests:

• plug a recording system on your functional tests, such as Gatling recorder, and obtain your load tests base
• edit them to add loops, users, dynamic data, etc
• when your functional tests evolve, you can re-run the first step and make a diff with your previous output so you can easily identify what you have to modify. This is easier in Gatling than in JMeter as scenarios are not a big chunck of XML but an expressive DSL.

Answer 5

I find that using a combination of both gives you the most realistic results. As the posters have already mentioned Selenium requires a large number of servers to get a similar load to tools such as jmeter.

However if you incorporate the browsermob proxy into your selenium tests (http://www.theautomatedtester.co.uk/blog/2012/automating-web-performance-data-collection-with-browsermob-proxy-and-selenium.html) and run the tests in parallel you get real browser performance for a user.

Atlassian also looks to be developing a tool using webdriver in this area (you can find it on their blog) which hopefully they will open source.

Answer 6

I wouldn't use JMeter personally. I would probably use SoapUI (executed by Maven) because it also fits nicely into Jenkins. SoapUI can do everything JMeter can, and more, especially with its Maven integration. I also prefer SoapUI for its Groovy integration and if you need to use BDD, you can integrate JBehave that way (with xUnit).

Answer 7

I experienced issues when trying to scale to large numbers of users with Selenium Grid. I ended up going with Funkload because:

1. It is significantly easier to scale and run tests quickly (without random hang-ups of browser instances)
2. It has an easy-to-use recorder of http requests
3. It makes it easy to load test API's by implementing the webunit Python module.
4. It has graphical output that can include server monitoring (if configured) which is very helpful in visualizing the output.

You can find the docs here: http://funkload.nuxeo.org/

Answer 8

Firstly, I work for IT Central Station, a platform for crowdsourced IT reviews. The easiest way to check out these various products on our site may be to look at a comparison page of these products you are researching.

If you're looking into other tools aside from JMeter and Selenium, you may want to look at StormRunner, HP's newer tool. A few of our real users have reviewed it recently.

Answer 9

I know it's an old post. Just take a look at this guys:

http://blazemeter.com/

They support 100% jmeter scripts, so you can have selenium webdriver plugin for your jmeter tests and run them against amazon EC2 instances.

The suggested "formula" per load generator (agent) is: C = B + 1 where C is number of concurrent users B is number of browser instances

taken from here: http://jmeter-plugins.org/wiki/WebDriverTutorial/

So, you can have both non-UI (api) and UI tests executed on blazemeter platform. Reports are excellent.

Pricing could be a little "problem" if you need to generate really huge loads but still it's possible and guys from blazemeter are providing great support.

excellent job blazemeter!

Answer 10

Disclaimer: First, I am affiliated with the tool suggested below.

Because we did not like the tools options 10 years ago, when we started our business (too expensive or too little features or just difficult to use), we wrote our own tool. Later, we started to offer it as product as well, after customers ask for an option to purchase it.

It is Java based and covers functional testing (we extended the Selenium WebDriver concept, fully compatible to WebDrivers) as well as load testing. There are several ways to write and compose tests. The core of the tool consist of HttpClient and HtmlUnit. The tool is engineering-centric, so more code and command line than UI stuff.

Lately we introduced a version where you can use WebDriver and Firefox to measure performance, not just of the interaction but we are hooked into FF to measure also the network stuff. If you have good enough machines, you can do load testing with it or just measure the performance development over time when you hook it up to Jenkins (we have a plugin for that).

Just have a look. You can play with the free edition and also use it forever for any kind of functional testing or even small performance comparisons. Upgrades are free too. https://www.xceptance.com/en/xlt/

We also open sourced one of our test suites to be able to easily build a simple test based on CSV data. More approaches will follow to make it easier to build test suites. https://github.com/Xceptance/testsuite-nocoding

Answer 11

I'll add Telerik Test Studio to the list. I use it for 10,000 virtual users across 5 PCs. If you can afford it, I couldn't recommend it enough. Way easier to use than JMeter. Also, my automated tests are in the same project, so there is no context switching or new tool to learn.