Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I have been using Selenium/Webdriver for functional testing in a grid setup. We are in the process of transitioning to a Solr based search service. I already have Solrmeter for load testing the Search engine core, load UI for load testing the Rest-ful service. We also would like to load test the entire end to end service chain, i.e User interface -> Restful service -> solr core. My boss wants me to use Selenium/webdriver for this end to end load testing so that it becomes integrated with our core testing framework.

I am skeptical about it because webdriver does not have supporting tools for load testing. The easiest way i can think of doing it in selenium/webdriver is to code timestamps in my scripts at the time when the query is sent and when the results are displayed on the webpage.Then write some reporting mechanism to gather and display reports.Also i imagine selenium would be slower.

On the other hand members from the stackoverflow community have suggested using Jmeter for the same which i assume is less of an hassle. Can somebody give me pros/cons of Selenium/Webdriver Vs Jmeter for load testing in this scenario?

share|improve this question
add comment

7 Answers

up vote 10 down vote accepted

Similar questions have been asked a number of times. The general consensus is that UI automation tools such as Selenium are not a good choice for load testing. There are a number of reasons why, but the main one is that it is a LOT slower. You can easily spin up thousands of concurrent requests in a second in a tool like JMeter which to reproduce in Selenium would take multiples of thousands of servers all orchestrated to give you the same amount of traffic. There are tools like selenium grid or browsermob which would allow you to orchestrate those kinds of tests, but it is really inefficient compared to a tool like JMeter.

Performance testing is a little different, since measuring the time it takes for UI to load or complete some operation can be done with Selenium, however you usually execute performance testing in tandem with load testing to answer questions like: How well does X action perform when the server is under no load/medium load/high load/varying load, etc.

As far as tools that are appropriate for load testing, JMeter is a very popular free tool, but there are a number of others including one I prefer - Visual Studio Load Testing (which requires a visual studio license which you may or may not already have).

share|improve this answer
add comment

I experienced issues when trying to scale to large numbers of users with Selenium Grid. I ended up going with Funkload because:

  1. It is significantly easier to scale and run tests quickly (without random hang-ups of browser instances)
  2. It has an easy-to-use recorder of http requests
  3. It makes it easy to load test API's by implementing the webunit Python module.
  4. It has graphical output that can include server monitoring (if configured) which is very helpful in visualizing the output.

You can find the docs here: http://funkload.nuxeo.org/

share|improve this answer
add comment

I wouldn't use JMeter personally. I would probably use SoapUI (executed by Maven) because it also fits nicely into Jenkins. SoapUI can do everything JMeter can, and more, especially with its Maven integration. I also prefer SoapUI for its Groovy integration and if you need to use BDD, you can integrate JBehave that way (with xUnit).

share|improve this answer
    
Jmeter has a maven plugin which allows you to plug it into Jenkins fairly easily. jmeter.lazerycode.com. –  Ardesco Jul 22 '13 at 9:48
add comment

I find that using a combination of both gives you the most realistic results. As the posters have already mentioned Selenium requires a large number of servers to get a similar load to tools such as jmeter.

However if you incorporate the browsermob proxy into your selenium tests (http://www.theautomatedtester.co.uk/blog/2012/automating-web-performance-data-collection-with-browsermob-proxy-and-selenium.html) and run the tests in parallel you get real browser performance for a user.

Atlassian also looks to be developing a tool using webdriver in this area (you can find it on their blog) which hopefully they will open source.

share|improve this answer
    
For Selenium performance testing on client side, while outdated, this is also a good resource: code.google.com/p/selenium-profiler –  David Jan 20 '13 at 6:04
add comment

I want to add that is it possible to use Selenium for performance testing. Example: you have application with Web-UI and you want to measure UI reaction with one user but dealing with huge database. In this case JMeter is a bad choice, because it can't process Ajax requests. I've worked with such situation on my past project, and for such volume testing we used Selenium. In other cases JMeter is better.

share|improve this answer
add comment

(disclaimer: I'm the Gatling project lead dev)

It currently seems unefficient to use Webdriver for load tests, even with a huge grid.

If you want to reduce the cost of maintaining functional tests AND load tests, you can build a system where you obtain your load tests base from your functional tests:

  • plug a recording system on your functional tests, such as Gatling recorder, and obtain your load tests base
  • edit them to add loops, users, dynamic data, etc
  • when your functional tests evolve, you can re-run the first step and make a diff with your previous output so you can easily identify what you have to modify. This is easier in Gatling than in JMeter as scenarios are not a big chunck of XML but an expressive DSL.
share|improve this answer
add comment

You could also check out Gatling, which is an open source stress tool based on akka.

On the Gatling homepage, you'll find a direct comparison of how your test specification will look in Gatling vs. in JMeter. This might help you decide between these too.

If you want to create massive amounts of load, I think there's nothing better than akka as an engine.

The more load you want to create for your tests, the better the dedicated load testing tools will work in comparison to general web testing frameworks like Selenium/Webdriver.

I think you'll very quickly reach the boundaries of what you can do with Selenium/Webdriver when it comes to load testing, or else you'll have to write a lot of code yourself for things that are built-in when using Gatling or JMeter. And in that case, you'll probably introduce a few of your own concurrency bugs into the test code if you're not extremely careful: Concurrency is always hard.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.