17

I have been using Selenium/Webdriver for functional testing in a grid setup. We are in the process of transitioning to a Solr based search service. I already have Solrmeter for load testing the Search engine core, load UI for load testing the Rest-ful service. We also would like to load test the entire end to end service chain, i.e

User interface -> Restful service -> Solr core.

My boss wants me to use Selenium/WebDriver for this end to end load testing so that it becomes integrated with our core testing framework.

I am skeptical about it because WebDriver does not have supporting tools for load testing. The easiest way I can think of doing it in Selenium/WebDriver is to code timestamps in my scripts at the time when the query is sent and when the results are displayed on the webpage. Then write some reporting mechanism to gather and display reports. Also I imagine selenium would be slower.

On the other hand members from the stackoverflow community have suggested using JMeter for the same which I assume is less of an hassle. Can somebody give me pros/cons of Selenium/WebDriver Vs JMeter for load testing in this scenario?

Improve this question

11 Answers 11

Reset to default
15

Similar questions have been asked a number of times. The general consensus is that UI automation tools such as Selenium are not a good choice for load testing. There are a number of reasons why, but the main one is that it is a LOT slower. You can easily spin up thousands of concurrent requests in a second in a tool like JMeter which to reproduce in Selenium would take multiples of thousands of servers all orchestrated to give you the same amount of traffic. There are tools like selenium grid or browsermob which would allow you to orchestrate those kinds of tests, but it is really inefficient compared to a tool like JMeter.

Performance testing is a little different, since measuring the time it takes for UI to load or complete some operation can be done with Selenium, however you usually execute performance testing in tandem with load testing to answer questions like: How well does X action perform when the server is under no load/medium load/high load/varying load, etc.

As far as tools that are appropriate for load testing, JMeter is a very popular free tool, but there are a number of others including one I prefer - Visual Studio Load Testing (which requires a visual studio license which you may or may not already have).

Improve this answer
3
  • Visual Studio Load Testing? Never heard of it before.... what's the cost as compared to HP loadrunner?
    – Pacerier
    Sep 18, 2015 at 10:09
  • Much cheaper. HP LoadRunner can cost millions of dollars for large organizations. Visual Studio Load Testing now comes with a Visual Studio Enterprise 2015 license (used to be Visual Studio Ultimate). It looks like since I used it they have also added the ability to run your tests in the cloud relatively cheaply: visualstudio.com/en-us/get-started/test/load-test-your-app-vs. Or, you can use your own hardware at no additional cost. The initial cost of Enterprise looks to be $6,000 with a yearly license renewal of $2,169.
    – Sam Woods
    Sep 18, 2015 at 17:54
  • Yea loadrunner is so expensive that it actually makes business sense to hire someone to crack the license instead of paying it upfront.
    – Pacerier
    Sep 21, 2015 at 11:43
8

You could also check out Gatling, which is an open source stress tool based on akka.

On the Gatling homepage, you'll find a direct comparison of how your test specification will look in Gatling vs. in JMeter. This might help you decide between these too.

If you want to create massive amounts of load, I think there's nothing better than akka as an engine.

The more load you want to create for your tests, the better the dedicated load testing tools will work in comparison to general web testing frameworks like Selenium/Webdriver.

I think you'll very quickly reach the boundaries of what you can do with Selenium/Webdriver when it comes to load testing, or else you'll have to write a lot of code yourself for things that are built-in when using Gatling or JMeter. And in that case, you'll probably introduce a few of your own concurrency bugs into the test code if you're not extremely careful: Concurrency is always hard.

Improve this answer
1
  • Gatling is great. The tests are also really concise and easy to read since it uses Scala
    – FearlessHyena
    Apr 17, 2017 at 6:13
3

I want to add that is it possible to use Selenium for performance testing. Example: you have application with Web-UI and you want to measure UI reaction with one user but dealing with huge database. In this case JMeter is a bad choice, because it can't process Ajax requests. I've worked with such situation on my past project, and for such volume testing we used Selenium. In other cases JMeter is better.

Improve this answer
6
  • ? Why would you say that JMeter isn't able to process Ajax requests? It's able to do so.
    – Pacerier
    Sep 18, 2015 at 10:16
  • Without using additional plugins it's rather tricky task, because you have to reproduce manually all requests, IMHO.
    – Dzmitry Kashlach
    Sep 21, 2015 at 6:08
  • What about the inbuilt JMeter recorder? Does it not work? You'd simply need to turn it on and then run the Ajax requests... and the recorder would automatically record all the requests. No manual work needed.
    – Pacerier
    Sep 21, 2015 at 11:45
  • Not so simple. Such requests can contain tokens, id's and once recorded, this scenario can be failed while replaying. Usually it's hundreds of requests.
    – Dzmitry Kashlach
    Sep 21, 2015 at 14:35
  • I mean yes we'd have to modify the tokens ids with variables like ${foobar}. After doing that it would work fine right?
    – Pacerier
    Sep 22, 2015 at 3:09
2

(disclaimer: I'm the Gatling project lead dev)

It currently seems unefficient to use Webdriver for load tests, even with a huge grid.

If you want to reduce the cost of maintaining functional tests AND load tests, you can build a system where you obtain your load tests base from your functional tests:

  • plug a recording system on your functional tests, such as Gatling recorder, and obtain your load tests base
  • edit them to add loops, users, dynamic data, etc
  • when your functional tests evolve, you can re-run the first step and make a diff with your previous output so you can easily identify what you have to modify. This is easier in Gatling than in JMeter as scenarios are not a big chunck of XML but an expressive DSL.
Improve this answer
1
  • How does Gatling make money? Basically, JM has been time-proven. Does Gatling actually have plans to stick around for at least the next 20 years?
    – Pacerier
    Sep 18, 2015 at 10:14
2

I find that using a combination of both gives you the most realistic results. As the posters have already mentioned Selenium requires a large number of servers to get a similar load to tools such as jmeter.

However if you incorporate the browsermob proxy into your selenium tests (http://www.theautomatedtester.co.uk/blog/2012/automating-web-performance-data-collection-with-browsermob-proxy-and-selenium.html) and run the tests in parallel you get real browser performance for a user.

Atlassian also looks to be developing a tool using webdriver in this area (you can find it on their blog) which hopefully they will open source.

Improve this answer
1
2

I wouldn't use JMeter personally. I would probably use SoapUI (executed by Maven) because it also fits nicely into Jenkins. SoapUI can do everything JMeter can, and more, especially with its Maven integration. I also prefer SoapUI for its Groovy integration and if you need to use BDD, you can integrate JBehave that way (with xUnit).

Improve this answer
1
  • Jmeter has a maven plugin which allows you to plug it into Jenkins fairly easily. jmeter.lazerycode.com.
    – Ardesco
    Jul 22, 2013 at 9:48
2

I experienced issues when trying to scale to large numbers of users with Selenium Grid. I ended up going with Funkload because:

  1. It is significantly easier to scale and run tests quickly (without random hang-ups of browser instances)
  2. It has an easy-to-use recorder of http requests
  3. It makes it easy to load test API's by implementing the webunit Python module.
  4. It has graphical output that can include server monitoring (if configured) which is very helpful in visualizing the output.

You can find the docs here: http://funkload.nuxeo.org/

Improve this answer
1

Firstly, I work for IT Central Station, a platform for crowdsourced IT reviews. The easiest way to check out these various products on our site may be to look at a comparison page of these products you are researching.

If you're looking into other tools aside from JMeter and Selenium, you may want to look at StormRunner, HP's newer tool. A few of our real users have reviewed it recently.

Improve this answer
4
  • 1
    sqa.stackexchange.com/help/promotion your answer although on-topic contains only links to signup pages. better summarise the results and link to "open" details. this is just another form of spam.
    – Niels van Reijmersdal
    Jul 27, 2015 at 21:21
  • Hi Abigail! Let me thank you for openly stating your affiliation. That's a rare thing as far as astrotufing goes. Now, while we certainly don't mind links to external content, our criteria for an acceptable answer is one that stands on its own as an answer without links. Exactly what needs to change about this answer to make it acceptable, well I'm not sure because I haven't signed up to see the report. But an answer has to stand on its own before reaching out to external resources for support. Thanks!
    – corsiKa ♦
    Jul 27, 2015 at 22:15
  • 1
    responding to this helpful feedback, i have made sure that all links do not require a signup.
    – abigail
    Jul 28, 2015 at 17:22
  • @corsiKa, He's not astroturfing of course... astroturfing is only if he doesn't state his allegiance.
    – Pacerier
    Sep 18, 2015 at 10:20
0

I know it's an old post. Just take a look at this guys:

http://blazemeter.com/

They support 100% jmeter scripts, so you can have selenium webdriver plugin for your jmeter tests and run them against amazon EC2 instances.

The suggested "formula" per load generator (agent) is: C = B + 1 where C is number of concurrent users B is number of browser instances

taken from here: http://jmeter-plugins.org/wiki/WebDriverTutorial/

So, you can have both non-UI (api) and UI tests executed on blazemeter platform. Reports are excellent.

Pricing could be a little "problem" if you need to generate really huge loads but still it's possible and guys from blazemeter are providing great support.

excellent job blazemeter!

Improve this answer
1
  • 3
    Do you have any affiliation with blazemeter? It would help if you said up front whether you're part of the project or simply a happy user.
    – Kate Paulk ♦
    Jun 5, 2014 at 14:19
0

Disclaimer: First, I am affiliated with the tool suggested below.

Because we did not like the tools options 10 years ago, when we started our business (too expensive or too little features or just difficult to use), we wrote our own tool. Later, we started to offer it as product as well, after customers ask for an option to purchase it.

It is Java based and covers functional testing (we extended the Selenium WebDriver concept, fully compatible to WebDrivers) as well as load testing. There are several ways to write and compose tests. The core of the tool consist of HttpClient and HtmlUnit. The tool is engineering-centric, so more code and command line than UI stuff.

Lately we introduced a version where you can use WebDriver and Firefox to measure performance, not just of the interaction but we are hooked into FF to measure also the network stuff. If you have good enough machines, you can do load testing with it or just measure the performance development over time when you hook it up to Jenkins (we have a plugin for that).

Just have a look. You can play with the free edition and also use it forever for any kind of functional testing or even small performance comparisons. Upgrades are free too. https://www.xceptance.com/en/xlt/

We also open sourced one of our test suites to be able to easily build a simple test based on CSV data. More approaches will follow to make it easier to build test suites. https://github.com/Xceptance/testsuite-nocoding

Improve this answer
0

I'll add Telerik Test Studio to the list. I use it for 10,000 virtual users across 5 PCs. If you can afford it, I couldn't recommend it enough. Way easier to use than JMeter. Also, my automated tests are in the same project, so there is no context switching or new tool to learn.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.