Problems with maintenance of Selenium on grid service

Question

We have cluster of 5 nodes with all of them are only Firefox nodes. We have run into so many issues maintaining the grid up and running.

• Selenium nodes goes out of memory (JAVA heap space issues)

  http://code.google.com/p/selenium/issues/detail?id=4969

• Some sessions stuck in the nodes and it will go for infinite loop hence my Jenkins job hangs and never come back. Sometime a session is hung at ClickEvent or sometime other place but Session is hung while reading on socket.

• Browser updates? selenium server jar updates ?

Whats the best way to handle the Grid service up and running ? Give us your thoughts.

Answer 1

Last year I wrote a blog post about similar issues I had: http://www.vanreijmersdal.nl/577/creating-a-maintainable-selenium-2-grid-setup/

In short:

• Finish all the tests (Put Jenkins in shutdown mode to prevent starting of new tests)
• Shutdown hub (with lifecycle manager)
• Daily reboot your nodes (auto start server on boot)
• Place all the executables/scripts on a central file share (for easy updating)
• Restart hub and take Jenkins out of shutdown mode

The post does not handle the browser updates, but a lot has been written already about updating software on computers with packaging systems. For example Firefox with MS SMS. Shouldn't be to hard to find information about this, maybe even your local IT department has experience with this.

In the end I think its cheaper to just run your tests against Sauce Labs (USA) or TestingBot (EU), because maintaining your own Grid and a lot of different browsers and operating systems can be very time consuming.

Answer 2

The memory issue you mentioned has apparently been fixed, so you've got that going for you.

I've had problems with browser updates causing problems; Firefox has a tendancy to throw some popups up when its updating versions which Selenium can't interact with. We've had to login over VNC to our headless servers to try and work out what was going on.

With regards to locking scenarios, a minutely cronjob on the server could check the last changed date of the selenium log; if its not changed in the last hour, restart the node. This might cause the node to keep restarting through the quiet times, but would also keep the locked jobs in Jenkins to a maximum of an hour.

Answer 3

You could always set timeout after which grid will lose connection.

You can as well update grid jars(even if not as jenkins plugin).

You need latest selenium drivers if your browsers update regularly.