Headless Selenium Failing for Ajax Calls

Question

How do you troubleshoot an Ajax call that never seems to complete in Selenium running in a headless environment?

I'm running Selenium 3.0.1 from a Django unittest to confirm content gets dynamically loaded via Ajax when buttons are clicked. My test's setUp() initializes Selenium with:

p = webdriver.FirefoxProfile()
p.set_preference("webdriver.log.file", "/tmp/firefox_console")
self.driver = webdriver.Firefox(p, capabilities={'marionette': False})
self.driver.implicitly_wait(20)

These tests run perfectly on my localhost, but when I run these tests on a headless integration server using Xvfb, they always fail, timing out looking for the IDs of the Ajax content that apparently never seems to populate.

I'm completely stuck with how to diagnose the problem. Is there something with Selenium+Firefox+Xvfb that causes Ajax calls to fail? Or are they succeeding, but the dynamically created elements are just not visible to Selenium?

I tried increasing the virtual screen resolution, to better simulate my localhost, with:

self.driver.set_window_size(1920, 1080)

but that seemed to have no effect.

I can't find any documentation on reading the status of Ajax calls from Selenium. Is this possible?

What other reason would Selenium not work on a headless server?

Answer 1

Selenium will perform differently in different environment as many folks have found.

Look into two things:

• increasing the application-wide timeout setting that is used by explicit waits (i.e. waits for specific elements or events and NOT sleeps) within selenium such as find, find_by, etc.
  e.g. default_max_wait_timeout = 30 in capybara.rb
• add your own code to explicitly wait for events such as page finished loading, animations finished, no ajax calls in progress, etc.

I'd avoid the window reisze option as that may start to contribute to more Net::ReadTimeout's error that plague us (if you're doing ruby that is, I think you are from the code)

For testing purposes you can add fixed sleeps but only while testing the test.

Getting a login to the integration server so you can diectly run tests there is essential. We can do this with CircleCI, not sure about your setup.

One example implementation from is here:

# spec/support/wait_for_ajax.rb
module WaitForAjax
  def wait_for_ajax
    Timeout.timeout(Capybara.default_max_wait_time) do
      loop until finished_all_ajax_requests?
    end
  end

  def finished_all_ajax_requests?
    page.evaluate_script('jQuery.active').zero?
  end
end

RSpec.configure do |config|
  config.include WaitForAjax, type: :feature
end

This was about ajax, however be aware that failures frequently happen in different environments and then for different devices and browsers. Timing and synchronization issues around page load, js/css load, rendering, animations and ajx can all lead to both failures in some environments and also intermittents in some environments also.

Answer 2

This page lists the --window-size=x,y which you can add with opts.add_argument("--window-size=1900,1080").

Complete example which works with latest firefox v57b8 (assumes you have geckodriver somewhere in your $PATH):

    caps = webdriver.DesiredCapabilities().FIREFOX
    opts = FirefoxOptions()
    opts.add_argument("-headless")
    opts.add_argument("--window-size=1900,1080")
    caps["marionette"] = True
    driver = webdriver.Firefox(capabilities=caps, firefox_options=opts)