Why are intermittent browser automation failures hard to fix and how to overcome that?

Question

They seem to pop up almost randomly and with varying amounts of frequency.

How to avoid them and fix the ones we have?

Answer 1

Intermittent failures are hard to fix due to their very nature - they don't always happen. So when you think you have fixed them you often can't be sure just by 'running the spec'. You usually need to run the spec many times to have a sense for whether a 'fix' has worked.

This makes addressing them hard. It requires a variety of techniques and approaches together to make a workable test suite that can reliably and repeatedly run as part of a devops pipeline which is frequently the goal in many companies today.

It can also be hard to fix them due to the sheer time that it can take to fix each one relative to the time saved when compared to the entire suite run time.

I have found the following approaches and techniques to be of value:

• You frequently need to run the spec 5, 10, 20, 100 or 1000 times to actually know if your 'fix' has worked. Also try and run in different local environments or a CI server environment.

• Use the ruby - rspec - capybara stack as capybara has had a great deal of work to address timing issues and, relative to its cousins is relatively mature in this space. It also provides a great DSL

• With capybara, learn how to use, , wait: time at the end of command like fill_in for polling waits. Example:
  find 'input#' + input_id, visible: true, wait: max_wait_time

  Polling waits have previously been called implicit (as opposed to explicit) wait but I prefer the term polling wait as it is clearer - the element is polled for and as soon as it is found the script continues, otherwise it waits for the seconds given. ** A polling wait of 4 seconds does NOT wait 4 seconds if the element is found, it immediately moves to the next step**

• Understand, for capybara, which commands will do the polling wait and which command will return immediately if element not found, e.g. first

• If all other attempts to address an intermittent failure have failed, you need to be willing to use a few short sleeps (explicit waits, i.e. for x seconds wait). A fixed sleep is, by and large a major no-no in UI automation. Given that there may be some cases - often where you have not figured out or do not know the real reason for the screen delay, you have to use a short sleep, usually 2-4 seconds. However you should be willing to spend hours trying to avoid this. Similarly, when you try to remove an existing sleep you need to spend hours and dozens of runs to know if the removal may be safe.

• If you have to have some fixed short sleeps try to use a global approach to setting the value itself so that you can adjust for different machines, environments, remote servers, etc.

Code sample:

sleepers.rb

module Sleepers
  def sleepy(sleep_length=2)
    sleep sleep_length
  end
  def sleep_short
    sleep Sleep_lengths[:short]
  end
end

sleep_lengths.rb

Sleep_lengths = Hash.new
if PRODUCTION
  Sleep_lengths[:short] = 2 
  Sleep_lengths[:medium] = 4 
  Sleep_lengths[:medium_long] = 16
  Sleep_lengths[:long] = 60
else
  Sleep_lengths[:short] = 2 
  Sleep_lengths[:medium] = 10
  Sleep_lengths[:medium_long] = 16
  Sleep_lengths[:long] = 90
end

• Address the underlying issues. What's usually going on with intermittent errors is that you are dealing with asynchronous process that are going on in the browser and you have javascript and ajax being used. If this is the case you can apply code to the specific case, for example, see if any ajax calls are outstanding or see if any javascript is running.

Code sample:

module WaitForAjax def wait_for_ajax Timeout.timeout(Capybara.default_max_wait_time) do loop until finished_all_ajax_requests? end rescue Timeout::Error end def finished_all_ajax_requests? page.evaluate_script('jQuery.active').zero? end end

Finally spec/spec_helper.rb gets:

...
require_relative 'support/sleep_lengths'
require_relative 'support/sleepers'
require_relative 'support/wait_for_ajax'
...

and specs get:

...
include Sleepers
include WaitForAjax
...

and usage such as:

find p.main_form, visible: true, wait: Sleep_lengths[:medium]
...
sleep_short
...
wait_for_ajax
...