They seem to pop up almost randomly and with varying amounts of frequency.
How to avoid them and fix the ones we have?
Intermittent failures are hard to fix due to their very nature - they don't always happen. So when you think you have fixed them you often can't be sure just by 'running the spec'. You usually need to run the spec many times to have a sense for whether a 'fix' has worked.
This makes addressing them hard. It requires a variety of techniques and approaches together to make a workable test suite that can reliably and repeatedly run as part of a devops pipeline which is frequently the goal in many companies today.
It can also be hard to fix them due to the sheer time that it can take to fix each one relative to the time saved when compared to the entire suite run time.
I have found the following approaches and techniques to be of value:
You frequently need to run the spec 5, 10, 20, 100 or 1000 times to actually know if your 'fix' has worked. Also try and run in different local environments or a CI server environment.
Use the ruby - rspec - capybara stack as capybara has had a great deal of work to address timing issues and, relative to its cousins is relatively mature in this space. It also provides a great DSL
With capybara, learn how to use,
, wait: time at the end of command like
fill_in for polling waits. Example:
find 'input#' + input_id, visible: true, wait: max_wait_time
Polling waits have previously been called implicit (as opposed to explicit) wait but I prefer the term polling wait as it is clearer - the element is polled for and as soon as it is found the script continues, otherwise it waits for the seconds given. ** A polling wait of 4 seconds does NOT wait 4 seconds if the element is found, it immediately moves to the next step**
Understand, for capybara, which commands will do the polling wait and which command will return immediately if element not found, e.g.
If all other attempts to address an intermittent failure have failed, you need to be willing to use a few short sleeps (explicit waits, i.e. for x seconds wait). A fixed sleep is, by and large a major no-no in UI automation. Given that there may be some cases - often where you have not figured out or do not know the real reason for the screen delay, you have to use a short sleep, usually 2-4 seconds. However you should be willing to spend hours trying to avoid this. Similarly, when you try to remove an existing sleep you need to spend hours and dozens of runs to know if the removal may be safe.
If you have to have some fixed short sleeps try to use a global approach to setting the value itself so that you can adjust for different machines, environments, remote servers, etc.
module Sleepers def sleepy(sleep_length=2) sleep sleep_length end def sleep_short sleep Sleep_lengths[:short] end end
Sleep_lengths = Hash.new if PRODUCTION Sleep_lengths[:short] = 2 Sleep_lengths[:medium] = 4 Sleep_lengths[:medium_long] = 16 Sleep_lengths[:long] = 60 else Sleep_lengths[:short] = 2 Sleep_lengths[:medium] = 10 Sleep_lengths[:medium_long] = 16 Sleep_lengths[:long] = 90 end
loop until finished_all_ajax_requests?
Finally spec/spec_helper.rb gets:
... require_relative 'support/sleep_lengths' require_relative 'support/sleepers' require_relative 'support/wait_for_ajax' ...
and specs get:
... include Sleepers include WaitForAjax ...
and usage such as:
find p.main_form, visible: true, wait: Sleep_lengths[:medium] ... sleep_short ... wait_for_ajax ...