Choosing wait timeout values

Question

Explicit Waits are highly recommended to be used instead of hardcoded time delays. But, Explicit Waits still require a timeout value to be specified.

The Story

Up until now, we've typically used 5, 10 or 20 seconds using our own judgement of the speed of the functionality under test which has the "human bias" component unfortunately.

The problem though arose when we've started to execute tests on different servers against different instances of our application which brought extra impacts of the network and different performances of the servers.

Then, we've started to increase wait timeouts here and there to counteract network speed and slow application performance.

The problem is, now, when there is an actual timeout failure, sometimes we wait for much longer than it is actually needed making the feedback turnaround slower.

The Question

What considerations should we take into account when choosing Explicit Wait timeout values?

We've also thought about developing a system of weights for different instances of our application, which would dynamically adjust all the timeouts we are using inside the test automation codebase. Would really appreciate if someone would share experience doing something similar.

Answer 1

Rather than observed response time, I prefer this limit: The maximum acceptable response time. This may differ for different requests.

This becomes trickier, of course, when the test code and test environment inject delays over and above the system response time. So I sometimes (grudgingly) increase the wait times to account for those. And put in a lot of effort to reduce those test-induced delays.

If “maximum acceptable response time” is too painful to tolerate in tests, that’s a sign to reconsider what is acceptable.

Answer 2

I have no problem having quite long explicit waits because my test rely on expected conditions instead. So wait fails only when expected condition failed and so test failed.

We have two helper methods. Wait for regular actions expires in 60 sec, and for some extremely long calculations at 180 sec. So I guess you can set those limits as config parameters and use different configuration values for different system architectures/settings.

Because most test pass, we are not concerned that for few failing tests we waited extra 60 or 180 sec. Our website and our tests are so complex that setup+run of some tests can take up to 10 minutes or more, and we have many tests.

If you are so concerned about "fail fast", maybe you need a separate subset of "quick smoke tests" which you will run before full test. But even those would use standard ("worst case") wait limit.

You should never want to have a test fail because test not waited long enough: Before investigating you are going to rerun the failed test again anyway, just to see if it is a persistent failure or just a flaky test.

Answer 3

Consider creating a good structure to support different test speed requirements.

For example classify test speed as normal, slow and extra-slow. Make the standard 'normal'. Then identify those tests that can be slower and allow them to have longer explicit waits with the longest waits being used for extra-slow test processes.

As you identify the slow tests, tag them as slow or extra-slow and adjust the test runner to use these waits for the appropriately tagged tests.

Also, examine the tests in detail. Slow tests are commonly due to using real services, backends and dependencies. Stubbing and Mocking these dependencies, even for user acceptance tests, can address the underlying issue. If you experience a lot of slowness due to dependency issues in user acceptance testing it is a sign that you may not have enough / correct unit tests

Answer 4

The answer lies in another question: what are you testing?

Automated end-to-end GUI tests check for functionality, above all. But because you can have time-outs, you implicitly test 'performance' as well.

However, test servers typically have less capacity than production servers, so you cannot expect similar and/or steady performance times. In my projects, time-outs between 30 and 60 seconds are a good compromise between speed and reduced test server performance.

Here's an idea I read somewhere: after running tests, you will have

• passed tests: no action needed
• failed tests due to anything except a time-out: to investigate
• failed test due to a time-out: run these tests again (automatically). If they now pass, the failure was simply a local performance issue. If they fail again, either performance is consisently below the acceptable or there is an actual bug.

Answer 5

In my personal experience, faced this issue earlier. Resolved by the below approach in our automation test strategy.

Defined the value of waittime variables based on the request type and configuration maintained in a separate file. For example, for the page load, given more waittime, and ajax requests given less waittime. When execute in different servers, adjusting the values of waittime variables. The staging server, the values defined as like this, pwaittime(variable for Pageload waittime) as 60 seconds, awaittime(variable for ajaxrequest wait time) 10 seconds for our application. For production server, the values are updated as 20 seconds and 5 seconds respectively in the config file. Since the production server is more powerful with multiple nodes than the staging server.