1

Explicit Waits are highly recommended to be used instead of hardcoded time delays. But, Explicit Waits still require a timeout value to be specified.

The Story

Up until now, we've typically used 5, 10 or 20 seconds using our own judgement of the speed of the functionality under test which has the "human bias" component unfortunately.

The problem though arose when we've started to execute tests on different servers against different instances of our application which brought extra impacts of the network and different performances of the servers.

Then, we've started to increase wait timeouts here and there to counteract network speed and slow application performance.

The problem is, now, when there is an actual timeout failure, sometimes we wait for much longer than it is actually needed making the feedback turnaround slower.

The Question

What considerations should we take into account when choosing Explicit Wait timeout values?

We've also thought about developing a system of weights for different instances of our application, which would dynamically adjust all the timeouts we are using inside the test automation codebase. Would really appreciate if someone would share experience doing something similar.

  • What sort of testing are you doing? – Kevin McKenzie Aug 1 '17 at 17:24
  • @KevinMcKenzie these are automated end-to-end tests based on Protractor/WebDriverJS. Sorry, for not mentioning that explicitly. – alecxe Aug 1 '17 at 17:25
  • 1
    I haven't actually implemented this, but I've thought about enhancing our internal test framework to include a server health check in the explicit wait (i.e. we have an internal monitoring server as well as internal health endpoints for the various services), and bail on the explicit wait loop if the service is detected to be down. – ernie Aug 1 '17 at 18:17
8
+50

Rather than observed response time, I prefer this limit: The maximum acceptable response time. This may differ for different requests.

This becomes trickier, of course, when the test code and test environment inject delays over and above the system response time. So I sometimes (grudgingly) increase the wait times to account for those. And put in a lot of effort to reduce those test-induced delays.

If “maximum acceptable response time” is too painful to tolerate in tests, that’s a sign to reconsider what is acceptable.

  • 1
    I do this too and totally agree. If it's really that slow I usually mention that as a performance bug to fix as it's a usability issue in the wait time. Usually there is an acceptable max wait that will work for all pages to keep one under a desired performance marker. Then anything that fails due to the wait time is manually tested and either is a performance bug or a functional bug if it literally doesn't return at all. I ensure these are all implicit waits though and never an explicit counting wait. – mutt Aug 7 '17 at 21:27
4

I have no problem having quite long explicit waits because my test rely on expected conditions instead. So wait fails only when expected condition failed and so test failed.

We have two helper methods. Wait for regular actions expires in 60 sec, and for some extremely long calculations at 180 sec. So I guess you can set those limits as config parameters and use different configuration values for different system architectures/settings.

Because most test pass, we are not concerned that for few failing tests we waited extra 60 or 180 sec. Our website and our tests are so complex that setup+run of some tests can take up to 10 minutes or more, and we have many tests.

If you are so concerned about "fail fast", maybe you need a separate subset of "quick smoke tests" which you will run before full test. But even those would use standard ("worst case") wait limit.

You should never want to have a test fail because test not waited long enough: Before investigating you are going to rerun the failed test again anyway, just to see if it is a persistent failure or just a flaky test.

2

Consider creating a good structure to support different test speed requirements.

For example classify test speed as normal, slow and extra-slow. Make the standard 'normal'. Then identify those tests that can be slower and allow them to have longer explicit waits with the longest waits being used for extra-slow test processes.

As you identify the slow tests, tag them as slow or extra-slow and adjust the test runner to use these waits for the appropriately tagged tests.

Also, examine the tests in detail. Slow tests are commonly due to using real services, backends and dependencies. Stubbing and Mocking these dependencies, even for user acceptance tests, can address the underlying issue. If you experience a lot of slowness due to dependency issues in user acceptance testing it is a sign that you may not have enough / correct unit tests

2

The answer lies in another question: what are you testing?

Automated end-to-end GUI tests check for functionality, above all. But because you can have time-outs, you implicitly test 'performance' as well.

However, test servers typically have less capacity than production servers, so you cannot expect similar and/or steady performance times. In my projects, time-outs between 30 and 60 seconds are a good compromise between speed and reduced test server performance.

Here's an idea I read somewhere: after running tests, you will have

  • passed tests: no action needed
  • failed tests due to anything except a time-out: to investigate
  • failed test due to a time-out: run these tests again (automatically). If they now pass, the failure was simply a local performance issue. If they fail again, either performance is consisently below the acceptable or there is an actual bug.
1

In my personal experience, faced this issue earlier. Resolved by the below approach in our automation test strategy.

Defined the value of waittime variables based on the request type and configuration maintained in a separate file. For example, for the page load, given more waittime, and ajax requests given less waittime. When execute in different servers, adjusting the values of waittime variables. The staging server, the values defined as like this, pwaittime(variable for Pageload waittime) as 60 seconds, awaittime(variable for ajaxrequest wait time) 10 seconds for our application. For production server, the values are updated as 20 seconds and 5 seconds respectively in the config file. Since the production server is more powerful with multiple nodes than the staging server.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.