What Are Some Examples of Why Explicit Sleep Statements Are Bad

Question

I've read various places that explicit sleep statements are a bad practice in automated tests, even if I've used them occasionally. They can be annoying (creating unnecessary waiting in tests) but I can't think of any examples of why they're actually harmful. What are some examples of sleep statements causing harm in automated tests?

Answer 1

A hard coded sleep statement is generally supposed to represent some sort of mocked delay in the application that doesn't exist during testing.

The harmful aspect of this is that a hard coded value can't represent the complexity that actually exists in what its mocking.

Take for example a network delay. Your production system usually takes 4 seconds to go out over the network, take some data, and come back over the network with the result. So you simply mock up the data, toss in a 4 second delay, and give it back. Sounds simple.

Because it is simple - almost too simple. If something on your network changes, your test is now inaccurate. You can't see how the routine will react under abnormally heavy or abnormally light network load. In other words you're trying to test what you cannot control. You're make something repeatable (with the 4 second sleep) that really isn't repeatable (no run over the network will ever be the same twice).

And if the correctness of the test relies on an invalid and potentially often changing assumption, then what does that say about the reliability of the test?

Answer 2

I have a series of about 50 Selenium tests that take less than 10 minutes to run. I've been able to cut the execution time down by several minutes by writing the tests in such that no sleep() is required to know when the output has arrived.

I find that when I use sleep() in my tests that it's a kludge for when I don't have a good way to actually monitor the state of the thing I'm testing for when it's actually finished with what it's doing. Using sleep() amounts to "Are you ready yet? Are you ready yet?" @Dmitry Zhariy points out that adding sleeps enforced delays in your tests. This is bad. Your tests should always run as fast as possible. If there are unavoidable delays in the operation of your project, so be it (but try to see if you can find a way to make even those go faster.)

Answer 3

Well, when the test wastes your time – that is harmful I think. But, it depends. Example: you have several test environments. MachineA is hight performance real machine. MachineB – is slow virtual machine on remote server in the middle of Antarctica.

On MachineA you need 1 sec. sleep. On MachineB you need 10 sec. So, you make Sleep(10s) in the code which waste the MachineA performance.

So now your test time depends on the slowest machine in your lab. Let’s say you have 100 tests with a single 10 seconds delay. 100*10 = 1000 sec = 17 minutes of wasted time per each run on high performance workstations.

On my previous project we have fight a lot with Sleeps and have reduced the test execution from 12 to 9 hours for 1200 tests. My colleagues that used came to the working place early were not waited additional 2-3 hours for the test suite before it finishes.

But anyway we left some sleeps that we were not able to fix fast.

Answer 4

The main harm occurs when the time value you use in your Sleep statement isn't appropriate for the current instance of the test.

Your script might Sleep for 4 seconds, because that's how long it took for some important object to appear when you created your script. Then the script moves on with actions using that new object.

But when you re-run the script, there's no guarantee that the object will appear within 4 seconds this time. Your script which depends on the object being present after 4 seconds might fail, or run amok, or report errors - all of which you must later analyze.

What you intended to script was "Sleep until the desired object appears". Sometimes "Sleep 4 seconds" doesn't represent that intent very well.

Hard-coded Sleeps make your script less tolerant of varying timings, more brittle, and less likely to carry out the actions you intended.

Answer 5

In software, there are a couple of ways to wait for an event to occur. One way is to cause the event to notify you. Sometimes that is possible, sometimes not. Another way is to poll: you wait a while, check whether the event has occurred, and if it hasn't, you do it again.

A variation of polling is to wait a while and then assume that the event has occurred. Of course, if you do that, you had better have waited long enough. Sometimes an explicit sleep represents a guess at a wait time. That guess might be fine in one circumstance but wrong in another. Your guess may be wrong as soon as you move to a slower machine, a slower network connection, or a server that has slowed down because it is having to work harder.