Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I've read various places that explicit sleep statements are a bad practice in automated tests, even if I've used them occasionally. They can be annoying (creating unnecessary waiting in tests) but I can't think of any examples of why they're actually harmful. What are some examples of sleep statements causing harm in automated tests?

share|improve this question

5 Answers 5

up vote 19 down vote accepted

A hard coded sleep statement is generally supposed to represent some sort of mocked delay in the application that doesn't exist during testing.

The harmful aspect of this is that a hard coded value can't represent the complexity that actually exists in what its mocking.

Take for example a network delay. Your production system usually takes 4 seconds to go out over the network, take some data, and come back over the network with the result. So you simply mock up the data, toss in a 4 second delay, and give it back. Sounds simple.

Because it is simple - almost too simple. If something on your network changes, your test is now inaccurate. You can't see how the routine will react under abnormally heavy or abnormally light network load. In other words you're trying to test what you cannot control. You're make something repeatable (with the 4 second sleep) that really isn't repeatable (no run over the network will ever be the same twice).

And if the correctness of the test relies on an invalid and potentially often changing assumption, then what does that say about the reliability of the test?

share|improve this answer
    
+1 This is the kind of answer I'm looking for. I've mostly tested only desktop applications without much concern for networking aspects, so this helps me picture what can got wrong. –  joshin4colours Sep 7 '12 at 1:03
    
Excellent answer. But what can one do instead? If the explicit sleep with a constant is unwise, what would be a better method for this kind of testing? It does need to be tested and passed, but how to do that if not by 'sleep' with a constant? –  Mitch Sep 7 '12 at 15:27
    
I would add that a long sleep can be just as harmful. You might never fail because of a lack of waiting, but you will make every test take way longer, which can add up quickly and can be pretty counter-productive when you're running continuous integration. –  Sam Woods Sep 7 '12 at 18:52
2  
@Mitch You should poll for some state. If you have UI automation that can't continue until a new page loads and a "continue" button is visible and enabled then you should poll for the existence of the button, then poll the state of the button until it is visible and enabled. Polling is simply retrying with very short sleeps in between (100ms or so). You will still want to eventually time out too, because at some point it is a failure if it takes too long for the page to load and you don't want to be stuck in an infinite loop if the button never appears. In Selenium this is Explicit Waits. –  Sam Woods Sep 7 '12 at 18:58

I have a series of about 50 Selenium tests that take less than 10 minutes to run. I've been able to cut the execution time down by several minutes by writing the tests in such that no sleep() is required to know when the output has arrived.

I find that when I use sleep() in my tests that it's a kludge for when I don't have a good way to actually monitor the state of the thing I'm testing for when it's actually finished with what it's doing. Using sleep() amounts to "Are you ready yet? Are you ready yet?" @Dmitry Zhariy points out that adding sleeps enforced delays in your tests. This is bad. Your tests should always run as fast as possible. If there are unavoidable delays in the operation of your project, so be it (but try to see if you can find a way to make even those go faster.)

share|improve this answer

Well, when the test wastes your time – that is harmful I think. But, it depends. Example: you have several test environments. MachineA is hight performance real machine. MachineB – is slow virtual machine on remote server in the middle of Antarctica.

On MachineA you need 1 sec. sleep. On MachineB you need 10 sec. So, you make Sleep(10s) in the code which waste the MachineA performance.

So now your test time depends on the slowest machine in your lab. Let’s say you have 100 tests with a single 10 seconds delay. 100*10 = 1000 sec = 17 minutes of wasted time per each run on high performance workstations.

On my previous project we have fight a lot with Sleeps and have reduced the test execution from 12 to 9 hours for 1200 tests. My colleagues that used came to the working place early were not waited additional 2-3 hours for the test suite before it finishes.

But anyway we left some sleeps that we were not able to fix fast.

share|improve this answer

The main harm occurs when the time value you use in your Sleep statement isn't appropriate for the current instance of the test.

Your script might Sleep for 4 seconds, because that's how long it took for some important object to appear when you created your script. Then the script moves on with actions using that new object.

But when you re-run the script, there's no guarantee that the object will appear within 4 seconds this time. Your script which depends on the object being present after 4 seconds might fail, or run amok, or report errors - all of which you must later analyze.

What you intended to script was "Sleep until the desired object appears". Sometimes "Sleep 4 seconds" doesn't represent that intent very well.

Hard-coded Sleeps make your script less tolerant of varying timings, more brittle, and less likely to carry out the actions you intended.

share|improve this answer
1  
+1 for What you intended to script was "Sleep until the desired object appears" –  Alison Sep 10 '12 at 13:34

In software, there are a couple of ways to wait for an event to occur. One way is to cause the event to notify you. Sometimes that is possible, sometimes not. Another way is to poll: you wait a while, check whether the event has occurred, and if it hasn't, you do it again.

A variation of polling is to wait a while and then assume that the event has occurred. Of course, if you do that, you had better have waited long enough. Sometimes an explicit sleep represents a guess at a wait time. That guess might be fine in one circumstance but wrong in another. Your guess may be wrong as soon as you move to a slower machine, a slower network connection, or a server that has slowed down because it is having to work harder.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.