Should automated tests be efficient?

Question

I've written and edited a bunch of automated GUI tests. In some cases, the tests are inefficient (eg to select an item from a list, instead of a clever search, a dialog is opened and starting fro the top each item is selected until the item is found). This creates tests that run much longer than they could, but is this a problem? Most of the time the tests I run are run overnight unattended, so it's not too relevant if they take 4 or 10 hours to run. On the other hand, even small efficiencies could make a difference.

I think it doesn't matter so long as the tests are complete and up-to-date. Any other opinions?

EDIT Reading some comments, this question is a bit open ended and biased, so let me provide some more details. The tests I run are GUI tests (using SilkTest), so they run in almost real time to simulate and end user. So reducing test run times from 4-10 hours to 4-10 minutes isn't possible. It may be possible to reduce them from 4 hours to 3 hours, however. As well, the tests may be relatively difficult to optimize. Any ideas for reducing test time for GUI tests?

Answer 1

Any ideas for reducing test time for GUI tests?

One tip that many seem to overlook is to ensure that all your tests are actually useful. Many times, tests decay over time - their usefulness fades to the point that the tests themselves are a waste of time, no matter how quick they are.

I once joined an organization that ran automated overnight tests which emailed a completion report to a large group. After a few days I noticed that it always indicated an overall "Pass", yet my manual tests indicated major failures. When I dug in and looked at the automation, I found that the code was a mess and had no chance of actually detecting any real bugs. I removed most of the automation, and replaced it with a much smaller, shorter test that actually was capable of detecting some bugs.

Answer 2

I've been in this situation before. We were running real-time tests for similar reasons. I wouldn't optimize prematurely, but if you see a looming problem I also wouldn't wait until it's already an issue. Some things we did:

• Parallelized testing across many machines. Make it easy to select a subset of your tests, so you can break one test run into multiple mini-runs, and have a scheduler handle assigning those mini-runs to multiple machines. You will need to decide whether you want to run the same tests on the same machines all the time, for consistency, or switch around, for greater coverage (you may hit some bugs only on certain machines).
• Distinguished between feature tests and end-to-end tests. There is no need to simulate the user until you get to the feature you are testing for a feature test, but for an end-to-end test, you need to simulate the entire way through. You might be able to move quickly during setup, slow down and simulate during the actual test, then speed up again during cleanup for feature tests.
• Used configurable constants for periods of time between actions. Not only will this allow you to easily adjust the runtime if you find speed is becoming more important (and then maybe do a 'slow' run on the weekends), but by reconfiguring things to run at different speeds occasionally, you get more coverage (some nasty bugs are only hit if you do things faster than normal).
• Broke out time-consuming computations, so we had a "test" phase that just logged stuff, and an "analysis" phase that sorted through the information to get useful statistics. This was performance GUI testing, so we did a lot of data analysis - it might not apply for your situation.
• Note priority for each test, and don't run all tests every night if time is an issue. You can run low-priority tests just once a week to save resources. Or run tests in priority order, so you get the most important results first.
• Take notes on the slow pieces, give them rough time estimates to fix, and watch for low-hanging fruit.

It's often cheaper in the long run to pay for more hardware and parallelize than more tester time to optimize and re-stabilize GUI tests. At the extreme end, you could run one test per machine for hundreds of machines and finish the entire run in the time it takes to run a single test (including test run setup - e.g., do you reinstall the OS before doing a test run?). There is an initial cost to tester time to organize multiple machines (some sort of queueing service is needed, plus alerts for non-responsive machines, etc.), but it pays off going forward - if your product and test suite is big enough!

OTOH, if you've been working for 5 years and the number of GUI tests grows steadily and you are at 4 to 10 hours now, you might be able to have the number of tests double over the next 5 years and still run everything on just one machine with only a few simple optimizing strategies, like running in priority order and cutting the run short if it takes too long.

Answer 3

Yes, however this is one of those circumstances when it depends, an the final answer on "how efficient" will be different for everyone.

Why it matters

• If you use test automation for a smoke or BTV test, then you want to be able to execute broad coverage quickly
• If you need to debug or re-run a test to see if a failure occurrs you don't want to be sitting around for minutes waiting, you want to execute quickly.

Why it doesn't matter

• If you have enough harware (and licenses) to scale our your execution, as long as you can walk out the door of an evening, and walk back in the following morning and 100% of your tests have executed, then they are run fast enough.

With that said, once you have had tests that run as fast as the hardware will let them run, (i.e. the bottleneck is the sever response time) everything else just feels slow and inefficent.

Answer 4

Efficiency is relative to your needs. Rather than ask, "Should automated tests be efficient?", it might be more interesting to ask, "When does automated efficiency matter?" For example, if you expect a developer to run your automated test before checking in a change, it matters a great deal how long it takes that test to run. You could probably poll your developers and find the pain threshold beyond which developers will stop using your tests -- it is likely they have strong feelings about it.

Another thing: I have always suspected -- but have never tried to test empirically -- whether people are more patient with their own automated tests than with other people's automated tests. Perhaps that's because, as I have stated in answers to other questions, the one person who really understands what an automated test does is the person who wrote it. When you aren't sure what a test actually does, or more to the point whether it does anything useful, it's harder to justify taking the time to run it.

Answer 5

Automated tests should be just efficient enough, and no more.

"This creates tests that run much longer than they could, but is this a problem? Most of the time the tests I run are run overnight unattended, so it's not too relevant if they take 4 or 10 hours to run. On the other hand, even small efficiencies could make a difference."

Not trying to use circular logic here, but it's only a problem if it's a problem in your environment.

If you have an overnight available to run all your tests, and you will only ever use 10 hours to run your inefficient tests, then it probably doesn't matter. Time spent making your tests more efficient is probably wasted. You can always "make it better", the trick is getting to "good enough" and moving on.

On the other hand, if at some point you need to run so many tests that you start to run out of available overnight time, then it will be a problem.

Only you can determine if there's a problem, a potential problem, or no problem at all.

Answer 6

Scaling is always important to keep in mind. If you doubled the number of tests (which will happen eventually) to so now it goes up to 20 hours, would this be a problem? Maybe. You would start it at 5 when you leave, and get no feedback until after lunch the next day.

Another thing I'd consider is how fast could they be? If you were to take them from 4-10 hours down to 4-10 minutes, then you could run them hourly instead of daily. That seems well worth it. If you would take it down to 3-8 hours, then you're probably not gaining as much from it.

Another thing I'd consider is the effort required to optimize. How long will it take to optimize? Have you tried optimizing some of them? Can you do hard work on one and reuse that hard work on others?

A question to ask is... would it be just as easy to spawn a second machine to run tests on, and move some of the suites there? If it's going to be 40 hours to do some optimizations, and only 8 hours of labor plus some capital expense to "throw more hardware at it" it makes good economic sense to do that.

The bottom line is a simple cost-benefit analysis. Determine the benefit gained from optimizing it, and compare it to the cost of doing so. Do some research (which in this case consists of doing some optimization to get an idea of the scope of the time required) and crunch the numbers.

As a final thought, I bet any manager/interviewer would be glad to know the person on/about to be on his staff would be glad to know you're able to take the initiative and make the infrastructure better for the team!

Answer 7

I think you should work towards fast feedback. Using parallel test execution your testrun can theoretically take as long as your slowest test. A testrun which completes in minutes instead of hours helps you work in smaller faster cycles, catch errors faster, improve your testsuite faster etc.

Developers want to know a change they've made caused a bug in the system as soon as possible. If they hear about it the next day they will have to freshen up their memory about what the change was about. If they hear it a half hour after the fact it is far easier to fix it because the change they've made is still fresh in memory.

Parallel execution is probably step 1 in making your tests more efficient. A testrun which takes 4 hours takes approximately 1 hour if you run 4 tests simultaneously and will take approximately 8 minutes when you run 30 tests simultaneously. Parallel test execution can get hard to setup depending on the sytem but it could be worthwhile.

After making your testsuite run in parallel you can pick out the slowest tests and optimize.

Answer 8

I'm in the same boat, where 12-16 hours for tests to run is acceptable. However, if for no other reason than editing and debugging, efficient is the way to go. Although it may take you a while to write the clever search, it will save you a lot of time in the long run. Additionally, if anything ever happens where you have less of a window, there's no need to refactor to make it more efficient off the hop.

As a second point however, I bet that your team values the efficiency of the code that you're testing, you owe it to your team to write your automated test code just as efficiently.

Answer 9

When you are assuring the quality of the product, the quality of your test script is as important. Hence, the automation should be very efficient. (This is more of a good attitude than a requirement.) In the real world, the automation should be as efficient as practically possible.

Some ways:

• **Modularize the tests** - If the tests can run in entirety and independently, it allows you to execute what you need. A lots of parts of the application are never/not so frequently affected. With a good judgement, the need to run all the TCs all the time can be eliminated. We used to have a joke in our team, if a bridge I-5 near Fresno is reconstructed, no need to test the entire freeway from San Diego to Seattle.
• **Smoke/Sanity Test** - Having an efficient smoke test which can run regularly instead of the full fledged test. A sanity test can be run regularly and more frequently than the full fledged test suite.
• **Confidence level** (gray area) - It is extremely important for a QA team to determine confidence level in different modules of the application, so that the team can know which modules need to be tested more frequently than the others.
• **Housekeeping** - As already mentioned, it is very important to 'maintain' the test suite and deprecate or remove the TCs that are no longer important. As the product gets matured, quite a few TCs can be eliminated, which will save a lot of time. e.g. If a module in the product calculates the formulas, once the product is matured and those formulas are not being changed, fewer TCs should suffice to sanity test that module rather than going through numerous combinations.

Answer 10

Fast Feedback Loops Are Good

The faster the feedback loop the quicker you can act upon errors. Assuming this is a nightly run 24 hours is a long feedback loop, especially if you are working in an agile environment.

I would aim to get your entire test run running in 15-30 mins on a CI server that automatically runs the tests when code is checked in. This gives the devs fast feedback so that they can pick up (and fix) errors quickly, or they could even run the tests on their own machines.

How would you do this?

1. Run individual tests in parallel.
2. Write reusable code segments that you can use across a series of tests.
3. Remove pointless tests that have no value.
4. Run the tests on a more powerful machine.

What's In It For Me?

1. Reusable code segments will speed up future automation.

Write one fast and efficient function for checking drop down lists where you supply it data you are looking for. Reuse this for every test and you have removed lots of slow and inefficient code as well as reducing the amount of effort it will take to do drop down list tests in future.

It's easier to refactor a code segment that is used in multiple tests so even if it's not the most efficient piece of code in the world right now, you can always come back to it in the future and improving its efficiently will give you a speed boost in every test that you have written that utilises this function.

2. I Get More Testing Time If The Developers Know Thier Code Is Broken Early

Once you have made all the efficiency savings you can, stripping out the chaff and running your tests in parallel is the next step. Creating two test slices will half your run time (assuming you do it intelligently and place the tests based on their run time). Every extra slice you add will reduce test time and help you get down to that magic 15-30 mins run time.

Now get a CI server up and running and you can tell the developers that something is wrong within 30 mins of them checking in their code, they will still have that story fresh in their minds and should be able to quickly switch back to the code and fix the problems for you. If you have a 24 hour feedback loop the developers will probably have moved on to something else and forgotten what they were doing yesterday, it will take longer for them you work out where they were and longer for them to perform a fix.

As a tester you are at the end of the development cycle and if a deadline is fast approaching it is your time that is squeezed so that the project can hit the deadline. Speeding up tests as suggested above will give you more time to test. Assuming the devs fix each problem within 15 mins your CI server running every 15 mins could catch 14 different errors over the period of one day that would have taken it two weeks to catch if you were running the tests nightly.

That's potentially 13 days of test time saved, do you need a better reason than this?

Answer 11

Answer @corsiKa - Any ideas for reducing test time for GUI tests?

One point that I have experienced is for the GUI test, many times a tester add delay / think / wait time between every action, so that objects become available before the click / action. Usually automation tools provide the function / api as waitUntilVisible(object). If the function is used instead of random wait time (which is used mostly to complete the automation quickly & easily), the run time for the automation reduces significantly as number of steps increases significantly over the period of time.

It's a simple tip but I have seen many times, it gets ignored due to meeting deadline pressure, inefficient programming etc.

Answer 12

Additionally to the great answers seen here, I'd like to add the following:

• Have different test sets, one small "smoke" test set containing only a few tests to run on each CI build, and a more thorough one running over night.
• Avoid Sleep statements in your tests. Those are almost always a sign for bad test code, as they make your tests slow and/or unreliable. On fast machines, you're wasting time, and on slow machines the sleep-time might not be sufficient. Automated testing tools usually provide means to do some sort of synchonization, for example waiting for a dialog to appear or disappear.

The following ones fall in the category micro-optimizations, so they might not make a difference, but in some cases they will

• When finding objects, try to reduce the amount of objects that has to be searched. For example, if you want to find a button in a dialog, don't search through all objects starting from the root object, but search only within the dialog.
• Silk Test allows you to re-use handles, without having to re-find them. If you want to perform 20 actions on a dialog, it is ok to search for the dialog once and then do the 20 actions, rather than searching for it before each step.
• Also Silk Test specific: You didn't mention if you were using Classic Agent or Open Agent, but for new tests I suggest you use the Open Agent, as it is a lot faster than CA (You can also do mixed agent scripts if you want to migrate slowly).

Note: I work for Borland as part of the Silk Test team, so any opinions expressed are highly biased.

Answer 13

Automation is powerful, but not enough for a better product. Manual intervention is still important. However, automation allows test teams to focus on new functionality, non-automated functions, maintenance needs and test data needs, thereby enhancing and ensuring quality. This offers increased customer satisfaction and enhances the customer’s trust in the product as well as the capabilities of the company. [blog]: http://www.ivesia.com/technology/blog/automated-testing-in-software-product-development-2/ "click here for updates"