I've used lots of different tools - some commercial, some open-source, some home-grown - for test automation. I usually use a mix of such tools in my overall automation efforts.
Over the years, I have found some nice-to-have features and attributes that I end up looking for, or building, as I assemble a new Test Automation Suite. Some of these attributes are part of the tools themselves. Others come about because of the way I assemble my Test Suites and tools into a complete package.
Some things are must-haves, and most are obvious:
- Run in my environment
- Automate my System-Under-Test
- Be able to "see" most of the objects in my System-Under-Test
- Usable by my test team
- Be affordable
- Be generally more efficient than strictly manual testing
Other things are nice-to-have:
- Detect changes in the System-Under-Test
- Create Smoke Tests which run after every build
- Run unattended
- Run overnight, and have a report ready the next morning
- Automate the boring, repetitive stuff
- Run predictably and repeatedly
- Randomize
- Perform timings
- Run some load, stress, and volume tests
- Isolate failures easily
- Run many tests, in spite of unexpected failures along the way
- Start wide, build depth later
- Automate what users do first (Getting Started Manual?)
- Isolate the maintenance effort
- Produce "readable" scripts
- Ability to reset the environment as needed
- Avoid false failures
- Extensible - since we cannot predict all uses
- Survive trivial changes to the System Under Test
- Validate during tests, and at the end as appropriate
- Ability to select and run subsets of the entire test suite
- Ability to select and skip particular tests
- Variable log levels (Verbose, Normal, Minimal)
- Minimize dependencies between scripts
- Minimize learning curve
- Minimize maintenance time
- Minimize post-run analysis time
- Minimize dependence on golden machines
- Record and Playback capability
You can find more details here:
http://www.allthingsquality.com/2010/04/things-i-like-to-have-in-my-test.html