Fixed vs Dynamic AUT

Question

We develop automated user interface tests. The application under test is quite complex and it can be configured by the user (the real user) in many ways. Our tests to be automated are quite complex as well and perform many actions on the application.

My dilema is which strategy to take:

1.Tests are executed in a secure and well known application configuration. In every test execution this environment is restored to the previous state (by restoring the database, or by restoring the virtual machine, using Vagrant or any other technique).

2.Test are executed in any application configuration so the tests embody themselves the intelligence to detect the configuration at run time and change it so the test will not break due to a bad configuration from a previous test.

The first aproach lets you to develop tests quickly and execute them in a stable application. You know that tests will not disturb each other. On the other hand, tests can be executed through any installation and configuration and but these tests are more complex to implement so the time to develop them is higher.

Which aproach do you use? Which are the pros and cons of both techniques?

Answer 1

That "stable state" is a necessary concept for test automation (I usually refer to it as the "idle state"). However, it is important to try to run the tests in a variety of configurations and not just the single stable one. Here are some ideas that I hope may be helpful:

1. I try to write my scripts such that they return to the idle state after each test run.

2. You can have more than one idle state. That allows you to organize your scripts into logical groupings, thus cutting down on the configuration setup from each idle state. This gives you some flexibility in implementing a variety of configurations.

3. You may want to consider instead of an "automated" approach to run a "computer assisted" approach. In that case, the automated scripts do all of the configuration for a given scenario and then turn the application over to a tester to run the individual tests. This has the benefit of reducing your automation complexity enormously, but does require the scripts to be run interactively. It also allows the users to investigate any oddities that may not be picked up in the automation.

4. It is important to differentiate between an idle state and a test configuration. You can write your automation to provide a variety of data sets in a single setup, such that separate tests can be performed on different data sets without the need for resetting the automation. I am not sure from your description whether your configurations are mutually exclusive or can coexist in a single session.

Hope that helps!

Answer 2

This is one of those questions which has multiple valid answers, as well as being a common problem for test automation engineers.

The approach you take should be influenced by your knowledge of your user base:

• Do you know which configuration settings your users use?
• Are there known configuration settings used by your largest customers?
• Are there common configuration sets that can be identified and categorized by how common they are?

If 90% of your users use one configuration set, you can be reasonably safe focusing your automation on that specific configuration. If you have three or four common configuration sets, you can work with those. And if you have a very large customer who provides a sizeable chunk of your company's revenue, you absolutely must automate their configuration.

Some of the ways you can do this (depending on your exact circumstances):

• Use predefined data sets and restore them into your test database before beginning a test run: the advantage here is that you know what you are starting with, and can abort the run if restoration fails.
• Use 'smart' configuration detection - detecting the current configuration settings in conjunction with a known start point allows the automation to set exactly what is needed at any point in time.
• If the application has a large number of dependent operations where you need to perform a large number of tasks before you reach your target, consider making your tests dependent, but with validation on the fly. This is a difficult decision that is also influenced by your test resources, how many parallel tests you can run, and the need for timely reporting on automation results. I couldn't say if it's the best option for you, but it has been the best option for me more than once.
• If all your customers use the same database (this is common for web applications) and it's not practical to restore a clean database into the test system maintain a set of configurations and data as test 'customers'. These can start life as snapshots of customer data with the appropriate anonymization and data cleaning. Doing this means your database checks need to be smarter, and you need to use procedural rather than programmed methods to keep others from messing with your test data, but it can be done.
• Focus on the goal of your tests, not the process. It's easy to get lost in the process of writing automation and forget the goal of the automation. If the goal is to have the 20% of the system that gets 80% of the use automated, you don't need to worry as much about automating scenarios for the customer using a configuration nobody else uses. If there are large parts of the system that behave the same regardless of configuration, those can be tested separately with a common configuration set to minimize the number of configuration switches you need to make.

No matter which way you take it, you do need to have a means of ensuring you've got known data, and a means to reset the AUT to its known start state.