Random test data: How to do it right?

Question

Several months ago I started an experiment in test automation. Each test, before it starts, picks up random test data. For instance, when my test requires a user with administration privileges, I draw a random account and then I from a group of admin users for this account I draw a random admin user.

The reason for using random test data was two-fold:

1. First, when we were using static, i.e., hard-coded test data, sometimes tests started to fail, because test data no longer existed in DB, e.g., a user has been removed from the database (we're having production dump in our test DB, so I guess the user has been deleted in production). When we started to find such users at runtime, the problem disappeared.
2. Second problem was that it was not really clear why certain test data were used for a certain test. We have inherited tests from the other team and couldn't understand their intentions behind, for instance, using user "[email protected]". By writing queries to find data, we started to explicitly state what type of test data we want, e.g., that we want a user with administration privileges.

While my approach solved both problems, it introduced also some new issues:

• Tests occasionally fail, because they pick up wrong test data. This is because queries are sometimes incorrect. For instance, once a test drawn a random account that had no administrators at all. Good thing about that is that this way I keep learning new things about the system under tests, e.g., that there must be business reasons to have accounts without administrators.
• Test setup becomes more and more complex, as queries start to grow and thus tests seems now a bit more complex to understand.

I was also hoping to increase coverage and find more bugs by using random test data just like it is done in fuzz testing. Instead, I mostly found bugs in my test and learned how little I know about the system under test. Finally, ROI doesn't seem big. Before the change there was a cost of maintaining static test data (updating test data if they were outdates). Now I have to maintain complex queries when the test takes "wrong" test data and this tends to occur more often than outdating of static test data previously.

I wonder whether using random test data really makes sense and how to do it right?

Answer 1

Great question. I see two issues, as follows.

Using random data may lead to unrepeatable results. You can mitigate this by logging (or otherwise recording) every random choice you make, and then playing those choices back. That could be as easy as recording the initial seed to your random number generator, assuming your data does not change over time.

It can be hard to write tests in a general enough way to deal with arbitrary, randomly selected data. This is the harder problem. Choosing a different random integer is one thing; choosing a different user (with properties that vary from one user to the next) is something else. Consider how a test works in the abstract: it chooses some inputs, applies them to a function, and then verifies that the result is correct. There are a few ways to approach that:

• The developer precalculates the expected result and hardcodes it into the test. That only works if the developer decides on the inputs ahead of time.
• The test uses the inputs to calculate the expected result, and then compares that to the actual result. If the test can use arbitrary (random inputs), this means the test needs to replicate a lot of the logic in the system under test. This is almost certainly not what you want, especially for a complicated system. You are just as likely as the developer to implement a complicated algorithm in a buggy way.
• The test uses some other means of verifying that the result matches the input. Sometimes there are shortcuts, or at least alternatives, you can take to verify a result. As a naive example, if a system sums a list of numbers, you might verify the result but subtracting the list from the sum, and then checking whether the new result is zero. Most systems cannot be tested this way.

A good place to use randomly selected data is in a comparator test, where you compare two versions of the same system using the same inputs. A comparator test will not tell you whether a system is correct, but it will help you find changes in behavior. That might be something for you to consider.

Answer 2

However you obtain test data, you would like to know that it is appropriate to the intention of the tests.

When the test data is defined outside of the tests themselves, there is always the chance that the data will drift away from the intentions of the tests. The chance increases if the data is maintained by someone other than the test automators.

If you select test data from some predefined pool, your selection mechanism will have to apply all of the criteria that matter to your test. It's common (as you're learning) to overlook important criteria. Your tests end up making unwarranted assumptions about any criteria not enforced by the query. Or you apply all of the criteria, but as the test data changes, a query may eventually fail to find a satisfactory item.

The general danger: Making automated tests depend on anything they do not control can lead tests to fail regardless of whether the system satisfies its responsibilities. That is, your tests end up telling lies, and people very quickly stop trusting them.

There are a few ways to solve the problem.

Let tests control test data. One approach is to give each test control of its own test data. That is: Have each test create its test data, rather than select it from a pool of data maintained elsewhere. Establishing the test data can make each test run more expensive and time consuming. But it makes the tests far more reliable.

Fake the database. To solve the problem of expensive test data setup, you can remove the real database from the tests and use a fake one. The big advantage: You have control over the data without the overhead of setting up a real database. The big disadvantage: Your fake database might behave differently from the real one, and your tests end up telling lies. It's a tradeoff, made somewhat more palatable if you can create a fake database with sufficient fidelity.

Controlled randomness. With either of these approaches, if you want to randomize the data, you can do it directly, randomizing the particular attributes you care about, and randomizing them in ways that you care about.

Answer 3

Randomized testing certainly makes more sense than testing by example. Different situations would need different solutions:

• Random Users (or other objects): need that they are created every time test runs. Or they pick it by some query, but this is less robust.
• Testing validation with random data is the simplest - you generate required values with existing libraries and check if validation passed or not.
• Testing algorithms/calculations may require more complex approach - Property Based Testing. Instead of checking the output value you check the properties of the result.
• Randomized Behaviour is another nice technique when you can invoke random functions of your system. This can be achieved with Model Based Testing.

You can find more information about different techniques in this article.

Another question is how to simplify tests that leverage randomization:

• Most of the time you can afford creating your own domain model for the tests (or reusing classes used for production code). Such objects (e.g. User) when created can have all the required fields filled by default. E.g. if User needs username, password, email then you generate them randomly when you create a user (e.g. User.random()). If in tests you check username in particular, you can override it with the one needed for the tests. This way only data needed for the tests will be present in tests.
• Do not move data from the code into external files - otherwise you won't be able to randomly generate them or this will be much harder.
• Build test pyramid to keep most of the tests at Unit, Component levels. If you have thousands of System tests then the odds of collisions are getting higher because old records are left from previous runs. This may be an issue for the cases when you test min boundary for unique fields (e.g. it's 1 symbol). Lower level tests don't have this problem because they don't have storage or it's short living.
• Use specialized libs like Datagen for Java or random-ext for JS.