Several months ago I started an experiment in test automation. Each test, before it starts, picks up random test data. For instance, when my test requires a user with administration privileges, I draw a random account and then I from a group of admin users for this account I draw a random admin user.
The reason for using random test data was two-fold:
- First, when we were using static, i.e., hard-coded test data, sometimes tests started to fail, because test data no longer existed in DB, e.g., a user has been removed from the database (we're having production dump in our test DB, so I guess the user has been deleted in production). When we started to find such users at runtime, the problem disappeared.
- Second problem was that it was not really clear why certain test data were used for a certain test. We have inherited tests from the other team and couldn't understand their intentions behind, for instance, using user "firstname.lastname@example.org". By writing queries to find data, we started to explicitly state what type of test data we want, e.g., that we want a user with administration privileges.
While my approach solved both problems, it introduced also some new issues:
- Tests occasionally fail, because they pick up wrong test data. This is because queries are sometimes incorrect. For instance, once a test drawn a random account that had no administrators at all. Good thing about that is that this way I keep learning new things about the system under tests, e.g., that there must be business reasons to have accounts without administrators.
- Test setup becomes more and more complex, as queries start to grow and thus tests seems now a bit more complex to understand.
I was also hoping to increase coverage and find more bugs by using random test data just like it is done in fuzz testing. Instead, I mostly found bugs in my test and learned how little I know about the system under test. Finally, ROI doesn't seem big. Before the change there was a cost of maintaining static test data (updating test data if they were outdates). Now I have to maintain complex queries when the test takes "wrong" test data and this tends to occur more often than outdating of static test data previously.
I wonder whether using random test data really makes sense and how to do it right?