How important is it to test a build against a clean data set?

Question

Our web application is complicated, as are most modern apps. Typically, at the end of a release cycle, we branch our code and perform a regression test on the code. Additionally, all new features are thoroughly tested, as expected.

Because our application is complicated, and populating a set of data in order to perform the tests is time consuming, our QA utilizes the same database from previous tests. This database is old, probably over a year or two.

My position is that testing against a 'dirty' data set is approaching things incorrectly. From my understanding, when you perform a test, you want to test the application when the data is in a controlled and known state. What's happening now is we don't know the current state of the data, therefore when bugs come up we can't say with confidence whether the bugs are new, genuine bugs, or bugs that are the result of bad data. This can have us chasing our tails, and can lead to lots of wasted time.

My question is, am I correct in my position that we should have a clean test database, that is prepopulated with data in a known state? And if this is correct, how can I convince our QA that this is the right way to go?

Answer 1

As always, "it depends".

Some of the times you'd want to use a clean test database:

• You're performing regression: you want to start regression from a known starting point.
• You're testing setup and configuration issues
• You're comparing results against some baseline. Here you're going to want to start from the exact same condition your baseline started from.
• You're testing a new feature and you want to be sure that none of the issues you find are affected by older data (you'll do that testing later, with a different data set).

Some of the times you'd want to use a dirty test database:

• You're testing upgrading to a newer version and you need to make sure everything converts correctly.
• The problem you're looking at involves a long, complex setup and you already have that setup in place.
• You're looking at how a new feature interacts with existing, older functionality and data.
• You can't reproduce the problem without customer (or as Joe put it "wild") data. This is particularly common if customers are experiencing problems because they've been running the same database since 2002 and never archived anything (Yes, I've worked with customer data like that. The performance issues and database optimisations needed are... interesting). It can also happen with complex systems that have so many possible configurations you could spend a year tracking down the exact configuration that will reproduce the problem.

Both ways of testing have value, and both are useful. At my workplace, all the automated regression is done from a known if not necessarily clean starting point, and most of the testers will start a new, cleanish setup for each new major version we need to test with, which will grow progressively 'dirtier' the more different issues and developments we test.

One tactic we have found very helpful is to keep and archive (in known, accessible locations) data sets for specific configurations, especially the most time-consuming and easily misconfigured ones. That way, any of us can make use of a pre-set environment for pretty much everything without too much pain. You may find that your QA people are more amenable to the idea of starting semi-clean with a preset database containing most of what they need, and archiving databases with the more time-consuming feature sets so that everyone gets to benefit from them.

Answer 2

There are times I want to do that and have in the past, my clean/start data was set to copy over when I needed it and I had my Gold Standard to compare against. I could run this each time and be able to check where the system went wrong by checking against my Gold. However, things have changed. With Customer environments in flux, quick develop and release cycles and a need to pull down production data through which we troubleshoot issues and check new functionality against existing data has caused me issues. So there is what I WANT to do, and what I HAVE to do with the contraints I am in.

For that I have made certain allowances for the environment, and while not always for the better they serve us well.

• We have a few standard accounts that exist so we can check new functionality against existing accounts and permission sets
• With each data refresh there is a process of "cleaning the system" to prevent accidents such as emails going out to members of the site, in addition there is a "system prep" set of steps that happen where we Register specific Users to check various settings, attributes of Users through the registration system
• Registration is manual because we use Captcha, I have an open request to bypass this in Test as it did not always work and earlier I did load testing by running a registration script - now all I check for is that specific fields are good
• Because of the nature of our site, we have live data getting added every week, and the reliance on backend functionality to handle this data we refresh every few months. It's tough work because it then means a day or two of bringing the data down and then cleaning/prepping for use

Do I want to be working like this? Not really, but I understand the business constraints and they drive what I have to do. I set up my tests as stories to know what data I need for an account and then adjust it to be new so I can run my test. There are a suite of tests I have that tell me that the site is working, and data is moving properly and those tests never change, if they fail then I figure out what.

I guess I am a dirty tester but it works for me and our product because of the business requirements and I need to satisfy those in test and in prod so I have to be able to check them in both locations. Not fun, but I've adjusted and I've adjusted my test expectations accordingly to this.

Answer 3

From what you describe, using a clean test database makes some sense, at least for some of the testing. It can certainly make analyzing some bugs that much easier.

That said, realize that you may be missing a class of bugs that would only appear in the presence of data that you don't happen to have in your database. Perhaps a feature has been added or modified that demands new data to exercise those changes. Or perhaps "data in the wild" would present conditions that couldn't have appeared before, or had just not been considered before.

We often do both - we use a canned dataset for regression testing, then keep adding to it as features are added and enhanced. But we often use a snapshot of production data (data in the wild) for some testing (with all personally identifiable data masked). And we often synthesize test data using pseudo-random input for testing specific cases, sometimes with self-identifying fields that make it easier to debug. Finally, we sometimes create specific sets of data for stress/performance/load tests.

Answer 4

It is helpful for your tests to be repeatable. If you start from a known state, and you know what actions were taken before discovering a bug, it is easier for you to describe those circumstances to someone else. This is helpful to you, to the developer who must fix the bug, and to the tester who must test the fix.

On the other hand, the world is not repeatable. It changes moment to moment, and you would like for your software to work in each of those moments. Which moment should you test against? There is place for tests that start in a known state, but you must decide whether that state is representative of the real world. If not, there may also be a place for dirty, unpredictable starting points, coupled perhaps with additional logging.

Answer 5

Certainly You are correct. This is how I see it 1. What you are proposing is test the application with best cases. To make sure that whatever newly added works fine in addition to the existing features.If it works, then you can push new features to users. 2. What they are doing is testing with odd scenarios, which may or may not be required based on the relation of new features with existing features.

Reasons to convince them to use new database 1. We are in AGILE world, we need to release faster.. competition is tough. This will help your team to build and release faster. 2. Odd cases testing can be a secondary parallel activity along with testing the new features. 3. To test new features with old DB, they will need to partially update the db as per the feature's dependency. then why not do it completely.

hope this helps.