How to make sure your test data are correct?

Question

We are doing integration tests where we simulate manually data coming from external systems to our application. Integration tests task has become bottleneck in our team, because preparing test data is hard. The root cause of the problem is in complex format of test data:

• Hard to learn. Developers work on already parsed data, so they don't care about format. But at integration level I must know all the nuances of the format. It takes more to understand the format than to write the test itself.
• Error-prone. In the end you don't know why your test fails. Is it because of the system under test or incorrect test data?

Those are no so big problems for testers that have been involved in the project for long time, but are real obstacles for new-comers like me. So what we're doing:

1. We're creating test data generators. We're encoding into the generator the knowledge from documentation and answers of business analysts, other testers, and developers. Test data generators contain pre-defined blocks of data that a tester can compose together to create complete test data. A tester must learn DSL (based on Java), obviously, but I hope it is easier than the final format.

2. We're creating test data validators. Test data created by a generator are validated against some basic constrains (e.g. mandatory fields, values correct across different part of test data) before they get serialized to a specific XML format. That eliminates basic mistakes.

What else can we do to simplify the process of test data creation?

How can we make sure test data are correct?

Answer 1

Generating test data is a difficult problem because if you don't understand the symantics of the data then you are likely to generate test data that will throw false positives in your tests (test failure due to faulty data, not a bug in the product).

The approach I have used with great success is parameterized test data generation from equivalent partitions. I have used this approach with a wide variety of data such as Unicode strings, contact info on devices (e.g. names, phone #'s, etc), and JSON blobs for both positive and negative (fuzz) testing.

Essentially, this aproach requires modeling the data, decomposing the data into equivalent subsets, and then generating test data that will satisfy the constraints of the model. (It actually sounds like you are on the right path).

To get an idea of this approach see my white paper Parameterized Random Test Data Generation this white paper from MS Research. I also have a post here.

Of course a model is an abstraction of reality, and so your generated data is only as good as your model, your equivalent sets, and how you parameterize the use of those sets in your generator. Data generation can become quite complex and could also include weighting specific values, sequencing, etc.

With regards to validating the generated data, ALL oracles are heuristic in nature. So, in this case you validate the generated data satisfies the model. It is not bullet proof but it is better than gross random, or guessing.

Finally, don't forget to use real-like data also.

If you have more questions or I can help, let me know.

Answer 2

One reason why test data are invalid can be buggy generators.

Bugs can be in any software, including test automation framework. Therefore, test data generators should be also reviewed and tested. See more in "Lesson 118: Test automation is a software development process" in Lessons Learned in Software Testing.

It may seem that testing the testing framework will lead to testing ad infinitum. In practice, unit tests by definition must be simpler than the tested system, so they will not need to be tested. Second, instead of verifying new test data every time you write it, you will verify a generator only once. Finally, with tests for generator you're also more confident that fixing a generator will not introduce any new bugs.