We are doing integration tests where we simulate manually data coming from external systems to our application. Our integration tests have become the bottleneck in our process, because preparing test data is hard due to the complex format of test data:

  • Hard to learn. Developers work on already parsed data, so they don't care about the format. But at an integration level I must know all the nuances of the format. It takes more to understand the format than to write the test itself.
  • Error-prone. In the end you don't know why your test fails. Is it because of the system under test or incorrect test data?

Those are not significant problems for testers that have been involved in the project for a long time, but they are significant obstacles for new folks like myself. So what we're doing is:

  1. We're creating test data generators. We're encoding into the generator the knowledge from documentation and answers of business analysts, other testers, and developers. Test data generators contain pre-defined blocks of data that a tester can compose together to create complete test data. A tester must learn DSL (based on Java), obviously, but I hope it is easier than the final format.

  2. We're creating test data validators. Test data created by a generator are validated against some basic constrains (e.g. mandatory fields, values correct across different part of test data) before they get serialized to a specific XML format. That eliminates basic mistakes.

What else can we do to simplify the process of test data creation?

How can we make sure that test data is correct?

  • 1
    I like that approach. An advantage of using a DSL is that your test cases can be more concise (and probably easier to maintain) than they would be in XML.
    – user246
    Commented Oct 22, 2012 at 2:23
  • Yes, they seems to be easier to maintain. For instance, cost of refactoring is lower, and I got immediate feedback about syntax thanks to compiler.
    – dzieciou
    Commented Oct 22, 2012 at 20:30

4 Answers 4


Generating test data is a difficult problem because if you don't understand the symantics of the data then you are likely to generate test data that will throw false positives in your tests (test failure due to faulty data, not a bug in the product).

The approach I have used with great success is parameterized test data generation from equivalent partitions. I have used this approach with a wide variety of data such as Unicode strings, contact info on devices (e.g. names, phone #'s, etc), and JSON blobs for both positive and negative (fuzz) testing.

Essentially, this aproach requires modeling the data, decomposing the data into equivalent subsets, and then generating test data that will satisfy the constraints of the model. (It actually sounds like you are on the right path).

To get an idea of this approach see my white paper Parameterized Random Test Data Generation this white paper from MS Research. I also have a post here.

Of course a model is an abstraction of reality, and so your generated data is only as good as your model, your equivalent sets, and how you parameterize the use of those sets in your generator. Data generation can become quite complex and could also include weighting specific values, sequencing, etc.

With regards to validating the generated data, ALL oracles are heuristic in nature. So, in this case you validate the generated data satisfies the model. It is not bullet proof but it is better than gross random, or guessing.

Finally, don't forget to use real-like data also.

If you have more questions or I can help, let me know.

  • Bj, Well said. I especially like the high "insight per word ratio" of these sentences: "Of course a model is an abstraction of reality, and so your generated data is only as good as your model, your equivalent sets, and how you parameterize the use of those sets in your generator. Data generation can become quite complex and could also include weighting specific values, sequencing, etc."
    – Justin
    Commented Feb 19, 2015 at 22:36

Test can return false positives or false negatives either because of incorrect oracle (and comparators) or incorrect test input data (aka test steps). To address those problems I found the following approaches in the literature.

Incorrect oracle and comparators

Binder in his book "Testing Object-Oriented Systems" suggests the following to limit the problem of designing incorrect oracle:

  • If possible, review some expected results produced by your oracle and your assumptions with system users. Values that may look correct to a developer or tester may be subtly incorrect due to some special constraints.
  • The more complex the oracle, the greater the chance of spurious test results. Try for the simplest solution.
  • If the oracle is specification-based, do not forget to verify the specification. Scrutinize the specification for omissions, contradiction.
  • Try the oracle for test cases that have obvious expected results – for example, all zeros in, all zeros out. Such test cases check the oracle and comparator.
  • If practical and feasible, try using several independent sources. For example, if you are picking values from a table in a reference work, try to find several other reference works that provide the same information. Interleave values from these sources. If you are usin an existing system as the oracle, try running the system in different configurations or platforms, varying the time of day, altering the background load, and so on
  • Although writing a program to generate millions of test case inputs is usually not difficult, producing their expected results is often equivalent to developing the SUT. Two of the oracle patterns can partially overcome this limitation. Built-in Test Oracle will detect some, but not all, incorrect output from any input. You may be able to use an existing system as gold standard oracle. Run the existing system with your test inputs. It will automatically produce some or all of your expected results (Trusted System Oracle pattern).
  • Design-for-testability tip: Consider abandoning a test strategy or test case if it requires a very difficult or costly oracle. Try to use existing code, files, or test suites as much as possible.
  • Design-for-testability tip: Consider reworking an application specification or requirement if its oracle would be very difficult or costly to develop.
  • Consider a partial or approximating oracle. Don’t assume that your oracle must generate complete expected results for every possible input and state. Concentrate on generating outputs that must be correct or that are difficult and/or time-consuming to check by hand.
  • Consider using several kinds of oracles to offset weakness. For example, you can use an existing system to generate about half of the critical outputs for a new system. You could implement built-in test assertions to check relationships on the newer output.

Incorrect test input data

One reason why test data are invalid can be buggy generators for test input data. Therefore, Lessons Learned in Software Testing suggest to unit test and review them as in any other piece of software.

It may seem that testing the testing framework will lead to testing ad infinitum. In practice, unit tests by definition must be simpler than the tested system, so they will not need to be tested. Second, instead of verifying new test data every time you write it, you will verify a generator only once. Finally, with tests for generator you're also more confident that fixing a generator will not introduce any new bugs.

  • this is a recursive answer and not very helpful...
    – Rsf
    Commented Oct 29, 2012 at 7:58
  • I updated my answer to show it is not recursive process, while still relevant for the invalid test data problem.
    – dzieciou
    Commented Oct 29, 2012 at 19:36

Since testers are entirely focused on creating test data that can be considered complete and offer full coverage to the testers, the correctness and quality of data highly define the test outcomes. The usual process which is adopted by the professionals from the industry includes taking data from the production environment and then creating flat files using mapping rules.

Furthermore, QA services simply retrieve SQL queries from the existing databases while leaning on automated test data generation tools to process the information.

If you need to have a detailed insight on how you can prepare test data for maximum accuracy or correctness, consider following this link.


  • Thanks. Can you summirize the content of the linked site? Links tend to become obsolete...
    – dzieciou
    Commented Sep 3, 2021 at 6:52

Use the same test data for all purposes

Typically in a modern organization you have test data in several areas such as:

  • the data used by unit tests in order to mock and stub dependencies
  • test data used for integration purposes
  • manually entered test data in the test sustem UI
  • scrubbed or created once data used by the test system UI

In each of these areas you have different groups, different people and different tools being used along with differing priorities about how to maintain the test data. Yuch.

One way to help ensure your test data is correct is to use the same data everywhere. This is an approach that I have had good success in using.

For example in one implementation I created a JSON file with the same format that the API returns.
I then use this as the source for unit tests, integration tests, server tests and the UI itself.
The result of these dependencies is that everyone has to to work together to keep data consistent and correct

This is a 'eat your own dogfood cake' approach

  • Hopefully no-one will wondier why cake is better than dog food ~ Commented Sep 4, 2021 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.