Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

We are doing integration tests where we simulate manually data coming from external systems to our application. Integration tests task has become bottleneck in our team, because preparing test data is hard. The root cause of the problem is in complex format of test data:

  • Hard to learn. Developers work on already parsed data, so they don't care about format. But at integration level I must know all the nuances of the format. It takes more to understand the format than to write the test itself.
  • Error-prone. In the end you don't know why your test fails. Is it because of the system under test or incorrect test data?

Those are no so big problems for testers that have been involved in the project for long time, but are real obstacles for new-comers like me. So what we're doing:

  1. We're creating test data generators. We're encoding into the generator the knowledge from documentation and answers of business analysts, other testers, and developers. Test data generators contain pre-defined blocks of data that a tester can compose together to create complete test data. A tester must learn DSL (based on Java), obviously, but I hope it is easier than the final format.

  2. We're creating test data validators. Test data created by a generator are validated against some basic constrains (e.g. mandatory fields, values correct across different part of test data) before they get serialized to a specific XML format. That eliminates basic mistakes.

What else can we do to simplify the process of test data creation?

How can we make sure test data are correct?

share|improve this question
1  
I like that approach. An advantage of using a DSL is that your test cases can be more concise (and probably easier to maintain) than they would be in XML. –  user246 Oct 22 '12 at 2:23
    
Yes, they seems to be easier to maintain. For instance, cost of refactoring is lower, and I got immediate feedback about syntax thanks to compiler. –  dzieciou Oct 22 '12 at 20:30
add comment

2 Answers

up vote 7 down vote accepted

Generating test data is a difficult problem because if you don't understand the symantics of the data then you are likely to generate test data that will throw false positives in your tests (test failure due to faulty data, not a bug in the product).

The approach I have used with great success is parameterized test data generation from equivalent partitions. I have used this approach with a wide variety of data such as Unicode strings, contact info on devices (e.g. names, phone #'s, etc), and JSON blobs for both positive and negative (fuzz) testing.

Essentially, this aproach requires modeling the data, decomposing the data into equivalent subsets, and then generating test data that will satisfy the constraints of the model. (It actually sounds like you are on the right path).

To get an idea of this approach see my white paper Parameterized Random Test Data Generation this white paper from MS Research. I also have a post here.

Of course a model is an abstraction of reality, and so your generated data is only as good as your model, your equivalent sets, and how you parameterize the use of those sets in your generator. Data generation can become quite complex and could also include weighting specific values, sequencing, etc.

With regards to validating the generated data, ALL oracles are heuristic in nature. So, in this case you validate the generated data satisfies the model. It is not bullet proof but it is better than gross random, or guessing.

Finally, don't forget to use real-like data also.

If you have more questions or I can help, let me know.

share|improve this answer
add comment

Test can return false positives or false negatives either because of incorrect oracle (and comparators) or incorrect test input data (aka test steps). To address those problems I found the following approaches in the literature.

Incorrect oracle and comparators

Binder in his book "Testing Object-Oriented Systems" suggests the following to limit the problem of designing incorrect oracle:

  • If possible, review some expected results produced by your oracle and your assumptions with system users. Values that may look correct to a developer or tester may be subtly incorrect due to some special constraints.
  • The more complex the oracle, the greater the chance of spurious test results. Try for the simplest solution.
  • If the oracle is specification-based, do not forget to verify the specification. Scrutinize the specification for omissions, contradiction.
  • Try the oracle for test cases that have obvious expected results – for example, all zeros in, all zeros out. Such test cases check the oracle and comparator.
  • If practical and feasible, try using several independent sources. For example, if you are picking values from a table in a reference work, try to find several other reference works that provide the same information. Interleave values from these sources. If you are usin an existing system as the oracle, try running the system in different configurations or platforms, varying the time of day, altering the background load, and so on
  • Although writing a program to generate millions of test case inputs is usually not difficult, producing their expected results is often equivalent to developing the SUT. Two of the oracle patterns can partially overcome this limitation. Built-in Test Oracle will detect some, but not all, incorrect output from any input. You may be able to use an existing system as gold standard oracle. Run the existing system with your test inputs. It will automatically produce some or all of your expected results (Trusted System Oracle pattern).
  • Design-for-testability tip: Consider abandoning a test strategy or test case if it requires a very difficult or costly oracle. Try to use existing code, files, or test suites as much as possible.
  • Design-for-testability tip: Consider reworking an application specification or requirement if its oracle would be very difficult or costly to develop.
  • Consider a partial or approximating oracle. Don’t assume that your oracle must generate complete expected results for every possible input and state. Concentrate on generating outputs that must be correct or that are difficult and/or time-consuming to check by hand.
  • Consider using several kinds of oracles to offset weakness. For example, you can use an existing system to generate about half of the critical outputs for a new system. You could implement built-in test assertions to check relationships on the newer output.

Incorrect test input data

One reason why test data are invalid can be buggy generators for test input data. Therefore, Lessons Learned in Software Testing suggest to unit test and review them as in any other piece of software.

It may seem that testing the testing framework will lead to testing ad infinitum. In practice, unit tests by definition must be simpler than the tested system, so they will not need to be tested. Second, instead of verifying new test data every time you write it, you will verify a generator only once. Finally, with tests for generator you're also more confident that fixing a generator will not introduce any new bugs.

share|improve this answer
    
this is a recursive answer and not very helpful... –  Rsf Oct 29 '12 at 7:58
    
I updated my answer to show it is not recursive process, while still relevant for the invalid test data problem. –  dzieciou Oct 29 '12 at 19:36
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.