Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I would like to know if there is any acceptable ratio for test size data size compared to production data size? If my prod data is 200TB would a test data size of 20TB be adequate? Does such a consideration exist?

share|improve this question
add comment

4 Answers

There are several factors to consider when selecting test data.

For example, if your 200TB of production data is essentially equivalent, then you could potentially calculate a statistical sample from the total equivalent population. If not you could group your 200TB of production data into smaller equivalent subsets and calculate sample sizes for each subset population of data. However, there could always be outliers and if those aren't correctly identified then you could be stepping over some pretty big holes. Also, you might want to identify any failure indicators such as production data that has been historically problematic.

Ultimately, there is no magic formula. Part of the challenge as a tester is to determine the appropriate test data, and the appropriate amount of test data that will provide confidence that the feature is well tested and has a low risk of failure with production data in the wild.

share|improve this answer
add comment

As Bj Rollison says, it depends on the nature of the data.

My view is that good test data meets these criteria:

  • Representative - the test data covers the largest possible combination of configuration/settings. For instance, if your application supports retail sales in multiple countries, your test data needs to include currency sets with zero, one, two, and three decimal places.
  • Comprehensive - the test data covers every feature of your application. For example, if your application includes the ability to securely store payment information, then you need test data that has payment information and test data that doesn't have payment information.
  • Flexible - your test data needs to be flexible enough that you can easily copy/adapt it to new feature development and clean corrupted data. It sounds from your question like you have one massive data store that does everything: in this case it's not practical to flush your data store and start clean for testing, so you need to a mechanism to quickly clear corrupted data (this is always necessary for test data, since I've never known a test situation that didn't end up generating bad data) as well as a way to use a base data set to generate new scenarios.
  • Versioned - This is particularly important when supporting non-web-based items, but if at all possible it helps to know which version of your application your test data is optimized for. Given your situation, I suspect you're stuck with whatever level of auditing and versioning your database schema is configured with.
  • Isolated - By this I mean that the test data your application works with is not mixed with the data you enter during your testing (if your testing involves data entry) and that you have some means of rolling back data changes when testing is done. With a large database this becomes complex. If your testing is mostly or all data retrieval and manipulation, this is less important.
  • Known - No matter what your test data looks like, you need to know what you have there and how it's structured. That doesn't necessarily mean you need to be intimately familiar with all 200TB of your production data (or the full amount of your test data), but it does mean you need to know enough to recognize the impact of schema changes and be able to recognize when a schema change needs to be propagated to your test data set.

The short version is that there is no hard rule for how much test data is enough. If the 20TB sample you take from your 200TB production database doesn't cover a particular feature, you've got a hole that will likely become a regression magnet.

share|improve this answer
add comment

Adding to the previous responses, it also depends on the kind of testing you are trying to perform.

Performance or load testing: Figure out the scale of your test environment compared to the production e.g. If your production has 100 application servers and 10 database servers to handle your 200 TB data with given set of server configuration, you might be able to achieve your load and performance testing with 10 application servers and 1 db server with 20TB of data (these numbers also depend on the configs of your servers).

In case of functional testing, you might want to cover various use cases with as less data as possible.

share|improve this answer
add comment

No, there is no apriori acceptable ratio. Here is why: without knowing the production data's probability distribution, you cannot determine how big your representative sample needs to be.

A way to proceed is to decide on a way to measure a sample's distribution and then measure progressively bigger samples until the distribution converges.

If you can't do that experiment, you may need to just choose the largest sample you can afford to work with, and make it clear to everyone why you made that choice.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.