Validation testing of large datasets

Question

I often have to write and execute validation tests on rather large datasets. The data comes in different quantities (one file vs. many files) and formats - sometimes it is table-like (csv, dbf, ...) and sometimes it is tree-like (JSON, XML, ...).

The tests that I have to execute are either simple checking of each value against a list or range of valid values (e.g. temperature > -20 AND temperature < 50 or sometimes checking interdependencies between multiple records (e.g. seven records belonging to the same type must have consecutive timestamps).

My preferred language for writing and executing such tests is Python but I am willing to learn something new if that would be helpful. If possible, I would like to use one of the established test runners such as UnitTest, nose, py.test, ... in order to profit from their already built-in infrastructure (logging which test is currently running, statistics on passed/failed tests, jUnit XML output etc.).

Now comes the problem: The data sets can be large (a few million records or more) and reading them completely into memory is not always possible/reasonable. Consequently, it is not possible to write something like this:

class Validate_Data(unittest.TestCase):
    def setUp(self):
        self.data = read_all_data_into_memory()

    def validate_something(self):
        for record in self.data:
            self.assertEqual(record['some_key'], 1)

    def validate_something_else(self):
        for record in self.data:
            self.assertEqual(record['some_other_key'], 2)

However, reading the data from arbitrary file formats such as CSV, XML, protobuf, ... might take some time, so it would neither be reasonable to read the records one by one again and again inside of each testing function.

What I would rather like to do would be reading the single records from file one by one (to stay low on memory consumption), passing each record into multiple testing functions one after the other and when done, continue with the next record, but still end the process with a nice overview of which tests passed/failed.

How would this be accomplished best? An idea that came to my mind for this: In the setup method, read the whole data record by record once and store it in some binary format on disk (pickle, messagepack, pytables, hdf5, ...). Then read it again from there in each testing method which should take way less time than reading it during setup. However, I am not sure whether this is the best way to go.

I also read a few examples of data-driven-testing (e.g. with nose test generators), generating a new test case for each record read - however, I think generating millions of test cases won't be a good idea performance-wise.

Answer 1

I think you have several directions you can potentially take here, based on the information you've given, but all of them require a degree of modification to standard unit testing structures.

Some unit test frameworks allow you to identify a data source and will run a given test once per record. If these read each record one at a time, this will work for your simpler validation tests (you would need to check the documentation of your chosen framework to determine whether this would work for you).

A second option is to extend the framework to add a query-based data access layer to your preferred framework. Here you'd build in a helper routine that would connect to your data source and retrieve a set of records based on some form of query. For your example of multiple consecutive records needing timestamps to increase from the first to the last, you could query for the records, then pass the data set to a separate helper that checks for dataset[n].timestamp < dataset[n+1].timestamp

A third option which can be used if expected data follows defined rules is to take a known-good snapshot and use this as a baseline. This can be stored in whatever format works best for you (with extremely large datasets, I'd suggest a database with indexes on the fields you're most likely to query by), and then use record and/or object comparison between your baseline and your test data. This method can get horribly complex and introduce a lot of dependencies, so I wouldn't recommend it unless you have no better option.

My personal preference would be to do something like this:

• Run in parallel, not series - if at all possible, farm your automation out across multiple machines, preferably physical machines because of the I/O requirements if you're running in-house and have a limited number of virtual servers to work with. That way, each machine can run against one or a small number of data files.
• Anything that can be run as a simple data loop should be run as a simple data loop - That is, any record-by-record validations should use the unit test standard of one read per record with the record access defined as part of the test. Note: - if you have to do multiple validations per record set with this method, treat your test as the caller, and pass each validation to a separate routine that logs results, so you can clearly tell which validation failed and you don't have the routine stop on the first failure (unless you need to to stop on the first failure - for data validation I wouldn't do that, but it depends on your situation).
• Only add complexity if simple structures can't handle your tests - I can't stress this enough. I've seen what happens when automation code gets over-complex, and it makes figuring out if a problem is caused by the automation or an actual fault a nightmare.
• Transform your source data to a common format - I'd honestly make this the first thing any of my automation code did, as a utility routine that ran before the actual tests. It's easier if you know that a particular type of data will always have a particular format (e.g. temperature readings will always be in CSV files), but even without that you can build readers that will create generic data objects pulling field names and values from anything known to contain data, and then write them to whatever works for you. You can even ignore the field names and create a truly horrible data structure that stores record type, record ID, field sequence, field value. (It's horrible but it works for a data dump that can be transformed later). The reason to do this is to allow you to use a simpler data access method during your test run - if your tests are pulling the data from the same source type, they'll be easier to manage.
• For the most complex validations, use record objects and recordset objects - This will give you more control over the information you're pulling, as long as you're pulling a limited set of data each time.

I don't think the language you're using matters: all my suggestions are language-agnostic.