I often have to write and execute validation tests on rather large datasets. The data comes in different quantities (one file vs. many files) and formats - sometimes it is table-like (csv, dbf, ...) and sometimes it is tree-like (JSON, XML, ...).
The tests that I have to execute are either simple checking of each value against a list or range of valid values (e.g.
temperature > -20 AND temperature < 50 or sometimes checking interdependencies between multiple records (e.g. seven records belonging to the same type must have consecutive timestamps).
My preferred language for writing and executing such tests is Python but I am willing to learn something new if that would be helpful.
If possible, I would like to use one of the established test runners such as
py.test, ... in order to profit from their already built-in infrastructure (logging which test is currently running, statistics on passed/failed tests, jUnit XML output etc.).
Now comes the problem: The data sets can be large (a few million records or more) and reading them completely into memory is not always possible/reasonable. Consequently, it is not possible to write something like this:
class Validate_Data(unittest.TestCase): def setUp(self): self.data = read_all_data_into_memory() def validate_something(self): for record in self.data: self.assertEqual(record['some_key'], 1) def validate_something_else(self): for record in self.data: self.assertEqual(record['some_other_key'], 2)
However, reading the data from arbitrary file formats such as CSV, XML, protobuf, ... might take some time, so it would neither be reasonable to read the records one by one again and again inside of each testing function.
What I would rather like to do would be reading the single records from file one by one (to stay low on memory consumption), passing each record into multiple testing functions one after the other and when done, continue with the next record, but still end the process with a nice overview of which tests passed/failed.
How would this be accomplished best? An idea that came to my mind for this: In the setup method, read the whole data record by record once and store it in some binary format on disk (pickle, messagepack, pytables, hdf5, ...). Then read it again from there in each testing method which should take way less time than reading it during setup. However, I am not sure whether this is the best way to go.
I also read a few examples of data-driven-testing (e.g. with nose test generators), generating a new test case for each record read - however, I think generating millions of test cases won't be a good idea performance-wise.