I'm writing a script that parses many thousands of lines from a server log and collates the data. I'm wondering how to go about creating unit-tests for such a program, to make sure it gathers the data correctly.
Should I just create a very small dataset and collate it manually and make sure that the unit test handles this small amount of data correctly? Is there some other way of determining that the calculations I did are correct? Ideally I would like to write the test based on a file that approximates what my script would be actually processing, rather a much much smaller sample size.