I'm writing a script that parses many thousands of lines from a server log and collates the data. I'm wondering how to go about creating unit-tests for such a program, to make sure it gathers the data correctly.

Should I just create a very small dataset and collate it manually and make sure that the unit test handles this small amount of data correctly? Is there some other way of determining that the calculations I did are correct? Ideally I would like to write the test based on a file that approximates what my script would be actually processing, rather a much much smaller sample size.


Ideally I would like to write the test based on a file that approximates what my script would be actually processing, rather a much much smaller sample size.

If it were me, I would test functional correctness and scale independently. Fault isolation is easier that way. Your code/test/debug cycle can go more quickly too.

The thinking here is that functional issues are typically independent of scale; for example, whether you handle really long URLs or a missing IP address has nothing to do with how big the file is. On the flip side, the kind of errors you encounter at scale (e.g. running out of memory or slowing to a crawl) are only likely to happen with a really big file.

  • So use an extremely small dataset first followed by a test that just ensures that it can handle a large file without breaking? Jun 21 '13 at 14:57
  • That's right. I'll expand a bit on my answer.
    – user246
    Jun 21 '13 at 15:08
  • I guess for functional issues my question is more like: am I correctly tabulating/summarizing the information? So even if it correctly processes one row of data, is it adding up the records correctly, and correctly sorting/filtering the data? Jun 21 '13 at 15:16
  • 1
    There are usually multiple dimensions to any piece of software. One way to test is to attack those dimensions independently. Boundary checking is one dimension; tabulating another; error reporting another; and scaling yet another. It's a divide and conquer approach, similar to what you might use for debugging.
    – user246
    Jun 21 '13 at 15:58

You could try processing your cut-down file in Excel which should make it a little easier to verify that your script comes up with the same figures. Not too good for large files from a 1 million line limit and speed wise.

  • I'm not sure if the business logic I need to perform can be done in Excel, honestly. If it can, I certainly don't have the Excel expertise to do it. Jun 25 '13 at 16:25
  • Jordan what kind of correlation do you need to do, Most data processing can be done in Excel (if the data is not too large and processing time is not too much of an issue).
    – Sperk
    Oct 15 '13 at 11:43
  • Part of the problem is that it relies on processing against database records. The file contains the record IDs and I need to pull in fields from the corresponding record. Oct 18 '13 at 17:34
  • These are web server log files, so just as an example I need to take a given IP address and determine whether it corresponds to a provided IP range for an organization. That's way outside of my abilities in Excel. Oct 18 '13 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.