In what cases are diffing result files better than using regular expressions when verifying test automated output

Question

I'm trying to write a script/framework to automate testing of our program. I need to figure a way to verify/validate the results, either using regular expressions or diffing the results against a known/good/expected results file.

I was warned if we go the route of diffing files, if someone decides to change the format/output of the program, then we'd have to regenerate all the files to diff against.

However, we could also argue that the same thing would happen with the regular expression route too, that if the output is changed that drastically, we'd need to edit and test the regular expressions too.

So, with that said, that comes back to my original question; which method is better for verifying output from automated testing? If there's another method, I'm all ears. =)

Answer 1

IMHO investing in a transition to a standardized results storage in a database will pay off big time in the long run:

• changes to script output would only require matching changes in the method of normalizing the results before storing them in the DB
• validating results would become simple comparisons between standard DB field values
• easy to handle various structure changes in complex testing environments

Answer 2

I'd go with a database or file system equivalent (such as CSV) for a number of reasons:

• SQL querying is a lot easier to learn, use, and maintain than regex
• Updating your baselines for changes is a matter of a simple update statement or file copy
• You can control what appears in your baselines and output if you pipe all your output to file/database (or if the application uses a database you just query that to check your results)
• You don't need anything more than a text editor to read or write to CSV files. That reduces licensing costs.
• Your baseline data and output data live outside your test code so if you use a compiled language you don't need to rebuild your test code for any baseline changes
• Adding a column to a CSV file or a database is trivial. Rewriting a regex is not (unless you're a guru in the things).