I guess I have never build any "real" data driven test suits, unless you count build some arrays which I feed into function data driven.
In my book this perfectly counts as data driven. In fact, tabular data is what most people think of when talking about data driven tests:
origin destination firstname lastname
------------------------------------------
WAW HTH Sam Johns
LAX LDN Marco Prauss
However, data for tests do not have to be only tabular. It can be texts, images, tree-structures, etc. For instance, imagine the following flight booking reservation. It contains arrays (flights, passenger) of objects that could not directly modeled in tabular format:
{
flights: [
{
origin: 'WAW',
destination: 'HTW'
},
{
origin: 'HTW',
destination: 'WAW'
}
]
passengers: [
{
firstname: 'Sam',
lastname: 'Johns'
}
]
}
My biggest worry is separating the test data from the actual automated test in separate files. Which does not help the readability of tests.
Test data do not have to be stored in a separate file. For instance, many frameworks allow to combine tabular test data and tests in the same file, e.g.:
Spock:
class MathSpec extends Specification {
def "maximum of two numbers"(int a, int b, int c) {
expect:
Math.max(a, b) == c
where:
a | b | c
1 | 3 | 3
7 | 4 | 7
0 | 0 | 0
}
}
Robot Framework:
*** Settings ***
Test Template Login with invalid credentials should fail
*** Test Cases *** USERNAME PASSWORD
Invalid User Name invalid ${VALID PASSWORD}
Invalid Password ${VALID USER} invalid
Invalid User Name and Password invalid invalid
Empty User Name ${EMPTY} ${VALID PASSWORD}
Empty Password ${VALID USER} ${EMPTY}
Empty User Name and Password ${EMPTY} ${EMPTY}
For more complex data you can even create your own DSL to represent test data. For instance, if you automate tests in Java, you can use Java to represent your test data. Here's an example of booking above written in Java instead of JSON:
b = booking()
.with(flight('WAW', 'HTW'))
.with(flight('HTW', 'WAW'))
.with(passenger('Sam', 'Johns'));
Obviously, there are situations when storing test data outside of a test make sense. For instance:
- your test data are in binary format, e.g., images
- you have a lot of test data, e.g., I recently tested two versions of the same NLP (Natural Language Processing) algorithm. I wanted to make sure both original Java and ported Python version generate same output for same word. I found a dictionary online of more than 30,000 words, ready to use. It would be nonsense copy-paste those data to the test file.
Also it feels harder to maintain in a version control system.
In what sense? If you store all test data in a binary form, it might be hard to see what they contain. For instance, if you store your test data in DB and you push DB dump to the version control system (VCS) and then you would like to update your DB with new test data it might be hard to spot in commits history what you have changed. But if store your test data in text format files (CSV, JSON, XML) or in as part of your tests (using Java, Python or any programming language), versioning should be easy. And if you store your test data in binary format, e.g., images, give each test data file a meaningful name.
Am I missing out on something, or are data driven patterns overrated?
Writing tests in data-driven style is just one of the styles. It's important to choose test style depending on your needs.
What are the Pro's and Con's from building data driven tests
I think user246 has already summarized that.
Real-life examples would be greatly appreciated
Example 1. I tested system for identifying cheating flight agents by generating fake flight bookings. I used Fitnesse for simple flight data (it had nice syntax for representing tabular data) and Java for more complex cases.
Example 2. NLP algorithm I've already mentioned.
Example 3. Anti-spam filters use lots of email samples for validation.
Example 4. Anti-viruses are tested with lots of potentially malicious code samples.
Example 5. Image classification algorithms are tested with lots of images.
