Every now and then I see a question coming by asking how to implement a data driven test framework in combination with a automated testing framework like Selenium. This gives me the feeling this is used or requested a lot in the industry.

I guess I have never build any "real" data driven test suits, unless you count build some arrays which I feed into function data driven.

My biggest worry is separating the test data from the actual automated test in separate files. Which does not help the readability of tests. Also it feels harder to maintain in a version control system.

  • Am I missing out on something, or are data driven patterns overrated?
  • What are the Pro's and Con's from building data driven tests
  • Real-life examples would be greatly appreciated

Data driven tests work well when you have a lot of test cases that can be expressed as points in a parameter space. Some advantages of data-driven approach are:

  • Test cases are easier to add or remove than when they are expressed as code
  • You duplicate less test code and consequently are less susceptible to code/paste errors in your test code
  • Someone other than a developer may be able to write the test cases

Some potential disadvantages are:

  • Often times, your test cases cannot be modeled easily as points in a parameter space. Once your model becomes too complicated, the notation for expressing the test case is just as complicated as a programming language. At that point, you are better off sticking with the programming language.
  • Introducing a new test parameter may require editing lots of test cases. You can make trade-offs between the verbosity of your data format and its malleability, e.g. a CSV representation may be less flexible than a JSON format because the latter could allow for default values.
  • If the data is stored in a separate file, you are susceptible to the same kinds of complaints made about any external configuration file (e.g. Spring XML files, translation files, or log4j configuration files), i.e. it's a little harder to view the data and the code at the same time. I think this issue is manageable.

I have used a data-driven approach to test a workflow, where each test case was a sequence of (action, expected state, expected values).

I have also used data driven-testing for a comparator test, where I compare the behavior of two different versions of a service using a set of requests. For each request, I send the request to both versions of the service and then compare the responses. If the responses differ, I may have found a bug. This is different from a functional test but can still be useful.

| improve this answer | |

The best explanation would be an example - if you have a situation of needing to test something - like a login page - where the steps of the tests don't vary but the input (and output) does then having a data driven test is a lot better then many test cases where the only differences are - in the login page example - users and passwords. This can be constructed similar ways depending on the test framework and the data source can be in code (in arrays/lists), spreadsheets, CSV or other files.

Data driven tests may be a little harder to read but the pattern I've used is have test input of the following columns - for the example above:

Enabled | TestName | User | Password | Login | Notes Y | baseline test | good-user | good-password | succeeds | if fails all wrong Y | bad password test | good-user | bad-password | fails | Y | disabled user test | disabled-user | good-password | fails | N | new test | new-user | new-password | fails | disable test until purpose clear

The first column is something that I code into the loop control so that I can skip a data input if needed. The second is for readability and for logging. the notes column is just that to add comments about that test data point.

NOTE - When testing a set of negative conditions where failure is expected, I make the first test/check a positive test to ensure that if the data was correct that the action would work. If this fails then something with the environment or test is wrong.

Use text based data sources to store data and not spreadsheets - this makes changes a lot clearer in source control systems.

I would say use data driven test but if the number of parameters or expected output becomes too large then it may start to be not worth it. With lots of inputs or outputs, you lose readability and the code performing the actions may get too complex.

Here is a little more about data driven tests: https://testautomationpatterns.org/wiki/index.php/DATA-DRIVEN_TESTING

| improve this answer | |
  • This is by far the best answer. – Fractal Jul 28 '19 at 19:42

I am going to go out on a limb here, quite likely in a minority opinion, but here I go!

I avoid using them!

Now I notice your first bullet point mentioned:
Am I missing out on something, or are data driven patterns overrated?

Yes, to they are. I frequently have junior engineers approach me breathlessly when they discover it and show me what they've built and usually I shudder.

The very ironic thing is that part of my objection is that they actually can make it too easy to have tests! Yow. Let me explain: once you have that data table, it's extremely easy to start multiplying those cases out, often a good liberal dose of copy and paste starts taking place. Frequently you will end up with dozens of cases trying various combinations. For UI tests this can certainly start to make test suites slow and long running very quickly.

They also don't focus on specific examples and give quality detailed feedback about the issue.

I much prefer the BDD 'example' style where you have a full test example that runs and specifies exactly when the conditions, inputs and boundaries are, with appropriate language to describe the condition in simple readable language.

All said, I do think they have their place. I say avoid for UI and integrated tests. Use for Unit and some Acceptance tests but make sure that they do not have heavy or slow dependencies. For example if you have unit tests that mock and stub all dependencies and each one runs in 0.01 seconds then ok, a couple dozen rows of a data table driven test is ok. Just remember that a human still has to maintain and at a very minimum frequently read those lines.

| improve this answer | |
  • What do you mean by "They also don't focus on specific examples and give quality detailed feedback about the issue."? It's up to the tester to write a framework/test case in such a way it provides enough diagnostics for troubleshooting. – dzieciou Jul 29 '19 at 15:53

I do not claim to be expert, just try to do due diligence research before diving in, and use best practices, so... :-)

IMHO start by trying to implement a pattern is wrong approach (trying to force pattern into solution). I start with "exploratory coding" and try to get the feeling what would be best approach (maybe after I have few years of experience with Selenium, I would develop better feel what works where). In few test I was able to field, data driven test was appropriate solution for few of them, and for most it was not.

One good example of a data driven test was test ( Example 1) which replays data entered by user to plan complex multi-page activity in real production. We capture all form data (widget ID and value) entered, move them to our test copy of production environment, and replay the data. Users have a lot of different customization, and we do not have good way to guess what combination makes best sense, so capturing user's input and replaying it makes a lot of sense (instead of just running exactly same combination of test inputs repeatedly).

Another ( example 2 ) was calculating complex price of a multi-leg trip. Problem was again to generate trip with meaningful data. We did not need ALL the complexity from example 1 above, so we captured some of the complexity, and designed data structure which allowed test designed to enter such trip in a concise manner (arrival of one leg is departure of next, using same vehicle and crew, departure time approximated from travel time for previous leg and some padding, etc). It helped to prepare consistent test data for trip evaluation (again using real customer's trips).

| improve this answer | |

One example: Data driven testing is useful for performance or load testing. This is where many (often hundreds or more) simulated users run a small number (often less than ten) test cases against a server. Commonly these tests cases are data driven with user-name and password, so each simulated user logs in as a different user.

| improve this answer | |

I guess I have never build any "real" data driven test suits, unless you count build some arrays which I feed into function data driven.

In my book this perfectly counts as data driven. In fact, tabular data is what most people think of when talking about data driven tests:

origin  destination  firstname  lastname
WAW     HTH          Sam        Johns
LAX     LDN          Marco      Prauss

However, data for tests do not have to be only tabular. It can be texts, images, tree-structures, etc. For instance, imagine the following flight booking reservation. It contains arrays (flights, passenger) of objects that could not directly modeled in tabular format:

   flights: [
           origin: 'WAW', 
           destination: 'HTW' 
           origin: 'HTW',
           destination: 'WAW'
   passengers: [
          firstname: 'Sam',
          lastname: 'Johns'

My biggest worry is separating the test data from the actual automated test in separate files. Which does not help the readability of tests.

Test data do not have to be stored in a separate file. For instance, many frameworks allow to combine tabular test data and tests in the same file, e.g.:

  • Spock:

     class MathSpec extends Specification {
       def "maximum of two numbers"(int a, int b, int c) {
         Math.max(a, b) == c
         a | b | c
         1 | 3 | 3
         7 | 4 | 7
         0 | 0 | 0
  • Robot Framework:

      *** Settings ***
      Test Template    Login with invalid credentials should fail
      *** Test Cases ***                USERNAME         PASSWORD
      Invalid User Name                 invalid          ${VALID PASSWORD}
      Invalid Password                  ${VALID USER}    invalid
      Invalid User Name and Password    invalid          invalid
      Empty User Name                   ${EMPTY}         ${VALID PASSWORD}
      Empty Password                    ${VALID USER}    ${EMPTY}
      Empty User Name and Password      ${EMPTY}         ${EMPTY}

For more complex data you can even create your own DSL to represent test data. For instance, if you automate tests in Java, you can use Java to represent your test data. Here's an example of booking above written in Java instead of JSON:

b = booking()
      .with(flight('WAW', 'HTW'))
      .with(flight('HTW', 'WAW'))
      .with(passenger('Sam', 'Johns'));

Obviously, there are situations when storing test data outside of a test make sense. For instance:

  • your test data are in binary format, e.g., images
  • you have a lot of test data, e.g., I recently tested two versions of the same NLP (Natural Language Processing) algorithm. I wanted to make sure both original Java and ported Python version generate same output for same word. I found a dictionary online of more than 30,000 words, ready to use. It would be nonsense copy-paste those data to the test file.

Also it feels harder to maintain in a version control system.

In what sense? If you store all test data in a binary form, it might be hard to see what they contain. For instance, if you store your test data in DB and you push DB dump to the version control system (VCS) and then you would like to update your DB with new test data it might be hard to spot in commits history what you have changed. But if store your test data in text format files (CSV, JSON, XML) or in as part of your tests (using Java, Python or any programming language), versioning should be easy. And if you store your test data in binary format, e.g., images, give each test data file a meaningful name.

Am I missing out on something, or are data driven patterns overrated?

Writing tests in data-driven style is just one of the styles. It's important to choose test style depending on your needs.

What are the Pro's and Con's from building data driven tests

I think user246 has already summarized that.

Real-life examples would be greatly appreciated

Example 1. I tested system for identifying cheating flight agents by generating fake flight bookings. I used Fitnesse for simple flight data (it had nice syntax for representing tabular data) and Java for more complex cases.

Example 2. NLP algorithm I've already mentioned.

Example 3. Anti-spam filters use lots of email samples for validation.

Example 4. Anti-viruses are tested with lots of potentially malicious code samples.

Example 5. Image classification algorithms are tested with lots of images.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.