5

Every now and then I see a question coming by asking how to implement a data driven test framework in combination with a automated testing framework like Selenium. This gives me the feeling this is used or requested a lot in the industry.

I guess I have never build any "real" data driven test suits, unless you count build some arrays which I feed into function data driven.

My biggest worry is separating the test data from the actual automated test in separate files. Which does not help the readability of tests. Also it feels harder to maintain in a version control system.

  • Am I missing out on something, or are data driven patterns overrated?
  • What are the Pro's and Con's from building data driven tests
  • Real-life examples would be greatly appreciated
3

Data driven tests work well when you have a lot of test cases that can be expressed as points in a parameter space. Some advantages of data-driven approach are:

  • Test cases are easier to add or remove than when they are expressed as code
  • You duplicate less test code and consequently are less susceptible to code/paste errors in your test code
  • Someone other than a developer may be able to write the test cases

Some potential disadvantages are:

  • Often times, your test cases cannot be modeled easily as points in a parameter space. Once your model becomes too complicated, the notation for expressing the test case is just as complicated as a programming language. At that point, you are better off sticking with the programming language.
  • Introducing a new test parameter may require editing lots of test cases. You can make trade-offs between the verbosity of your data format and its malleability, e.g. a CSV representation may be less flexible than a JSON format because the latter could allow for default values.
  • If the data is stored in a separate file, you are susceptible to the same kinds of complaints made about any external configuration file (e.g. Spring XML files, translation files, or log4j configuration files), i.e. it's a little harder to view the data and the code at the same time. I think this issue is manageable.

I have used a data-driven approach to test a workflow, where each test case was a sequence of (action, expected state, expected values).

I have also used data driven-testing for a comparator test, where I compare the behavior of two different versions of a service using a set of requests. For each request, I send the request to both versions of the service and then compare the responses. If the responses differ, I may have found a bug. This is different from a functional test but can still be useful.

2

I am going to go out on a limb here, quite likely in a minority opinion, but here I go!

I avoid using them!

Now I notice your first bullet point mentioned:
Am I missing out on something, or are data driven patterns overrated?

Yes, to they are. I frequently have junior engineers approach me breathlessly when they discover it and show me what they've built and usually I shudder.

The very ironic thing is that part of my objection is that they actually can make it too easy to have tests! Yow. Let me explain: once you have that data table, it's extremely easy to start multiplying those cases out, often a good liberal dose of copy and paste starts taking place. Frequently you will end up with dozens of cases trying various combinations. For UI tests this can certainly start to make test suites slow and long running very quickly.

They also don't focus on specific examples and give quality detailed feedback about the issue.

I much prefer the BDD 'example' style where you have a full test example that runs and specifies exactly when the conditions, inputs and boundaries are, with appropriate language to describe the condition in simple readable language.

All said, I do think they have their place. I say avoid for UI and integrated tests. Use for Unit and some Acceptance tests but make sure that they do not have heavy or slow dependencies. For example if you have unit tests that mock and stub all dependencies and each one runs in 0.01 seconds then ok, a couple dozen rows of a data table driven test is ok. Just remember that a human still has to maintain and at a very minimum frequently read those lines.

1

I do not claim to be expert, just try to do due diligence research before diving in, and use best practices, so... :-)

IMHO start by trying to implement a pattern is wrong approach (trying to force pattern into solution). I start with "exploratory coding" and try to get the feeling what would be best approach (maybe after I have few years of experience with Selenium, I would develop better feel what works where). In few test I was able to field, data driven test was appropriate solution for few of them, and for most it was not.

One good example of a data driven test was test ( Example 1) which replays data entered by user to plan complex multi-page activity in real production. We capture all form data (widget ID and value) entered, move them to our test copy of production environment, and replay the data. Users have a lot of different customization, and we do not have good way to guess what combination makes best sense, so capturing user's input and replaying it makes a lot of sense (instead of just running exactly same combination of test inputs repeatedly).

Another ( example 2 ) was calculating complex price of a multi-leg trip. Problem was again to generate trip with meaningful data. We did not need ALL the complexity from example 1 above, so we captured some of the complexity, and designed data structure which allowed test designed to enter such trip in a concise manner (arrival of one leg is departure of next, using same vehicle and crew, departure time approximated from travel time for previous leg and some padding, etc). It helped to prepare consistent test data for trip evaluation (again using real customer's trips).

1

One example: Data driven testing is useful for performance or load testing. This is where many (often hundreds or more) simulated users run a small number (often less than ten) test cases against a server. Commonly these tests cases are data driven with user-name and password, so each simulated user logs in as a different user.

1

The best explanation would be example - if you have a situation of needing to test something - like a login page - where the steps of the tests don't vary but the input (and output) does then having a data driven test is a lot better then many test cases where the only differences are - in the login page example - users and passwords. This can be constructed similar ways depending on the test framework and the data source can be in code (in arrays/lists), spreadsheets, CSV or other files.

Data driven tests may be a little harder to read but the pattern I've used is have test input of the following columns - for the example above:

Enabled | TestName | User | Password | Login | Notes Y | baseline test | good-user | good-password | succeeds | if fails all wrong Y | bad password test | good-user | bad-password | fails | N | new test | bad-user | bad-password | fails | disable test until purpose clear

The first column is something that I code into the loop control so that I can skip a data input if needed. The second is for readability and for logging. the notes column is just that to add comments about that test data point.

Use text based data sources to store data and not spreadsheets - this makes changes a lot clearer in source control systems.

I would say use data driven test but if the number of parameters or expected output becomes too large then it may start to be not worth it. With lots of inputs or outputs, you lose readability and the code performing the actions may get too complex.

Here is a little more about data driven tests: https://testautomationpatterns.wikispaces.com/DATA-DRIVEN+TESTING

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.