What are some good approaches to separating test data from test scripts?

Question

We're working on test code (mainly UI tests with a few different tools). One thing that's really stumping me is how to best separate test data from test scripts. By test data, I mean values that get input into the AUT.

Currently, I have a bunch of classes that simply contain test data in methods. This almost works but it's

1. not scalable and
2. not modifiable from outside the test code.

We've discussed some approaches like keeping data in a test data database, where different data are called when needed by scripts. This is likely the way to go, but I have no idea how to implement it. What are some approaches that others have had success with? I'd like to have test scripts as agnostic to incoming data as possible.

EDIT Some more details about these tests: All test data are parameters that are passed from an external source (different classes, xml, database, etc) into the test code. These tests are written in Java.

Answer 1

See if your test framework gives you a way to parameterize tests. Many test frameworks allow you to supply the values using a "data provider" method or object or class that you write yourself. If you have a framework like that, see if you can use its data provider mechanism to supply the values.

The usual mechanism is that your code fetches the values from somewhere, and hands them to the test framework as some sort of grid-like thing (list of lists, array of arrays, iterable of arrays). Each "row" of the grid represents the values for one execution of the test.

This is fairly easy in TestNG, and possible (with a bit of fiddling) in JUnit.

If at all possible, store the values in plain text files. The reason for this is so that you can put the data into version control along with the tests. Often a CSV file will do nicely, and there are likely plenty of CSV-reader libraries for whatever language you're using to automate tests. Any kind of binary data file (Excel, a database, ...) will not play well with version control.

Answer 2

There's two major directions you can take here.

If your tests are structured such that one test covers a full user scenario (By this I mean that a single test covers something like log on as user X, navigate to ordering, select quantity A of product B, quantity C of product D, check out, pay with card details Y and check that all the amounts add up correctly and got to where they were supposed to go)

• Any kind of file-based structure will work here. You can use the data driven harness built into most test tools where you define a data provider and point to the resource. The framework will iterate through the resource once for each row.
• You might need to add extra columns or fields and add handling for empty columns/fields in your scripts, but it's still pretty lightweight and trouble free.
• CSV is particularly good for this because you can version easily with it. You can also internally manage your CSV layouts so that it's easy to tell which data belongs in which column (if you do go this way, do not use Excel to edit, because Excel will strip out leading/trailing spaces added for readability - for data you aren't going to edit often, having it easy to read in a text file makes a big difference)
• It's also relatively easy to include expected results as CSV files to compare against so that your validations aren't hard-coded. You can do this either as a separate baseline CSV, or include columns/fields for expected values in your files.
• If you can work this way, it's the easiest and simplest way to data drive your scripts.

If your scripts go through multiple parameterized routines to perform a single scenario (An example of this would be a master handler routine that passes a user name and password to a logon routine, then calls a purchase routine which in turn passes quantities and products to a select items routine, and passes payment data to a purchase routine, then returns to the master routine, which calls the amount and stored data checking routines)

• This kind of structure tends to evolve where there are a lot of test cases with similar but not quite close enough for a single script per sequence structure and data. If, for instance, some kinds of purchase will prompt for extra information, rather than repeat the entire logon/selection code - which doesn't change - each step or common sequence of steps will get broken into a parameterized routine that's called from a master controller.

• Not all test tools support this model of parameterization. It can be done with some of the big box tools (as long as you work through the code interface and treat it as purely programmed test code), but not all of them.

• CSV and other flat file structures can still work here, provided you maintain extensive documentation, because you will end up building a relational database in files (how do I know this? I've done it). That makes versioning possible, but can introduce a steep learning curve to new automators who not only have to become familiar with the multitude of routines available and the control structure, they need to learn how the data files relate to the code structures.

• More structured file formats like XML may be workable, if you don't mind a lot of repeated data (which is an absolute pain if it's necessary to modify something that's used in multiple files) and again are reasonably easy to version.

• Ultimately, the best way to manage this kind of test data is via database, preferably with an editing front end. You can do this with something like Access Forms, or you can build an intranet front end - or a small application that's bound to your database. In this case, versioning becomes much more complex, since databases do not play nicely with version control. My preference with this method is to include versioning information in the database tables, so that each table has a column something like "ValidFrom", "ValidTo" flagged with the AUT versions where the data is first valid. That allows your scripts to avoid pulling from the wrong versions, although it adds another layer of complexity.

• If you do go with a database solution for this kind of test structure, backups are essential. You will also need to replicate your main test database so that you aren't developing new feature tests against live data (and potentially messing up scheduled scripts) and build merge scripts to update. It is a lot more work - but has the advantage, with a reasonable front-end, of making it possible for non-scripters to add new test cases to your system.

• This level of test structuring becomes essential when your test code base becomes large enough that you can no longer maintain almost the same code in multiple tests, or when your AUT is sufficiently complex that any test requires multiple almost identical steps before you reach the region of the system you're actually interested in testing.

• Even with this kind of structuring to eliminate most of the repeated code, you can end up with a lot of code to maintain: at my previous employer, the automation codebase was in the order of 1/2 million LOC with about the same number of lines of CSV data maintained in multiple files and religiously versioned. This was a reasonably mature test automation effort - test automation had been in place in some form for over 10 years and had evolved from record/playback through parameterized versions of the record/playback code, several iterations of object-oriented, data-driven script code into the form it was in when I left. It's not for the faint-hearted but does become essential eventually (the AUT in this case was a suite of 7 core applications, where the main one had over 2 million LOC the last time anyone checked, several hundred configuration flags, and was mission critical to a large number of very large enterprises - we could not afford to miss any problems with transaction or tax processing).

That should give you a little more insight into choosing a data-driving strategy.

Answer 3

In addition to what Dale proposed, it is also possible to encode test data in Java.

This will satisfy all your requirements:

• Modifiable from outisde the test code. If you defined your test data in separate Java classes, you do not need to compile them together with test script classes. You can link them at runtime as Java Beans, e.g. using Spring. Obviously, you are aware that putting test data in separate files will require switching between windows, so it will be harder to capture a relation between test data and expected behaviour.

• Test scripts agnostic to test data. First, test scripts will be agnostic to the original format of test data, as they will rely on getter methods of Java beans (or their interfaces), not on how data got into Java beans. So, you may start from designing your test data in Java, and if you change your mind (and move to using database, XML, or CSV files), you can easily map those test data sources to your Java Beans (e.g., using one of the libraries I suggested under Dale's answer). Second, test scripts will be agnostic to where the data come from. It will be up to @DataProvider method or class where it will read test data from: database, external file or Java beans defined at runtime by Spring context.

• Scalable. Adding new test data scales well. For instance, you may define a package where @DataProvider scans for new test data at runtime. It also scales well for the size domain, thanks to encapsulation. If you don't want to have large class, you hide irrelevant details to a super class, helper class, etc. For instance, we write createReturnTicket() instead of

  createTicket().with(Type.Return).from("New York").to("Los Angeles") 
  

  From a test case perspective, we don't really care if the ticket is from New York to Los Angeles. Those are internal details of this method that if revealed would blur the picture. XML-demarcated test data don't scale well for the size of the domain. When we had really large XML files, it was really hard to understand what they describe, or given two large XML files, how they differ.

I know that some people are against this approach, afraid of encoding test data in Java classes, because they are business people, not programmers. Also, XML files have their advantages especially when the AUT processes XML by design. Business people are more familiar with XML standards used to describe application data. And it is fairly easy to reproduce many issues found production: you just isolate a sample of XML files from production, and copy them to our test repository.

However, with a bit of effort from both parties we were able to some of those problems, while still getting advantages of using pure-Java approach. Here is how:

1. Design classes and methods to describe your domain well, this is often called embedded DSL (Domain-Specific Language), e.g. createReturnTicket() is example of a method from our DSL.
2. When designing your DSL, hide irrelevant details via encapsulation, dedicated methods, etc.
3. Use Builder pattern with fluent interface (see also another post in this thread) and static imports. This way, you will get great support from your IDE and Java compiler to define correct test data: autocompletion, syntax-highlithing and validation.
4. Provide sample test cases and test case writing session, so people can learn how to write test cases.
5. Refactor your DSL continuously, so it becomes easy to use as you learn the domain. Changing your test data structure when you have a lot of existing test data is relatively simple with Java test data, comparing to XML/CSV or database approaches.

Answer 4

I have written 2 automation frameworks for our SaaS application that takes in key/value pairs as a URL query (among other things behind the scenes) and dynamically renders images.

Original framework that did not work out

The first one was in PHP using a directory to store test scripts that contained test data and used shared functions to create test cases for the regression suite. This was not scalable after a while and depending on who dipped their fingers into writing these scripts, too many things could go wrong. It was a convenient approach at the time to just be able to drop in a script and have it execute during the next scheduled run.

This framework also had a 'bot' feature which was handy in finding corner cases and memory leaks. The bot would call x number of random modifiers for the URL query which would then call the associated functions to create valid key/value pairs for the URL.

The New Java Framework

I rewrote this framework in Java recently and decided to start small and use a database for the test data URLs. The database also stores the regression run results that can be viewed and drilled into by the associated Java Web App.

I use Hibernate for talking to the database for the command line and Java Web App which has worked out nicely. I have a servlet that will take either a named query or an HQL statement via POST or GET (giving the data drilling page more freedom for QA to view stock query results or custom ones via the URL) and spit out JSON which the data drilling page turns into readable organized data.

So far this framework works out better for non scripting QA. I give them an excel spread sheet and a wiki page on how to fill it out and they just input the data and I insert it with a script into the database. I plan on making a web UI for this that will display the request render before submission so the non scripting QA can insert test themselves.

The workflow of running this suite:

1. Run the Java command line app manually or via Jenkins which takes in a config file to tell it the various options/features to run with, locations to store dump files, and which 2 servers to use for comparisons (no baselines for this suite).
2. A call is made to the database to grab all test records and compile them into a flat file as requests minus the server:port. Creating a flat file also ensures we can run the exact same test again if need be.
3. The flat file is then read line by line forming the full URL request to each server. The response is then compared to each other for perceptual difference or text/xml/json differences depending on what is returned.
4. Each request result is written to a result file and dump files of each request are stored. At the end of the run, the result file is parsed, server data is collected, and inserted into the database for the Java Web App to pick up and turn it into a nice report with graphs for management.

Other frameworks we have written in Java

We have various other frameworks written in Java that do simply as you have it now where the test data is in the test method. I found this to be somewhat unorganized and hard to search for existing tests to see if I am duplicating a test case or not. Then there is a naming convention that has to be decided on for test method names and scaling those. Plus you can't have non scripting/programming QA insert tests and you have to build every time there is a new/modified test.

Another group has a similar Java framework using Junit for execution and XML files as the test data. Those XML files contain a lot of data and there are many XML files to cover the API they test. If I don't know what I'm looking for, I won't be able to contribute and probably end up duplicate tests. Plus if there is a change to the XML schema, you would have to figure out a safe search and replace for all those XML files.

Answer 5

I would recommend looking at the Builder pattern to create test data within the test script.

It is similar to having the data in the script BUT it is the required data not the actual data which is hard coded. In theory, the tests can then run in any environment as it is the test which creates its own data.

I often think that having an external data source is risky, as if the input data is incorrect, then the test is incorrect. The test makes assumptions that the data provided is correct where as the builder pattern means that the test knows exactly what state the data is in because it created it.

I have only considered using Data Providers if my test is data driven (e.g. ensure a large range of values return the same response). However, most my tests seem to be scenario driven, so fail to see how Data Providers can be used. If my test wants data in a particular state for that scenario, then I want my test to create that data in the required state.