I have written 2 automation frameworks for our SaaS application that takes in key/value pairs as a URL query (among other things behind the scenes) and dynamically renders images.
Original framework that did not work out
The first one was in PHP using a directory to store test scripts that contained test data and used shared functions to create test cases for the regression suite. This was not scalable after a while and depending on who dipped their fingers into writing these scripts, too many things could go wrong. It was a convenient approach at the time to just be able to drop in a script and have it execute during the next scheduled run.
This framework also had a 'bot' feature which was handy in finding corner cases and memory leaks. The bot would call x number of random modifiers for the URL query which would then call the associated functions to create valid key/value pairs for the URL.
The New Java Framework
I rewrote this framework in Java recently and decided to start small and use a database for the test data URLs. The database also stores the regression run results that can be viewed and drilled into by the associated Java Web App.
I use Hibernate for talking to the database for the command line and Java Web App which has worked out nicely. I have a servlet that will take either a named query or an HQL statement via POST or GET (giving the data drilling page more freedom for QA to view stock query results or custom ones via the URL) and spit out JSON which the data drilling page turns into readable organized data.
So far this framework works out better for non scripting QA. I give them an excel spread sheet and a wiki page on how to fill it out and they just input the data and I insert it with a script into the database. I plan on making a web UI for this that will display the request render before submission so the non scripting QA can insert test themselves.
The workflow of running this suite:
- Run the Java command line app manually or via Jenkins which takes in
a config file to tell it the various options/features to run with,
locations to store dump files, and which 2 servers to use for
comparisons (no baselines for this suite).
- A call is made to the database to grab all test records and compile
them into a flat file as requests minus the server:port. Creating a
flat file also ensures we can run the exact same test again if need
- The flat file is then read line by line forming the full URL request
to each server. The response is then compared to each other for
perceptual difference or text/xml/json differences depending on what
- Each request result is written to a result file and dump files of
each request are stored. At the end of the run, the result file is
parsed, server data is collected, and inserted into the database for
the Java Web App to pick up and turn it into a nice report with
graphs for management.
Other frameworks we have written in Java
We have various other frameworks written in Java that do simply as you have it now where the test data is in the test method. I found this to be somewhat unorganized and hard to search for existing tests to see if I am duplicating a test case or not. Then there is a naming convention that has to be decided on for test method names and scaling those. Plus you can't have non scripting/programming QA insert tests and you have to build every time there is a new/modified test.
Another group has a similar Java framework using Junit for execution and XML files as the test data. Those XML files contain a lot of data and there are many XML files to cover the API they test. If I don't know what I'm looking for, I won't be able to contribute and probably end up duplicate tests. Plus if there is a change to the XML schema, you would have to figure out a safe search and replace for all those XML files.