Test Tool for our environment

Question

I need support to take ideas for a new project for my company.

We have a Tool that is composed from a lot of modules that are written in Java, bash scripts and other languages.

The tool takes a file as input and produces a lot of file as output.

The input file is compiled from customers, and is composed with more than 1000 parameters, that can also not be there.
Normally we receive until 1000 input files at day, and the most of them are processed automatically. A minimum of these files have problems and are processed manually with the tool.

Every 3 months there are changes that we have to introduce in the tool and there are not only one responsible form modules, but we are in 15 people that work on it, with different Java, script levels of knowledge.

At this moment we want to have a tool that help us to see, if we made changes, that the older input files also work (like as Regression test).

The people that will develop this system are developers (mostly Java, with good knowledge of Javascript and Spring).
The people that will configure the scenarios are not developer, but technicians.

Now the questions are:
1. it is better to start over the tool, or better to start over single modules?
2. how to manage the complexity of input file with a lot of parameters and the same at output with file with a lot of parameters to check?

Ideas are:
for 1. make a great effort over the tool, can during a lot of time, with not feedback for a long time (we don't know where we are going), but we have a "Integration" check
for 1. start over a single module give us a faster feedback, but we lose the "Integration" check
for 2. we have to configure every possibility as input, and check all outputs at level of tool and module -> great scenario configuration at beginning, great maintenance of new scenarios, high time traversal of test for all scenarios, high quality
for 2. configure the main scenarios -> medium scenario configuration at beginning, medium maintenance of new scenarios, medium time traversal of test for all scenarios, medium quality

I think there are other options now. But I don't know which or I don't know what or where read to find ideas or solutions about it.

Txs

Answer 1

From your description your areas for failure are in the interactions between the modules as opposed to put in x file get y file out. This means tools that mimic user behavior may not be the most applicable place to improve your test coverage.

As such, I would recommend focusing on Unit testing coverage with a framework such as TestNG or JUnit. There are many tool to help you understand the coverage of the unit tests, the one we happen to use is Sonar

The unit tests you write can be incorporated into a CI process (Jenkins or similar) which will act as a regression suite on commit of new code.

Answer 2

You could just create a batch job that

• takes a file input
• runs your tool
• produces the output file
• compares the new output file to a saved baseline
• reports a "failure" if there are any differences

Run this batch job for as many of your saved baseline files as necessary (perhaps all 1000) until you have the desired level of confidence.