How to automate system tests for a Windows file-processing system?

Question

I'm trying to identify the best approach to begin automating the testing of an existing software system, so that we can refactor as needed with more confidence.

We have some automated unit testing, but system testing is still a very manual and tedious process. It can take a week or more to completely test the system, so it is done very infrequently.

Since it is going to take a considerable amount of time and effort to automate this testing, I want to make sure that whatever test method/framework we choose is a good fit.

Some relevant characteristics of this system are:

• Core system consists of multiple background processes running as Windows services.
• Configuration files define the mode and various options for each process.
• Inputs to the system come in the form of data files or web service requests, or both.
• Outputs from the system can be web service responses or data files, or both.

A typical system test would be:

1. Install new build on a test system.
2. Set up configuration files for specific mode and options being tested.
3. Start the processes.
4. Copy input data file(s) to configured input folder(s).
5. Wait for system to process the input and create output file(s).
6. Verify output file contains expected data.

Web service APIs are optionally used to do things such as request status or trigger the processing of input data files.

I'm currently thinking that developing a custom test framework in PowerShell is the best option, but would be very interested in any other suggestions.

Answer 1

I would start with a clean state - create a VM, install whatever is necessary for the SUT (system under test) to run (beside the SUT itself). Save the VM state and reuse it each test. If you want to test multiple Windows OS - proceed the same way with different OS versions and create more VM's.

Then prepare some shell scripts (I guess Powershell?) and proceed with steps 2-5. Once this is done verify the output. I guess the API is a separate thing, it would impact the setup as you will probably need to setup some ports/dns etc.