How should I design a testing framework for a trading system?

Question

I work in finance and my firm would really like me to automate our execution platform's testing suite. Right now the tests communicate directly with the system; they are extremely brittle and tightly coupled to changes in the SUT (System under Test). I want to abstract it in the form of a testing framework, but I'm having a hard time thinking of a good design strategy to do so.

Right now the app has this kind of behavior:

1. Listens for request from client
2. Perform few operation(s) based on some logic on that request
3. Modifies the message to be in the form of the specific exchange (vague because it is a complex process and the information added/modified isn't important)
4. Sends it off to the exchange

The above is the general workflow, but it can also do internal rejects for things that range from bad client requests, invalid exchange definitions, etc. It also publishes updates from the exchange back to the client and several other internal systems.

Given the above application description, how would you design a framework to test it?

I am looking for answers that are at the design level. The implementation details are not important at the moment as I am looking for places to get started.

Answer 1

If I understand correctly, you're working with an n-tier system that communicates via something resembling an API.

Given this assumption, I'd build a framework that allows the creation, transmission, receiving and parsing of messages to and from the application. At this point your actual test data can be in any format, and don't need to be coupled to your code. You also remain independent of any changes to the client (which can either be set up with automation testing separately if needed or tested purely manually).

This is how I'd handle the design:

• Test harness that calls the other routines. This would be your "master" that runs whatever tests you've defined. I'd data drive it, giving it a set of messages and expected outcomes for each message.
• Message sender. This is a pretty simple thing that uses the communications interface of your application to send messages and receive any responses. I'd set it up to pull the host and port from some kind of config setting so you don't have that information in with your code.
• Message parser. This module would analyze responses and broadcast messages for the specific patterns you're expecting. Since you're storing these outside your code, updates can be fairly minor.
• Message builder/loader. Depending on how sophisticated you want to be, you can build your messages from raw data (this is more flexible and allows you to do things like reuse message headers, but has an added cost in setup and maintenance time) or simply load a file containing the full message (which is simpler but can need bulk replacement if some part of the message format changes).

Your process flow would work something along these lines:

• On start, your test code looks for the tests to run. These would be stored in an external file/data store that allows you to easily modify the test list without needing to change code. (I'd use a config flag to tell the code where to look)
• The test data defines which test messages to send and what responses to look for.
• Depending on your situation, you may have an initialization routine to start the application and wait until it's ready to receive messages.
• The test harness loops through each test in turn.
• For each test, the harness calls the message builder/loader for the message to send.
  • Then the harness calls the message sender and captures the response.
  • The harness takes the response and passes it to the parser to determine the result of that test.
  • The harness reports (either using the built-in reporting of your test engine, or a custom format) the result of the test.
• Once all tests have been run, the harness would perform any cleanup needed, including shutting down the application if need be, collating report results, closing any open files and so forth.