How to verify software when no pre-verified input and the corresponding output are given?

Question

Let's say I have a piece of software that takes X as input (information about a vehicle), processes it and then returns Y as output (valid transfers at the next stop). The way X is processed is known (including source code available).

[vehicle data] -> [PROCESSING (fusion, filtering)] -> [transfer data]

The processing part involves retrieving ever changing online data (transfers at the given stop). No verified input and the corresponding output is provided. The testing is not about what load the server can take care of but rather if the software that was developed doesn't output garbage.

The input is fused and filtered (individual filters are known in detail; for example excluding a transfer as a valid one due to departure time being before the arrival time of own vehicle) with data from a remote server and a specific output is expected. Due to the dynamic nature of the remote server's content and the inability to access and alter the data it sends (in order to cover various test cases including deliberately inserting content that will try and break the software) I'm currently looking into mimicking the server (so that the data can be changed at will). However the problem remains - how am I supposed to say "This server gives you Y if you input X" and be sure that the mapping between input and output is correct?

Answer 1

The correct mapping from inputs to outputs is a matter of policy. Somebody somewhere has a policy that defines what correct means. Either that, or you've found scenarios for which policy has not been defined.

Find the policy. Find the policymakers.

If the policy has any ambiguity (hint: it does!), test your understanding of it. Construct examples that express your understanding of the policy, and ask the policymakers to verify your understanding.

If there are situations where you can't make a reasonable guess about the right answer, give a clear example of the inputs and ask the policymakers what the correct answer is.

Also, you can run tests even if you don't know the right answer. Even if you can't assess the output yourself, the output is information that you can present to policymakers for assessment and clarification.

Answer 2

This is the kind of situation where a static version of the remote server with known good and known problematic data is essential.

Your version can be a server that mimics the real one, or it can be a set of mock objects with rules about what they should return for a given call, depending on just how complicated the data you're working with happens to be.

While you can evaluate what you receive in live data, you absolutely need a known canonical source of transfer data to work with so you can verify your application's handling of the data with confidence.

I'd suggest that your first priority should be to get a copy of real data that's been sanitized to remove/change anything that could identify a real user or real vehicle. That lives in your environment and gets modified and patched to remain as faithful a reflection as possible of the type of data held by the real environment.

If you can't do that, you will need to construct a mockup to match the real environment. The risk you run with that is tailoring your mock environment to your scenarios and missing important variations in data.

Either way, you'd want to do your primary validation against your known data source, and sanity checks against the real source to ensure that everything continues to flow correctly.

Answer 3

When you take X as input process it and then returns Y as output.

Can you try asking business analyst regarding the business rule regarding how X is "processed"? Or if it is possible, can you try asking developers about how this processing algorithm looks like?

The input is fused and filtered with data from a remote server and a specific output is expected.

Looks like you will need to verify with developers or business analysts regarding this filtering rules (algorithm) as well.

in order to cover various test cases including deliberately inserting content that will try and break the software.

You are talking about some kind of boundary / stress testing, again you will need to find out where the boundaries are somehow.

I admit that sometimes, you can not find all the information you need in a 100% transparent fashion; if that is the case, you will need to use common senses or a rule of thumb or experimenting. For example, the developers may not even know how many concurrent users it takes to crash a server, then you will need to:

1. Build an isolated testing environment
2. Simulate a large number of concurrent users on this isolated testing environment, say 10k, if this does not kill the server, increase it to 15k and so on until the server breaks down, then you will know you have managed to find a breaking point via experiment.

Answer 4

There are often a multitude of values that can be used in any application that has several inputs.

For testing pick some example scenarios that cover the business cases you consider to be different and worth testing.
For example:

You could test for $ amounts of 1,2,3,4,5,8,9,10,11,40,50,99,100,101, 200,999,1000,5000,1000000 but in reality if the end results depends on:

<= 100
> 100 but <= 1000
> 1000

Then you only need (boundary) cases such as:

1,99,100,101,999,1000,1001

Another example:

I recently worked at a company that served thousands of zip codes.
However they only tested one - CA 90210 !
Why - because all they needed to test was:

blank (invalid - required)
00000 (invalid zip code)
90210 (valid 5 digits)
99999 (invalid zip code)
9999 (invalid length too short)
999999 invalid length (too long)
ABCDE (invalid format - non-digits)

If NO internal information is known I typically do exploratory manual testing poking at boundaries that usually exist for that datatype until I find the different results and then I automate based on that boundary knowledge.