How do you simulate years of real world conditions in a testing framework?

Question

If I am writing a piece of software that needs to be highly reliable for years at a time how can I simulate years of test data in a shorter time frame? Say hours instead of years.

I know the problem is highly generic but imagine a piece of software that listens to a network adapter, reads the inputs, does some display, and then takes basic inputs, returning them through the network adapter. At times, different data sets might come through the network interface simultaneously but just flooding the network adapter with packets would not give me the results I need from the tests.

I basically need to test for long term issues that are more subtle and arise after specific events occur rather than logical assertions.

Things I would like to test as an example;

1. Ten years of no user input, just network input and display
2. Ten years of heavy usage over both user input and network input
3. Randomly generated patterns of low to medium to bursts of high usage over years of simulated time.

Thanks.

Answer 1

I can not help you there but I can point out some common issues:

• power outages
• day light savings time changes
• leap years
• network outages

Answer 2

How much are you willing to invest ? The simplest and most common way I have used is running a system at medium to high load continuously for weeks or months.

At the high end I can imagine an over clocked system running x N faster so you can squeeze years into weeks or months.

There is also a way to partially work around the problem and use mock instead of the slower parts of the system.

Answer 3

This would sound like a generic answer, but this worked for me in past. When I had a problem like this in testing phase of currently a big app, I paid attention to little events which were happening. In my case, events were like this: 1. A new user is registered 2. User adds xyz as a friend 3. User logs out (I had about 50 of them before moving to the next step)

Then I wrote a simulation script which triggers such events in random order. Some events were dependent on another - the script took care of that.

After that, I programmed a runner which basically triggered multiple events simultaneously in the app. I would configure, say 5 events, per second - and it would start shooting them. Because it was all random, it looked like a real-world live app traveling in time machine to future. In about 3 weeks, I could simulate several years of data and cases. But to be honest, in the end, this looked good more in theory as I lost touch with goals of building this.

Answer 4

Let me try to give some inputs based on what we do to compress several years in a few days. Database: if the application under testing is connected to a database, you can take data from existing databases (or simulate them if your application is new) and automate insertion, update and deletion using directly the api of your "piece of software". Network: using performance counters properly set for your application/service, you could analize the bad or not common behaviours of the network, and then you could simulate them with automated scripts that compress those behaviours in a short period. HW Resources: limit the resources of your system (it's simple with virtual machines) or simulate long period of heavy load to check how the system responds under stressing conditions.

Answer 5

If you can get data from an existing similar system and stream it to the new system (maybe with some massaging), it could simulate long periods of use.

Once you understand the format of the input, write a program that simulates it as closely as possible, and produce copious amounts of it.

It is hard to do time based simulation in condensed time. Time has nuances on systems, waves of flow, peak periods, real-world factors etc.

Answer 6

What you are looking for is often referred to as Software reliability testing Some I have found useful in the past are;

• Good details but long
• Simplified ISEB definition

I won't repeat to much from the various sources, but at its heart is defining "what constitutes reliability?" You need to define these scenarios then extrapolate from the result base you have available. This includes, Feature, Load and Regression testing.

As with anything, the further in the future you plan, more more scope you have to get it wrong, so make sure you make the stakeholders aware.