Should environment for performance tests be isolated?

Question

I've been tasked with arranging performance tests for a Web application to measure response time and memory consumption in different load scenarios.

However, our current test environment is shared across many test teams. This is acceptable for functional tests but I guess for performance tests I may get results that will be false. E.g., spikes in response time caused by other people interfering.

Creating a separate end-to-end test environment seems an expensive option now for many reasons, so I wonder whether such environment can give any, even rough picture of environment performance? What kind of tests or performance measure could be investigated with environment that is not isolated?

Answer 1

Ideally, you will need to have an isolated test environment for several reasons:

• limit the performance related variables to minimum e.g., network latency, IO
  • This allows you to identify the bottleneck easily.
  • as mentioned by Rsf, you can repeat the run and averaging how the value. The cost of analysing the data might be more expensive than the setup cost itself. Another thing you have to consider before averaging the data is to run them in different time e.g., 1-2 hours apart to sample the data in slightly different settings. Running same test back to back might not be ideal.
• for debugging purposes.
  • if you need the developer to look at the problem and change the product's code, you cannot use something that is shared among all the testers.

Here is a list of things you can do:

• run the tests in quiet time (when all the functional testers are not using it).
• engineer performance counters or logs which you can inspect the server's response and request in more granular level e.g., how long it takes to process data/file and before sending data out etc.

Answer 2

Taking user246 answer one step forward- it is usually best to run performance tests multiple times and average the results, taking a look at the standard deviation.

Most of us test complex systems where performance is not 100% repeatable and is affected by many semi-random variables and processes.

The trick part is deciding on the magic number of how many repetitions and the duration of each test. This should be decided using statistical methods, and sometimes should be left dynamic and calculated on the fly- you run tests and wait until it has converged enough for your needs (look for Statistical significance in wikipedia).

Answer 3

If others are using the system at the same time, you may want to run multiple performance tests are different times of day to average out the spikes you mentioned.

Your performance test may interfere with functional testing.

Most applications have configuration settings. You probably want to know what performance-sensitive settings are in effect during your performance test, which means you do not want the functional tests to change those settings.