Can a proportionately scaled down testing environment find performance load issues?

Question

At my company, we are working on ways to improve our testing of database changes, in our testing environment first.

Since there are no resources to have a production like volume of data in servers with the same RAM, disk and other resources, I just thought we could scale it down and still have a good way to test those changes.

When I mention scaling, this would mean:

• using 100 times less data

• in a computer with 100 times less RAM and disk

• set the database instance with all the thresholds 100 times smaller

Does this actually catch, before production, many of the "surprise" problems we might anticipate? Or is there a more fundamental flaw in the approach that will cause deleterious changes to pass testing and affect production?

Note 1: I am not presenting this as a way to avoid specific tests for the specific change. This is just a way to catch problems you have not anticipated.

Note 2: I am not concerned with getting a few failures in testing that would pass production. But I want to avoid the reverse as much as possible.

Answer 1

Does this actually catch, before production, many of the "surprise" problems we might anticipate? Or is there a more fundamental flaw in the approach that will cause deleterious changes to pass testing and affect production?

You are wise to have a test system that you can use for catching performance issues, but your "scaling" approach is flawed.

Performance doesn't scale this way.

In all systems there are bottlenecks. Sometimes, under normal or much larger load, we know where these bottlenecks are (memory, disk, network, etc), but often we don't. That's why we do performance/load testing in as close to a production environment as we can get.

Scaling down the system, doesn't mean that the bottleneck will scale in a similar fashion. For example, restricting memory to 1/100th of production might mean that your system won't run at all.

If you want to find the bottlenecks that will actually occur in production - you would want to use the actual production system (or a non-scaled-down replica).

But if you want to find out if your recent changes have adversely affected the performance of your system, many folks run them in a scaled-down system before installing the changes, then run the same scenarios in the scaled-down system after installing the changes.

You won't get the complete picture, but you can get some hints as to what your changes might do in production.

(And you probably can't catch many of the "surprise" problems you might anticipate, since surprises aren't anticipated by definition.)

Answer 2

No. The items you mention scale very differently and there are far too many factors and resources that will get used.
For instance if I time requests on a local server using an application I'll find things like

1 user = 2 second response average (time per request)
10 users = 2.5 second average
100 users = 2.5 second average
1000 users = 20 second average
10000 users = 10 minute average

i.e. not linear, as @dzieciou points out. Given that, it also means that you can't just take 1% of the factors you list. Some applications will not behave differently at 1%, others may crash entirely in deadlocks.

So basically "Since there are no resources to have a production like volume" means your company does not consider useful volume testing to be worth spending the money on and may need to be convinced otherwise.

One thing you can do for now is use monitoring of the live production performance over time and that would help you monitor spikes and issues. A lot of the applications that monitor server performance have charts that display it. You'll at least know how performance handles your current peaks and you'll be able to do useful before and after software releases change comparisons to see if recent changes have affected it a lot. It's kinda late in the development cycle but better than nothing.

Answer 3

Presumably the database is just a piece of your overall system, and your goal is to determine whether database changes break things or slow the system down.

Using a downsized database is a reasonable way to check whether database changes break things.

I'm not sure it makes sense to use a downsized database for load testing. Whether this makes sense depends on whether the database is a bottleneck. If your test database is only 1% as big as your production database, the test database is liable to be much faster, and it may be difficult to discern slowdowns.

Answer 4

Test environment will NOT scale proportionally.

But what you can do is:

• make test environment as similar to PROD as you possibly can (with possibly less servers, that's what we do: have about half of the servers in PROD)
• replicate most of the complexity you can (fallback servers as in PROD)
• replicate the load from PROD in a reasonable way (we collect few major types of user submissions in PROD, and replay them in QA using selenium)
• closely monitor trends. We use Graphite to collect about hundred of metrics and monitor them closely.

Such settings gives you some chance to detect performance problems before code is deployed to PROD, even if it is far from foolproof.

Of course, you will detect only the "known unknowns", not the "unknown unknowns". But you have at least a fighting chance to detect them, and test the fix before patching it to PROD. You need the understanding from management that even with best efforts, it is inevitable that some performance problems will be detected only in PROD, and it is nobody's fault - universe is quirky this way and little can be done to fight it.

Also: law of diminishing returns apply: If your testing system is 100% mirror of your PROD, your system administration team have double the work to keep both of them running.

Answer 5

1. Ideally the best thing is to have UAT or Staging environment which would be exact replica of the production to test your changes there
2. If your application has "dead" time i.e. in the night or during weekends it should be possible to test it on production environment during that period.
3. Load testing on scaled down environments is the "last resort". You'll possibly be able to capture and fix some issues by using profiler tools, but you'll never able to tell what will break under heavy production load as the reasons might be numerous. See Performance Testing in Scaled Down Environments. Part One: The Challenges article for more details.