10

At my company, we are working on ways to improve our testing of database changes, in our testing environment first.

Since there are no resources to have a production like volume of data in servers with the same RAM, disk and other resources, I just thought we could scale it down and still have a good way to test those changes.

When I mention scaling, this would mean:

  • using 100 times less data

  • in a computer with 100 times less RAM and disk

  • set the database instance with all the thresholds 100 times smaller

Does this actually catch, before production, many of the "surprise" problems we might anticipate? Or is there a more fundamental flaw in the approach that will cause deleterious changes to pass testing and affect production?

Note 1: I am not presenting this as a way to avoid specific tests for the specific change. This is just a way to catch problems you have not anticipated.

Note 2: I am not concerned with getting a few failures in testing that would pass production. But I want to avoid the reverse as much as possible.

9

Does this actually catch, before production, many of the "surprise" problems we might anticipate? Or is there a more fundamental flaw in the approach that will cause deleterious changes to pass testing and affect production?

You are wise to have a test system that you can use for catching performance issues, but your "scaling" approach is flawed.

Performance doesn't scale this way.

In all systems there are bottlenecks. Sometimes, under normal or much larger load, we know where these bottlenecks are (memory, disk, network, etc), but often we don't. That's why we do performance/load testing in as close to a production environment as we can get.

Scaling down the system, doesn't mean that the bottleneck will scale in a similar fashion. For example, restricting memory to 1/100th of production might mean that your system won't run at all.

If you want to find the bottlenecks that will actually occur in production - you would want to use the actual production system (or a non-scaled-down replica).

But if you want to find out if your recent changes have adversely affected the performance of your system, many folks run them in a scaled-down system before installing the changes, then run the same scenarios in the scaled-down system after installing the changes.

You won't get the complete picture, but you can get some hints as to what your changes might do in production.

(And you probably can't catch many of the "surprise" problems you might anticipate, since surprises aren't anticipated by definition.)

  • I am aware of the non linearity of the scaling issues.I was just wondering if there was an obvious way to "fix" this approach in a scaled down testing environment. And yes, I really love the idea of testing before and after applying changes to the testing environment. The dialogue with @MichaelDurant was leading me to the same point. – Alexandre Martins Mar 14 '16 at 18:11
10

No. The items you mention scale very differently and there are far too many factors and resources that will get used.
For instance if I time requests on a local server using an application I'll find things like

1 user = 2 second response average (time per request)
10 users = 2.5 second average
100 users = 2.5 second average
1000 users = 20 second average
10000 users = 10 minute average

i.e. not linear, as @dzieciou points out. Given that, it also means that you can't just take 1% of the factors you list. Some applications will not behave differently at 1%, others may crash entirely in deadlocks.

So basically "Since there are no resources to have a production like volume" means your company does not consider useful volume testing to be worth spending the money on and may need to be convinced otherwise.

One thing you can do for now is use monitoring of the live production performance over time and that would help you monitor spikes and issues. A lot of the applications that monitor server performance have charts that display it. You'll at least know how performance handles your current peaks and you'll be able to do useful before and after software releases change comparisons to see if recent changes have affected it a lot. It's kinda late in the development cycle but better than nothing.

  • I am not assuming there are no other factors or that the scaling is linear in any way. But the system is already in production and as been running along for many years. When I read your answer, I started wondering if we could compare the performance before after the changes. The comparison might be good enough to raise alarm bells before further investigation. I wonder if that would work? – Alexandre Martins Mar 12 '16 at 21:12
  • 1
    Yes, you could monitor the live production performance over time and that would help you monitor spikes and issues. A lot of the applications that monitor server performance have charts that display it. You'll at least know how performance handles your current peaks and you'll be able to do useful before and after change comparisons to see if recent changes have affected it a lot. It's kinda late in the development cycle but better than nothing. Added this to the answer. – Michael Durrant Mar 12 '16 at 21:27
  • Sorry. I meant to compare the changes in a scaled down testing environment before and after applying them. We already do it for production but that is already too late. – Alexandre Martins Mar 12 '16 at 21:53
4

Presumably the database is just a piece of your overall system, and your goal is to determine whether database changes break things or slow the system down.

Using a downsized database is a reasonable way to check whether database changes break things.

I'm not sure it makes sense to use a downsized database for load testing. Whether this makes sense depends on whether the database is a bottleneck. If your test database is only 1% as big as your production database, the test database is liable to be much faster, and it may be difficult to discern slowdowns.

  • 3
    I'm not an expert in performance testing, but I would add that systems rarely scale linearly, i.e., having 100 less data does not automatically mean system will behave 100 times faster. At certain thresholds different behaviours come into play, including caching, route balancing, memory swapping, etc. – dzieciou Mar 11 '16 at 19:50
  • @user246 You make good about catching errors being a goal. I guess I may have asked two questions into one. – Alexandre Martins Mar 12 '16 at 21:06
3

Test environment will NOT scale proportionally.

But what you can do is:

  • make test environment as similar to PROD as you possibly can (with possibly less servers, that's what we do: have about half of the servers in PROD)
  • replicate most of the complexity you can (fallback servers as in PROD)
  • replicate the load from PROD in a reasonable way (we collect few major types of user submissions in PROD, and replay them in QA using selenium)
  • closely monitor trends. We use Graphite to collect about hundred of metrics and monitor them closely.

Such settings gives you some chance to detect performance problems before code is deployed to PROD, even if it is far from foolproof.

Of course, you will detect only the "known unknowns", not the "unknown unknowns". But you have at least a fighting chance to detect them, and test the fix before patching it to PROD. You need the understanding from management that even with best efforts, it is inevitable that some performance problems will be detected only in PROD, and it is nobody's fault - universe is quirky this way and little can be done to fight it.

Also: law of diminishing returns apply: If your testing system is 100% mirror of your PROD, your system administration team have double the work to keep both of them running.

2
  1. Ideally the best thing is to have UAT or Staging environment which would be exact replica of the production to test your changes there
  2. If your application has "dead" time i.e. in the night or during weekends it should be possible to test it on production environment during that period.
  3. Load testing on scaled down environments is the "last resort". You'll possibly be able to capture and fix some issues by using profiler tools, but you'll never able to tell what will break under heavy production load as the reasons might be numerous. See Performance Testing in Scaled Down Environments. Part One: The Challenges article for more details.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.