In performance testing, when have MFAT approaches been successfully applied (in addition to an OFAT approach)?

Question

Background:

MFAT = "Multiple Factors at a Time" OFAT = "One Factor at a Time"

In the field of Design of Experiments, practitioners in a large variety of fields (including software testing) have found that an approach that varies multiple factors at a time in each test case is often preferable to an OFAT approach. This is because, MFAT approaches are often much more efficient (e.g., actionable information is revealed in fewer experiments), and effective (e.g., more complete information is revealed).

I'm aware of many examples of MFAT approaches (like pairwise) being used successfully in functional software testing but have not seen many examples in performance testing. There may well be tons of examples but I'm relatively experienced with performance testing.

Question:

I'm seeking specific examples of where MFAT approaches have been successfully used in performance testing. I've found articles and blog posts where people advocate MFAT approaches be used in performance testing (in addition to OFAT approaches), but I'm asking this question because I'm seeking first-hand experience reports and/or experience-based advice for when it might be especially important to test for possible interactions between different performance testing variables.

The image shown below is from Ben Simo's blog post encouraging performance testers to use run MFAT tests in addition to running OFAT tests. Do you follow this advice when doing performance tests? If so, has it been useful? If not, why do you feel an OFAT approach is sufficient?

Answer 1

It depends on the goal of your testing, the system and how complex it is, how easy it is to monitor, how much information is logged and I'm sure some other factors that I am missing.

I can give you specific examples of where I have used MFAT in "performance" testing and it made a lot of sense. I put performance in quotes because it is generally used as a catch all that includes other kinds of testing like scalability and load testing whereas I like to keep it separate to include only tests that measure the performance of specific actions in the system (like a response time from a web service for example).

In extremely simple architectures, like let's say a web site with a lot of static content it is easy to test a few things at once. I can pretty easily run my load tests in a step load pattern (incrementing traffic by x every y minutes) while also running my performance tests in another process. This will tell me how fast response times are (speed) at different levels of load, as well as give me information on reliability at different levels of load, and eventually will also tell me where the peak load I can handle is (scalability) before the system becomes unreliable or so slow that it no longer meets our performance goals. This does not necessarily address availability, but it does give me some clues that can help determine if the system will be reliable. For example, I can say with fairly high confidence that if during the step-load test we see the memory usage growing exponentially that we have a memory leak (or at the least an issue with inefficient garbage collection).

In highly complex architectures it may also be useful, however for this it depends a lot more on how much you can monitor and log. For example a web service that takes data, processes it, stuffs it into storage (sql, hadoop, etc) and another service that reads from the storage, processes the data again and sends a response to a web client. Pinpointing bottlenecks in this system can be difficult. You might get better results using OFAT if there is very little logging or monitoring done, however if the system is good enough at monitoring itself and logging information you could get away with MFAT. You also need to watch out for gotchas, for example your storage may be tuned for fast reads or vice versa, for fast writes - so you need to be putting load that will both write and read simultaneously otherwise you can miss some serious issues. You may be able to pump a ton of data into the system no problem, you might have another test that reads a ton of data no problem, but when you try to do both, you can run into locks that cause severe performance problems.

I have also found that when I get pulled into performance testing there are often constraints on how long we can use the hardware and/or the available window for executing the tests. In these cases again you can get some additional coverage by using MFAT to reduce the amount of time it takes to complete your tests.