Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Background:

MFAT = "Multiple Factors at a Time" OFAT = "One Factor at a Time"

In the field of Design of Experiments, practitioners in a large variety of fields (including software testing) have found that an approach that varies multiple factors at a time in each test case is often preferable to an OFAT approach. This is because, MFAT approaches are often much more efficient (e.g., actionable information is revealed in fewer experiments), and effective (e.g., more complete information is revealed).

I'm aware of many examples of MFAT approaches (like pairwise) being used successfully in functional software testing but have not seen many examples in performance testing. There may well be tons of examples but I'm relatively experienced with performance testing.

Question:

I'm seeking specific examples of where MFAT approaches have been successfully used in performance testing. I've found articles and blog posts where people advocate MFAT approaches be used in performance testing (in addition to OFAT approaches), but I'm asking this question because I'm seeking first-hand experience reports and/or experience-based advice for when it might be especially important to test for possible interactions between different performance testing variables.

The image shown below is from Ben Simo's blog post encouraging performance testers to use run MFAT tests in addition to running OFAT tests. Do you follow this advice when doing performance tests? If so, has it been useful? If not, why do you feel an OFAT approach is sufficient?

Image from Ben Simo's blog post

share|improve this question
add comment

1 Answer

up vote 3 down vote accepted

It depends on the goal of your testing, the system and how complex it is, how easy it is to monitor, how much information is logged and I'm sure some other factors that I am missing.

I can give you specific examples of where I have used MFAT in "performance" testing and it made a lot of sense. I put performance in quotes because it is generally used as a catch all that includes other kinds of testing like scalability and load testing whereas I like to keep it separate to include only tests that measure the performance of specific actions in the system (like a response time from a web service for example).

In extremely simple architectures, like let's say a web site with a lot of static content it is easy to test a few things at once. I can pretty easily run my load tests in a step load pattern (incrementing traffic by x every y minutes) while also running my performance tests in another process. This will tell me how fast response times are (speed) at different levels of load, as well as give me information on reliability at different levels of load, and eventually will also tell me where the peak load I can handle is (scalability) before the system becomes unreliable or so slow that it no longer meets our performance goals. This does not necessarily address availability, but it does give me some clues that can help determine if the system will be reliable. For example, I can say with fairly high confidence that if during the step-load test we see the memory usage growing exponentially that we have a memory leak (or at the least an issue with inefficient garbage collection).

In highly complex architectures it may also be useful, however for this it depends a lot more on how much you can monitor and log. For example a web service that takes data, processes it, stuffs it into storage (sql, hadoop, etc) and another service that reads from the storage, processes the data again and sends a response to a web client. Pinpointing bottlenecks in this system can be difficult. You might get better results using OFAT if there is very little logging or monitoring done, however if the system is good enough at monitoring itself and logging information you could get away with MFAT. You also need to watch out for gotchas, for example your storage may be tuned for fast reads or vice versa, for fast writes - so you need to be putting load that will both write and read simultaneously otherwise you can miss some serious issues. You may be able to pump a ton of data into the system no problem, you might have another test that reads a ton of data no problem, but when you try to do both, you can run into locks that cause severe performance problems.

I have also found that when I get pulled into performance testing there are often constraints on how long we can use the hardware and/or the available window for executing the tests. In these cases again you can get some additional coverage by using MFAT to reduce the amount of time it takes to complete your tests.

share|improve this answer
    
Sam, thanks for taking the time to write a nice, thoughtful answer! Reading lots of data and writing lots of data is a really nice, concrete example too. –  Justin Nov 22 '13 at 18:59
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.