What types of testing are LEAST suited to using pairwise testing (and similar test design approaches)?

Question

For those of you who have used pairwise test design methods (or similar combinatorial software testing methods) in an attempt to zero in on a manageable number of unusually powerful tests from among a very large number of potential tests that could be executed...

1. Have you ever applied these test design methods and achieved poor results? If so, what poor results did you observe? What factors led to the poor results?

2. Have you thought about applying these test design methods and concluded that a pairwise (or similar) test design approach would not work in a given situation? If so, what factors led to your conclusion that pairwise or other combinatorial test design methods could not be used to achieve desired results?

Thanks in advance for sharing your experiences.

Answer 1

Actually, the underlying heuristic for combinatorial testing of multiple input parameters is that they are interdependent and the specific values assigned affect on a common output condition or state.

Based on the concept of testing various combinations of input variables that affect a common output condition or state, combinatorial testing may not be suited for testing

• sequentially ordered inputs
• state transition or business flow
• customer scenarios
• etc

I have also discovered that conbinatorial testing may also miss single mode errors where the order in which the parameter variables are changed trigger a problem, but if the order is different the single mode failure may not be detected.

Answer 2

The underlying assumption for those methods is that any two parameters are independent from each other, so any two parameters that do have some kind of dependency will show poor results, i.e. your coverage will not be as good as expected. When I tried to use pairwise the first step was to find independent enough parameters, I couldn't find any that will also save run time. On top of that I got loud complaints from the developers that using combinations of parameters will make theirs life harder finding and debugging problems. I test complex embedded systems BTW.

Answer 3

The short answer is situations when a bug can occur only when three or more states must be set a certain way.

The long answer ...

Pairwise testing good technique that actually applies some pretty sound logic. That said, it is fairly easy to understand what the potential tradeoffs are .. let me illustrate using an example I have used previously.

Let’s assume that we are building an application that we use determine the cost of a flight.

We can choose from Sydney, Melbourne or Brisbane as destinations. We can choose from Business, Economy or Budget Economy as our type of ticket. Finally, we can choose Virgin Blue, Qantas or Jetstar as our airline.

Let’s assume that there is a bug which occurs when Business and Jetstar are selected together, because Jetstar don’t have any business class seats. With all 27 possible combinations, we would get three test failures, as shown below:

Destination     Class           Airline         Result
Sydney          Business        Jetstar         Error
Sydney          Business        Qantas  
Sydney          Business        Virgin Blue 
Sydney          Economy         Jetstar 
Sydney          Economy         Qantas  
Sydney          Economy         Virgin Blue 
Sydney          Budget Economy  Jetstar 
Sydney          Budget Economy  Qantas  
Sydney          Budget Economy  Virgin Blue 
Melbourne       Business        Jetstar         Error
Melbourne       Business        Qantas  
Melbourne       Business        Virgin Blue 
Melbourne       Economy         Jetstar 
Melbourne       Economy         Qantas  
Melbourne       Economy         Virgin Blue 
Melbourne       Budget Economy  Jetstar 
Melbourne       Budget Economy  Qantas  
Melbourne       Budget Economy  Virgin Blue 
Brisbane        Business        Jetstar         Error
Brisbane        Business        Qantas  
Brisbane        Business        Virgin Blue 
Brisbane        Economy         Jetstar 
Brisbane        Economy         Qantas  
Brisbane        Economy         Virgin Blue 
Brisbane        Budget Economy  Jetstar 
Brisbane        Budget Economy  Qantas  
Brisbane        Budget Economy  Virgin Blue

After we apply pairwise (I used allpairs) we reduce to the following 10 tests

Destination     Class           Airline         Result
Sydney          Business        Jetstar         Error
Sydney          Economy         Qantas  
Sydney          Budget Economy  Virgin Blue 
Melbourne       Business        Virgin Blue 
Melbourne       Economy         Jetstar 
Melbourne       Budget Economy  Qantas  
Brisbane        Business        Qantas  
Brisbane        Economy         Jetstar 
Brisbane        Economy         Virgin Blue 
Brisbane        Budget Economy  Jetstar

So we found our bug in this case, so we didn't loose anything. So what is the trade-off with using pairs?

Well if a bug only occurs when three or more things occur, such as a Qantas business class flight to Melbourne, then you are not guaranteed of finding it. You may get lucky if your test case just happens to match, but you are not guaranteed to find it, that is the tradeoff with pairwise.

You will get this kind of tradeoff with any testing approach, not just pairwise. My example shows what happens with 2 variables, when moving into combinations with more than just pairs, you can increase the coverage.

There are also some tool limits. AllParis (which I used) can only do pairs, whilst other tools will have less limitations.