Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

For those of you who have used pairwise test design methods (or similar combinatorial software testing methods) in an attempt to zero in on a manageable number of unusually powerful tests from among a very large number of potential tests that could be executed...

  1. Have you ever applied these test design methods and achieved poor results? If so, what poor results did you observe? What factors led to the poor results?

  2. Have you thought about applying these test design methods and concluded that a pairwise (or similar) test design approach would not work in a given situation? If so, what factors led to your conclusion that pairwise or other combinatorial test design methods could not be used to achieve desired results?

Thanks in advance for sharing your experiences.

share|improve this question
add comment

3 Answers 3

The underlying assumption for those methods is that any two parameters are independent from each other, so any two parameters that do have some kind of dependency will show poor results, i.e. your coverage will not be as good as expected. When I tried to use pairwise the first step was to find independent enough parameters, I couldn't find any that will also save run time. On top of that I got loud complaints from the developers that using combinations of parameters will make theirs life harder finding and debugging problems. I test complex embedded systems BTW.

share|improve this answer
    
Rsf, Thanks for sharing your experiences. FYI, many pairwise test design tools have constraint handling features that mitigate the first problem (e.g., when some parameters are dependent upon others). Also, about the debugging challenge issue, while a valid point, (a) in my experience, I'd rather have the problem of "How do we debug this reproduce-able defect?" if the alternative would be "Why didn't we detect this defect during testing?", and (b) you can often quickly eliminate the vast majority of possible causes by looking at passed tests that contain similar combinations of test inputs. –  Justin May 18 '11 at 19:08
add comment

Actually, the underlying heuristic for combinatorial testing of multiple input parameters is that they are interdependent and the specific values assigned affect on a common output condition or state.

Based on the concept of testing various combinations of input variables that affect a common output condition or state, combinatorial testing may not be suited for testing

  • sequentially ordered inputs
  • state transition or business flow
  • customer scenarios
  • etc

I have also discovered that conbinatorial testing may also miss single mode errors where the order in which the parameter variables are changed trigger a problem, but if the order is different the single mode failure may not be detected.

share|improve this answer
    
Upvoting this comment. For those who are interested, Dr. Atif Memon at the University of Maryland and his collaborators have published some papers about the application of combinatorial testing to user interfaces. They talk about the issue the commenter described in the third paragraph. –  user246 May 27 '11 at 12:35
    
@user246 Why don't you just give us link to the papers? And BTW, what comment you are talking about? There is no other than yours at time I am reading it... –  Alois Mahdal May 17 '12 at 8:57
    
At least one of the papers @user246 is referring to is called "A Comprehensive Framework for testing graphical user Interfaces" found here: cs.virginia.edu/~soffa/research/dissertations/atif.thesis.pdf –  Chris Kenst Aug 29 '12 at 19:58
    
Dr. Memon's publications are here: cs.umd.edu/~atif/publications.shtml –  user246 Aug 29 '12 at 20:56
add comment

The short answer is situations when a bug can occur only when three or more states must be set a certain way.

The long answer ...

Pairwise testing good technique that actually applies some pretty sound logic. That said, it is fairly easy to understand what the potential tradeoffs are .. let me illustrate using an example I have used previously.

Let’s assume that we are building an application that we use determine the cost of a flight.

We can choose from Sydney, Melbourne or Brisbane as destinations. We can choose from Business, Economy or Budget Economy as our type of ticket. Finally, we can choose Virgin Blue, Qantas or Jetstar as our airline.

Let’s assume that there is a bug which occurs when Business and Jetstar are selected together, because Jetstar don’t have any business class seats. With all 27 possible combinations, we would get three test failures, as shown below:

Destination     Class           Airline         Result
Sydney          Business        Jetstar         Error
Sydney          Business        Qantas  
Sydney          Business        Virgin Blue 
Sydney          Economy         Jetstar 
Sydney          Economy         Qantas  
Sydney          Economy         Virgin Blue 
Sydney          Budget Economy  Jetstar 
Sydney          Budget Economy  Qantas  
Sydney          Budget Economy  Virgin Blue 
Melbourne       Business        Jetstar         Error
Melbourne       Business        Qantas  
Melbourne       Business        Virgin Blue 
Melbourne       Economy         Jetstar 
Melbourne       Economy         Qantas  
Melbourne       Economy         Virgin Blue 
Melbourne       Budget Economy  Jetstar 
Melbourne       Budget Economy  Qantas  
Melbourne       Budget Economy  Virgin Blue 
Brisbane        Business        Jetstar         Error
Brisbane        Business        Qantas  
Brisbane        Business        Virgin Blue 
Brisbane        Economy         Jetstar 
Brisbane        Economy         Qantas  
Brisbane        Economy         Virgin Blue 
Brisbane        Budget Economy  Jetstar 
Brisbane        Budget Economy  Qantas  
Brisbane        Budget Economy  Virgin Blue

After we apply pairwise (I used allpairs) we reduce to the following 10 tests

Destination     Class           Airline         Result
Sydney          Business        Jetstar         Error
Sydney          Economy         Qantas  
Sydney          Budget Economy  Virgin Blue 
Melbourne       Business        Virgin Blue 
Melbourne       Economy         Jetstar 
Melbourne       Budget Economy  Qantas  
Brisbane        Business        Qantas  
Brisbane        Economy         Jetstar 
Brisbane        Economy         Virgin Blue 
Brisbane        Budget Economy  Jetstar

So we found our bug in this case, so we didn't loose anything. So what is the trade-off with using pairs?

Well if a bug only occurs when three or more things occur, such as a Qantas business class flight to Melbourne, then you are not guaranteed of finding it. You may get lucky if your test case just happens to match, but you are not guaranteed to find it, that is the tradeoff with pairwise.

You will get this kind of tradeoff with any testing approach, not just pairwise. My example shows what happens with 2 variables, when moving into combinations with more than just pairs, you can increase the coverage.

There are also some tool limits. AllParis (which I used) can only do pairs, whilst other tools will have less limitations.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.