Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I've used Phadke's design of experiments approach to select parameter values for testing (software) systems.

In a nutshell, the approach uses minimum, typical and maximum values of test (method) parameters and varies them according to an appropriately chosen Taguchi array.

Exhaustively testing a system with four parameters, each having three values means that 3^4 tests must be run. Using an L9 array, only 9 tests need to be run instead of 81 (= 3^4).

The claim is that this approach will find most of the defects that exhaustively testing will find, at a fraction of the cost.

Are there any other well-known methods for selecting test parameter values?

share|improve this question
    
I call myself senior software tester because I worked as a tester for 7 years now. I know many things about testing, but I know that I can never know enough. Questions like these make my heart warm because I know what I have to learn next and what still can be learned about testing. Awesome Question. +1. Thanks! –  Hannibal May 24 '11 at 6:52
2  
@Hannibal: Thanks! I see you've seen Justin's awesome answer, too. It's definitely worth the price of admission. ;-) –  Peter K. May 24 '11 at 12:17
add comment

2 Answers

up vote 14 down vote accepted

Peter,

Good question. And an extremely important one for software testers.

I have studied the question of how can software testing inputs be combined most efficiently and effectively pretty steadily for the last five years.

By coincidence, slightly over five years ago, I started with almost precisely the same question you're posing now. At that point, I had:

  • Googled "Design of Experiments" and "software testing" and found references to Dr. Madhav Phadke (who, by coincidence, turns out was a former student of my father).

  • Discovered that Dr. Phadke had designed RDExpert which, although it had been primarily created to help with Research & Design projects, could also be used to select small sets of powerful test sets in software teting projects, using the Orthogonal Array-based test selection criteria

  • Used RDExpert to create test sets (and compared those test sets against sets of tests that had been selected manually by software testers)

  • Gathered results by asking one tester to execute the manually selected tests and another tester to execute the the Ortogonal Array-based tests; the OA-based tests dramatically outperformed the manually-selected ones in terms of defects found per tester hour and defexts found overall.

So, in short, I had confirmed to my satisfaction that an OA-based test data combination strategy was far more effective than manually selecting combinations for the kinds of projects I was working on, but I was curious if other techniques worked better.

Here's how my thinking has progressed:

  • Pairwise is more efficient and effective than Orthogonal Arrays for software testing.

  • Orthogonal Arrays are more efficient and effective for manufacturing, and agriculture, and advertising, and many other settings.

Why is this?

  • Pairwise tests almost always require fewer tests than Orthogonal Array-based solutions.

  • The reason that Orthogonal Array-based solutions require the same coverage goal that pairwise solutions do (e.g., that every pair of inputs is tested at least once) PLUS an additional hurdle / characteristic, that there be a uniform distribution throughout the domain.

  • The "cost" of the extra tests (AKA experiments) is worth paying in many settings outside of software testing BECAUSE THE RESULTS ARE NON-BINARY in those tests. Someone seeking a desired darkness and gloss and luminosity and luster for a particular shade of green in the processing of film, for example, would benefit from with the information obtained from the added information gathered from Orthogonal Arrays.

  • In software testing, however, the added costs imposed by the the extra tests are not woth it. You're generally not seeking some ideal point in a continuum; you're looking to see what two specific pieces of data will trigger a defect when they appear in the same transaction. To identify that binary approach most efficiently and effectively, what you want is a pairwise solution (with fewer tests), not a longer list of Orthogonal Array-based tests.

Having shared my views, let me also add these points.

  • First, unlike some of my other views on combinatorial test design, my opininon on this narrow subject is not based on multiple empirical studies; it is based on (a) the reasoning I laid out above, and (b) a dozen or so conversations I've had with PhD's who specialize in the intersection of "Design of Experiments" and software test design, and (c) anecdotal evidence from using both methosds.

  • Secondly, to my knowledge,very few, if any, studies have gathered empirical data showing benefits of pairwise solutions vs. orthogonal Array-based solutions in software testing scenarios.

  • Thirdly, I strongly suspect that if you asked Dr. Phadke, he would give you his reasons for why Orthogonal Array-based solutions are appropriate (and even preferable) to pairwise test case selection methods for certain kinds of software projects. I have a huge amount of respect for both him and his son.

  • Time doesn't allow me to get into this last point much now, but "mixed strength" tests are another even more powerful test design approach for you to be aware of as well. With mixed strength testing solutions, the test designer is able to select a default coverage strength for the entire plan (e.g., pairwise / AKA 2-way coverage) AND, in the same set of tests, select certain high priority values to receive higher coverage strength (e.g., 4-way coverage strength selected for each "Credit Rating" and "Income" and "Loan Amount" and "Loan to Value Ratio" would give you a palm that achieved pairwise coverage for everything in the plan PLUS comprehensive coverage for every imaginable combination of values from those four high priority parameters. This approach allows you to focus on risk-based testing considerations.

I hope this answer helps. Sorry if I got a bit long-winded. It's a topic I'm passionate about.


Additional note added after the first 3 comments were submitted:

@Hannibal, @Peter K., and @MichaelF, Thanks for your comments! If you'd like to read more about this stuff, I recommend the multiple links available through this "bundle of links" about pairwise testing and combinatorial testing. In particular, Michael Bolton's article on pairwise testing is directly relevant and very clearly written. It is one of the few introductory articles around that accurately describes the difference between Orthogonal Array-based solutions and pairwise solutions. If I remember correctly though, the example Michael uses is a rare exception to the rule; the OA solution has the same number of tests as an optimal pairwise solution does.

share|improve this answer
1  
Hollly Hell man. Nice answer. I would give you two points if I could. Respect. –  Hannibal May 24 '11 at 6:50
1  
Great answer, Justin! Thanks. I'll give it another few hours, but you're going to get the tick! –  Peter K. May 24 '11 at 12:16
    
Nice! I'm going to have to read up on these, some good stuff here! –  MichaelF May 24 '11 at 13:02
add comment

All-Pairs is another well-known method for choosing test vectors. Roughly speaking, the test matrix is guaranteed to include each pair of parameter values at least once. Some All-Pairs generators can deal with constraints, e.g. that one parameter depends on another (e.g. need to test Internet Explorer on Windows but not on Linux) or that certain parameter values are mutually exclusive.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.