Peter,
Good question. And an extremely important one for software testers.
I have studied the question of how can software testing inputs be combined most efficiently and effectively pretty steadily for the last five years.
By coincidence, slightly over five years ago, I started with almost precisely the same question you're posing now. At that point, I had:
Googled "Design of Experiments" and "software testing" and found references to Dr. Madhav Phadke (who, by coincidence, turns out was a former student of my father).
Discovered that Dr. Phadke had designed RDExpert which, although it had been primarily created to help with Research & Design projects, could also be used to select small sets of powerful test sets in software testing projects, using the Orthogonal Array-based test selection criteria
Used RDExpert to create test sets (and compared those test sets against sets of tests that had been selected manually by software testers)
Gathered results by asking one tester to execute the manually selected tests and another tester to execute the Orthogonal Array-based tests; the OA-based tests dramatically outperformed the manually-selected ones in terms of defects found per tester hour and defects found overall.
So, in short, I had confirmed to my satisfaction that an OA-based test data combination strategy was far more effective than manually selecting combinations for the kinds of projects I was working on, but I was curious if other techniques worked better.
Here's how my thinking has progressed:
Pairwise is more efficient and effective than Orthogonal Arrays for software testing.
Orthogonal Arrays are more efficient and effective for manufacturing, and agriculture, and advertising, and many other settings.
Why is this?
Pairwise tests almost always require fewer tests than Orthogonal Array-based solutions.
The reason that Orthogonal Array-based solutions require the same coverage goal that pairwise solutions do (e.g., that every pair of inputs is tested at least once) PLUS an additional hurdle / characteristic, that there be a uniform distribution throughout the domain.
The "cost" of the extra tests (AKA experiments) is worth paying in many settings outside of software testing BECAUSE THE RESULTS ARE NON-BINARY in those tests. Someone seeking a desired darkness and gloss and luminosity and luster for a particular shade of green in the processing of film, for example, would benefit from with the information obtained from the added information gathered from Orthogonal Arrays.
In software testing, however, the added costs imposed by the the extra tests are not worth it. You're generally not seeking some ideal point in a continuum; you're looking to see what two specific pieces of data will trigger a defect when they appear in the same transaction. To identify that binary approach most efficiently and effectively, what you want is a pairwise solution (with fewer tests), not a longer list of Orthogonal Array-based tests.
Having shared my views, let me also add these points.
First, unlike some of my other views on combinatorial test design, my opinion on this narrow subject is not based on multiple empirical studies; it is based on (a) the reasoning I laid out above, and (b) a dozen or so conversations I've had with PhD's who specialize in the intersection of "Design of Experiments" and software test design, and (c) anecdotal evidence from using both methods.
Secondly, to my knowledge, very few, if any, studies have gathered empirical data showing benefits of pairwise solutions vs. orthogonal Array-based solutions in software testing scenarios.
Thirdly, I strongly suspect that if you asked Dr. Phadke, he would give you his reasons for why Orthogonal Array-based solutions are appropriate (and even preferable) to pairwise test case selection methods for certain kinds of software projects. I have a huge amount of respect for both him and his son.
Time doesn't allow me to get into this last point much now, but "mixed strength" tests are another even more powerful test design approach for you to be aware of as well. With mixed strength testing solutions, the test designer is able to select a default coverage strength for the entire plan (e.g., pairwise / AKA 2-way coverage) AND, in the same set of tests, select certain high priority values to receive higher coverage strength (e.g., 4-way coverage strength selected for each "Credit Rating" and "Income" and "Loan Amount" and "Loan to Value Ratio" would give you a palm that achieved pairwise coverage for everything in the plan PLUS comprehensive coverage for every imaginable combination of values from those four high priority parameters. This approach allows you to focus on risk-based testing considerations.
I hope this answer helps. Sorry if I got a bit long-winded. It's a topic I'm passionate about.
Additional note added after the first 3 comments were submitted:
@Hannibal, @Peter K., and @MichaelF, Thanks for your comments! If you'd like to read more about this stuff, I recommend the multiple links available through this "bundle of links" about pairwise testing and combinatorial testing. In particular, Michael Bolton's article on pairwise testing is directly relevant and very clearly written. It is one of the few introductory articles around that accurately describes the difference between Orthogonal Array-based solutions and pairwise solutions. If I remember correctly though, the example Michael uses is a rare exception to the rule; the OA solution has the same number of tests as an optimal pairwise solution does.
