What does "all triples" test suite means?

Question

Suppose there are 4 classifications namely: cancel,find_next,match_case,direction and 1 composition find_what with 2 classifications Attributes and cursor position.

What will be the "all triples" test suite for the above classification tree.

Answer 1

All Pairs and "All Triples" (and the difference between these two coverage levels) can be best understood via an example. I was a bit confused by the System Under Test you described so I have provided an alternative one that should provide a clear explanation.

Here is a simple System Under Test.

Before describing 3-way coverage, let's define the closely-related concept of 2-way coverage. All pairs coverage (AKA 2-way coverage or pairwise coverage) requires that every pair of inputs would need to appear together in at least one test case.

That would mean, for example, that since the Philippines is a possible destination and First is a possible class of travel, there must be at least one test including that specific pair of test inputs (and every other possible pair of test inputs). Many testers (or at least those testers that use this test design approach) find it to be an excellent way of packing a lot of variation into a manageable number of tests. But it it is important to point out that 2-way coverage does not guarantee coverage of every possible 3-way combination (e.g., the "triplet" of

• Flying from the U.S. together with
• Class = Coach together with
• Children = "more than 1"

might NOT be included in a set of 2-way tests. (But multiple triplets ARE included in all sets of 2-way tests so any particular triplet MIGHT appear in a set of 2-way tests if you get lucky. The triplet (i) Flying from India with (ii) to the U.S. with (iii) Class = Coach does appear in the first test case of the 2-way test set for example).

Here is the complete 2-way set of tests for the above example. You will notice that only 10 combinations of inputs (out of a possible 162) are required to achieve coverage of every pair:

If you had additional time to test and you wanted to achieve more thorough testing coverage, you could generate a new set of more thorough 3-way tests. Doing so would almost always generate a larger number of tests than your 2-way test set (so there is a test execution time vs. thoroughness trade-off to consider). You will notice that it takes 34 tests to achieve the 3-way coverage goal (which in this particular example turns out to be more than 3 times as many tests than were required for the less thorough 2-way solution):

In the interest of full disclosure, I am the Founder of Hexawise (the tool that generated the test sets above). I am a strong proponent of using these types of test design approaches because testers often find that these approaches to selecting test suites results in more highly varied test sets that have more variation and fewer coverage gaps than test suites that are selected manually by testers. Once a test suite starts to involve several thousand possible scenarios, for example, it becomes virtually impossible for a tester to efficiently and reliably select a set of tests by hand that would both (a) cover every single possible 3-way combination and (b) achieve that coverage level without a great deal of extra tests and unnecessary repetition.

In my experience, the increased variation, minimized amounts of repetition, and coverage efficiency characteristic of 2-way and 3-way test sets pretty consistently leads to significantly more defects being found in significantly less time (as compared to manually-selected tests). (From my experience in working with testing teams around the world at a many different companies, most testers significantly underestimate the number of coverage gaps that exist in manually-selected test suites and underestimate the amount of wasted repetition in manually-selected test suites). These typical results are not unique to our tool; they would tend to occur regardless of which tool is used to generate the 2-way or 3-way test suites.

Answer 2

All-triples or all-pairs or all-n coverage is generally referred to as combination testing.

The point of all-triples coverage is that you create tests that set a value for every variable and ensure that every possible triplet of values appears in at least one test. To get help with this topic I'd definitely recommend reading The Domain Testing Workbook, chapter N, by Cem Kaner.

It's not entirely clear what variables you have or what the values of those variables are but you can either manually create a table / matrix or use a tool like the NIST's Automated Combinatorial Testing For Software or the ACTS tool.

From the site I just referenced, the important thing to remember is:

1. A critical problem for multi-variable testing is called combinatorial explosion. The more variables you combine, the higher the number of possible tests. Combination testing is about picking a few tests that can teach us what we need to know about the space of combinations of these variables.
2. The variables should be independent of one another (or at least your understanding should suggest they are independent).

The tool / matrix will help you identify tests that you can choose to include in whatever suite of tests you want.