Is there open-source software available for summarizing combinatorial test results?

Question

Now and then I run a test of all combinations of a test of variables. The outcome of each test is binary (e.g. pass/fail). Typically, some combinations will pass and others will fail. After the test is over, I want to look for patterns in the results.

An obvious way to look for results is to visualize them. For example, I might load the results into a spreadsheet and experiment with sorting various combinations of columns. After sorting, if the outcome column contains a long sequence of the same value, I may have found a pattern.

An alternative would be to let the computer look for patterns. I envision a command line program that takes a matrix of combinations and corresponding outcomes as input and produces a set of rules as output, e.g. "always passes when variable-1 == 3", or "always fails when variable-2 == 4 and variable-3 == 'b'".

I think I know how I would go about writing something like that, but surely this has been done before. Is there any open-source software available for summarizing combinatorial test results?

Answer 1

Espresso is an algorithm for converting a truth table to a Boolean expression consisted of ANDed expressions that ORed together. For example, a truth table like this:

v1 v2 v3 result
-- -- -- ------
 0  0  0  0
 0  0  1  0
 0  1  0  1
 0  1  1  1
 1  0  0  1
 1  0  1  1
 1  1  0  1
 1  1  1  1

The Boolean expression would be:

v1 OR v2

For tests with Boolean-valued variables, it should be clear how to use Espresso to summarize combinatorial test results.

When the test variables are categorical but not Boolean, you can transform the problem into the Boolean problem as follows. For each categorical variable, create a Boolean variable for each possible value. For example, if you have a variable for operating system with three possible values, e.g. Windows, Linux, and MacOS, you create three corresponding Boolean variables, only one of which will be set to True for each test case. After making this transformation, you can use the Espresso algorithm to summarize the results.

You can find source code for Espresso here.

I don't have a good solution for real-valued variables, but Decision Trees seem like a possibility.