Now and then I run a test of all combinations of a test of variables. The outcome of each test is binary (e.g. pass/fail). Typically, some combinations will pass and others will fail. After the test is over, I want to look for patterns in the results.

An obvious way to look for results is to visualize them. For example, I might load the results into a spreadsheet and experiment with sorting various combinations of columns. After sorting, if the outcome column contains a long sequence of the same value, I may have found a pattern.

An alternative would be to let the computer look for patterns. I envision a command line program that takes a matrix of combinations and corresponding outcomes as input and produces a set of rules as output, e.g. "always passes when variable-1 == 3", or "always fails when variable-2 == 4 and variable-3 == 'b'".

I think I know how I would go about writing something like that, but surely this has been done before. Is there any open-source software available for summarizing combinatorial test results?

  • open-source or free ? It's not the same. Do you need the source code of the tool?
    – Rsf
    Nov 27, 2013 at 7:46
  • My first preference is open-source, but I would consider a free tool as well.
    – user246
    Nov 27, 2013 at 11:09
  • There are simple simple data mining techniques like association rule learning that will look for association rules like (1,1,0) => failure, based on the statistical data you have from your tests.
    – dzieciou
    Jan 26, 2014 at 22:11

1 Answer 1


Espresso is an algorithm for converting a truth table to a Boolean expression consisted of ANDed expressions that ORed together. For example, a truth table like this:

v1 v2 v3 result
-- -- -- ------
 0  0  0  0
 0  0  1  0
 0  1  0  1
 0  1  1  1
 1  0  0  1
 1  0  1  1
 1  1  0  1
 1  1  1  1

The Boolean expression would be:

v1 OR v2

For tests with Boolean-valued variables, it should be clear how to use Espresso to summarize combinatorial test results.

When the test variables are categorical but not Boolean, you can transform the problem into the Boolean problem as follows. For each categorical variable, create a Boolean variable for each possible value. For example, if you have a variable for operating system with three possible values, e.g. Windows, Linux, and MacOS, you create three corresponding Boolean variables, only one of which will be set to True for each test case. After making this transformation, you can use the Espresso algorithm to summarize the results.

You can find source code for Espresso here.

I don't have a good solution for real-valued variables, but Decision Trees seem like a possibility.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.