Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Now and then I run a test of all combinations of a test of variables. The outcome of each test is binary (e.g. pass/fail). Typically, some combinations will pass and others will fail. After the test is over, I want to look for patterns in the results.

An obvious way to look for results is to visualize them. For example, I might load the results into a spreadsheet and experiment with sorting various combinations of columns. After sorting, if the outcome column contains a long sequence of the same value, I may have found a pattern.

An alternative would be to let the computer look for patterns. I envision a command line program that takes a matrix of combinations and corresponding outcomes as input and produces a set of rules as output, e.g. "always passes when variable-1 == 3", or "always fails when variable-2 == 4 and variable-3 == 'b'".

I think I know how I would go about writing something like that, but surely this has been done before. Is there any open-source software available for summarizing combinatorial test results?

share|improve this question
open-source or free ? It's not the same. Do you need the source code of the tool? – Rsf Nov 27 '13 at 7:46
My first preference is open-source, but I would consider a free tool as well. – user246 Nov 27 '13 at 11:09
There are simple simple data mining techniques like association rule learning that will look for association rules like (1,1,0) => failure, based on the statistical data you have from your tests. – dzieciou Jan 26 '14 at 22:11
up vote 2 down vote accepted

Espresso is an algorithm for converting a truth table to a Boolean expression consisted of ANDed expressions that ORed together. For example, a truth table like this:

v1 v2 v3 result
-- -- -- ------
 0  0  0  0
 0  0  1  0
 0  1  0  1
 0  1  1  1
 1  0  0  1
 1  0  1  1
 1  1  0  1
 1  1  1  1

The Boolean expression would be:

v1 OR v2

For tests with Boolean-valued variables, it should be clear how to use Espresso to summarize combinatorial test results.

When the test variables are categorical but not Boolean, you can transform the problem into the Boolean problem as follows. For each categorical variable, create a Boolean variable for each possible value. For example, if you have a variable for operating system with three possible values, e.g. Windows, Linux, and MacOS, you create three corresponding Boolean variables, only one of which will be set to True for each test case. After making this transformation, you can use the Espresso algorithm to summarize the results.

You can find source code for Espresso here.

I don't have a good solution for real-valued variables, but Decision Trees seem like a possibility.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.