Is there a name for this type of vector operation?

Question

I often find myself wanting to generate a lot of negative test cases by mutating a known good set of inputs.

Let's say I have a function that will accept (a, b, c) as an expected valid input for a function. I have sample invalid values for each parameter: {a1, a2}, {b1}, {c1, c2, c3}. For my negative test cases, I would then want to send the mutated tuples: (a1, b, c) (a2, b, c) (a, b1, c) (a, b, c1) (a, b, c2) (a, b, c3)

Is there a name for such an operation? Are there libraries that make this simple?

Answer 1

The name for this type of testing is combinatorial testing or sometimes referred to as pairwise testing. There is a very effective free tool from Microsoft that we also use for combinatorial testing called PICT.

More info is also available at http://www.pairwise.org and my blog post on combinatorial testing with negative values.

Essentially the input file for you example would look like this:

param1: a, ~a1, ~a2

param2: b, ~b1

param3: c, ~c1, ~c2, ~c3

The tilde character indicates to PICT that the value is a negative value. PICT will only use 1 negative value at a time in any combination (read my blog post above for suggestions on potential error masking).

The output from PICT for this input model is:

param1 param2 param3

a b c

~a2 b c

a b ~c3

a b ~c1

~a1 b c

a b ~c2

a ~b1 c

Let me know if you have any questions.

Answer 2

Jogo,

Nice question. I've upvoted your question because it's a good one as well as Bj's answer because (a) he's 100% correct and (b) his blog post on combinatorial testing with negative values is very good too.

Here's my 2 cents. I simply call them negative pairwise tests. Technically, maybe I should add "... with a mitigation strategy to address an error masking risk."

FWIW, here's what those tests look like in my company's Hexawise test design tool. I've taken the liberty of rewriting Parameter A to be a type of Card Used in a transaction, Parameter B to be an Expiration Date of the card, and Parameter C to be a Denomination of Payment. Each Parameter has one valid Value and the same number of invalid Parameters in your example.

First, you'd enter your Values (and indicate any constraints):

Next, (optionally) you'd add a few simple business rules to calculate Expected Results:

And then you'd calculate the results which have:

• A maximum of one negative Value per test, and
• Expected Results included