I often find myself wanting to generate a lot of negative test cases by mutating a known good set of inputs.

Let's say I have a function that will accept (a, b, c) as an expected valid input for a function. I have sample invalid values for each parameter: {a1, a2}, {b1}, {c1, c2, c3}. For my negative test cases, I would then want to send the mutated tuples: (a1, b, c) (a2, b, c) (a, b1, c) (a, b, c1) (a, b, c2) (a, b, c3)

Is there a name for such an operation? Are there libraries that make this simple?

2 Answers 2


The name for this type of testing is combinatorial testing or sometimes referred to as pairwise testing. There is a very effective free tool from Microsoft that we also use for combinatorial testing called PICT.

More info is also available at http://www.pairwise.org and my blog post on combinatorial testing with negative values.

Essentially the input file for you example would look like this:

param1: a, ~a1, ~a2

param2: b, ~b1

param3: c, ~c1, ~c2, ~c3

The tilde character indicates to PICT that the value is a negative value. PICT will only use 1 negative value at a time in any combination (read my blog post above for suggestions on potential error masking).

The output from PICT for this input model is:

param1 param2 param3

a b c

~a2 b c

a b ~c3

a b ~c1

~a1 b c

a b ~c2

a ~b1 c

Let me know if you have any questions.

  • Pairwise is a subset of combinatorial, theoretically you can choose triplet of values. To be fair Hexawise sells (or gives for free to 1-5 users or Non-Commercial use) a tool that I heard is as good as PICT
    – Rsf
    Aug 15, 2012 at 9:58
  • Also note that many test harnesses such as NUnit also implement Pairwise support out of the box.
    – Sam Woods
    Aug 15, 2012 at 17:19
  • Thanks BJ. Your post on error masking with negative values in pairwise testing was actually what got me interested in this operation to begin with.
    – jogo
    Aug 15, 2012 at 20:36
  • I've been calling a set of inputs for pairwise testing a "pairwise crossproduct" (or an "n-wise crossproduct with n=2"). The set of inputs each containing a single invalid parameter value seems mathematically different enough to deserve its own name, especially since the verification steps for positive and negative test cases are often different. I've got "pairwise crossproduct" for the positive input operation: what's a good name for the negative?
    – jogo
    Aug 15, 2012 at 20:41


Nice question. I've upvoted your question because it's a good one as well as Bj's answer because (a) he's 100% correct and (b) his blog post on combinatorial testing with negative values is very good too.

Here's my 2 cents. I simply call them negative pairwise tests. Technically, maybe I should add "... with a mitigation strategy to address an error masking risk."

FWIW, here's what those tests look like in my company's Hexawise test design tool. I've taken the liberty of rewriting Parameter A to be a type of Card Used in a transaction, Parameter B to be an Expiration Date of the card, and Parameter C to be a Denomination of Payment. Each Parameter has one valid Value and the same number of invalid Parameters in your example.

First, you'd enter your Values (and indicate any constraints):

enter image description here

Next, (optionally) you'd add a few simple business rules to calculate Expected Results:

Expected results for negative pairwise tests in Hexawise

And then you'd calculate the results which have:

  • A maximum of one negative Value per test, and
  • Expected Results included

negative pairwise testing with error masking and expected results in Hexawise

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.