Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I often find myself wanting to generate a lot of negative test cases by mutating a known good set of inputs.

Let's say I have a function that will accept (a, b, c) as an expected valid input for a function. I have sample invalid values for each parameter: {a1, a2}, {b1}, {c1, c2, c3}. For my negative test cases, I would then want to send the mutated tuples: (a1, b, c) (a2, b, c) (a, b1, c) (a, b, c1) (a, b, c2) (a, b, c3)

Is there a name for such an operation? Are there libraries that make this simple?

share|improve this question

2 Answers 2

The name for this type of testing is combinatorial testing or sometimes referred to as pairwise testing. There is a very effective free tool from Microsoft that we also use for combinatorial testing called PICT.

More info is also available at http://www.pairwise.org and my blog post on combinatorial testing with negative values.

Essentially the input file for you example would look like this:

param1: a, ~a1, ~a2

param2: b, ~b1

param3: c, ~c1, ~c2, ~c3

The tilde character indicates to PICT that the value is a negative value. PICT will only use 1 negative value at a time in any combination (read my blog post above for suggestions on potential error masking).

The output from PICT for this input model is:

param1 param2 param3

a b c

~a2 b c

a b ~c3

a b ~c1

~a1 b c

a b ~c2

a ~b1 c

Let me know if you have any questions.

share|improve this answer
    
Pairwise is a subset of combinatorial, theoretically you can choose triplet of values. To be fair Hexawise sells (or gives for free to 1-5 users or Non-Commercial use) a tool that I heard is as good as PICT –  Rsf Aug 15 '12 at 9:58
    
Also note that many test harnesses such as NUnit also implement Pairwise support out of the box. –  Sam Woods Aug 15 '12 at 17:19
    
Thanks BJ. Your post on error masking with negative values in pairwise testing was actually what got me interested in this operation to begin with. –  jogo Aug 15 '12 at 20:36
    
I've been calling a set of inputs for pairwise testing a "pairwise crossproduct" (or an "n-wise crossproduct with n=2"). The set of inputs each containing a single invalid parameter value seems mathematically different enough to deserve its own name, especially since the verification steps for positive and negative test cases are often different. I've got "pairwise crossproduct" for the positive input operation: what's a good name for the negative? –  jogo Aug 15 '12 at 20:41

Jogo,

Nice question. I've upvoted your question because it's a good one as well as Bj's answer because (a) he's 100% correct and (b) his blog post on combinatorial testing with negative values is very good too.

Here's my 2 cents. I simply call them negative pairwise tests. Technically, maybe I should add "... with a mitigation strategy to address an error masking risk."

FWIW, here's what those tests look like in my company's Hexawise test design tool. I've taken the liberty of rewriting Parameter A to be a type of Card Used in a transaction, Parameter B to be an Expiration Date of the card, and Parameter C to be a Denomination of Payment. Each Parameter has one valid Value and the same number of invalid Parameters in your example.

First, you'd enter your Values (and indicate any constraints):

enter image description here

Next, (optionally) you'd add a few simple business rules to calculate Expected Results:

Expected results for negative pairwise tests in Hexawise

And then you'd calculate the results which have:

  • A maximum of one negative Value per test, and
  • Expected Results included

negative pairwise testing with error masking and expected results in Hexawise

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.