Currently I have to create manual test cases for a feature with 64 different combinations. I think it is not useful to create test cases for all of them. How do I find the most important combinations?

I think about border cases, but it is very hard to identify them. A other idea is to ask the business analyst what the most used combinations are. The feature is not online, so it would just be assumptions.

To automate them is no option - at the moment.

  • 2
    When you say important what exactly in your environment is important? The most frequently used, most business critical, most resource intensive, longest logic path, or some other criteria? Commented Jul 24, 2014 at 11:39
  • @NigelHoath That is a really good question. Maybe I can pick up these criterias and talk with the business analyst and developers to find an answer. With this answer I can create important test cases.
    – Twaldigas
    Commented Jul 24, 2014 at 11:48

1 Answer 1


There are tools that will generate a set of tests that cover every combination at least once without hitting all possible combinations (these work on the principle that most bugs will be surfaced by exercising pairs of settings).

Since the feature hasn't gone live, your business analyst is probably your best reference for the combinations that are most likely to be used. Depending on your customer base, you can probably start from the assumption that the feature will be most likely to be used with the default settings (Someone once did this - they asked Word users to send in their config files so they could analyze them and determine the most used options. What they found was that overwhelmingly most people went with the defaults).

From there, I'd look at settings that make sense for known users of other features: say you know that users of Feature A overwhelmingly have Application Setting X enabled. You also know that with Application Setting X, the new feature will only work with Feature Settings 1, 5, and 9 enabled. That's obviously a combination you need to cover.

Of course, if the new feature is independent of other application settings, that isn't relevant (with the caveat that I've found some really interesting bugs where settings should be independent and aren't...).

Without a pairwise testing tool to generate combinations, I'd analyze the settings myself as well, to try to determine the most likely to be used. After that, I'd look at building a set of tests that covers each setting in combination at least once. For on-off settings my starting points would be all on and all off, then a selection of on and off combinations.

For example, with four independent on-off settings A, B, C, D, I'd look at these mixes (in order, 1 is on, 0 is off):

  1. 1-1-1-1
  2. 0-0-0-0
  3. 1-0-1-0
  4. 0-1-0-1

This is pretty arbitrary and gets more... interesting when you have dependent settings and multi-valued settings. In those cases, I'd look to boundaries and dependencies as well.

  • Thank you for your answer. I used a pairwise testing tool, so I had get the number of 64 combinations. The "[...] that covers each setting in combination at least once. " approach is a really nice idea.
    – Twaldigas
    Commented Jul 24, 2014 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.