How to identify and justify how many test cases is enough when you have a potentially huge number of combinations?

Question

I'm working on a form builder project.

Users can drag and drop elements to build questionnaires.

If we simplify and say there is only 3 form elements that can be used:

1. Numeric answer, with configurable number of decimal points allowed 0 to 12
2. Multiple choice, click 1 radio button
3. yes or no, click 1 radio button

My maths says this gives us 3 forms with 1 control, 3 with 2 and 1 with 3.

We can then say that the numeric has 3 test configuration , (0,2 and 12 decimal points - the min/max plus happy common value, debatable but hopefully sensible).

Which gives us 5 forms with 1 control, 5 with 2 and 1 with 3.

However, one can have multiple of each control on a page. Lets say a page is limited to 100 controls.

It makes sense to me to test all of the above scenarios but then also where each control is duplicated - to see if having e.g. 2 numeric controls somehow interact with unintended consequences.

Beyond having 2 of each control - is there value in having 3 of each or 4 of each? How do I pick a number and justify it? I can't imagine if having 2 of each control on a page in each configuration doesn't cause any interactions, that having 3 or 4 or 9 - suddenly would do so?

Answer 1

There is much information to be found about these problems.

There are tools to generate combinations based on possible inputs, they try to help you to limit the total number but keep a representative 'coverage'. See here: http://www.pairwise.org/ for more information. I have used PICT and Hexawise in the past to generate combinational choices for testing.

If you want to dive deeper into this field, Cem Kaner has published a book about it: The Domain Testing Workbook

Whether it is "enough" is always a choice :-)

edit: updated links