Should A Formal Test Plan For Dynamic String Testing Include All Combinations?

Question

Situation:

• Large enterprise web application
• Formal test plan in HP Application Lifecycle Management (née Quality Center)
• Many dynamic text strings in the web application with several variables

An example of a dynamic text string:

{X1} prefer {X2} when going to the {X3}?

X1: IF UserHasDog = 1 THEN "Does your dog" ELSE "Do you"

X2: IF PlanetOfResidence = "Earth" AND UserHasDog = 1 THEN "bones or treats"
IF PlanetOfResidence = "Mars" AND UserHasDog = 1 THEN "rocks or dust"
IF PlanetOfResidence = "Earth" AND UserHasDog = 0 THEN "ice cream or donuts"
IF PlanetOfResidence = "Mars" AND UserHasDog = 0 THEN "dried vegetables or reconstituted meats"

X3: IF UserHasDog = 1 THEN "park" ELSE "supermarket"

My question is: is it really necessary to test for all possible combinations of the predicates (the part after the "IF" statement)? If so, then by this example, we'd have to take the exponential function 2^n where n is the number of unique predicates, to determine the number of test conditions. And then write out a truth table basically testing each logically-consistent interleaving of predicate truth values.

We have a few dynamically constructed strings where we have six or more predicates, leading to a huge number of test conditions if we test all combinations.

There are basically two ways to test it:

1. Navigate through the web app in such a way as to produce each predicate exactly once, in isolation, without worrying about the other predicates for the other dynamic variables. If the part of the text string that said predicate purports to modify contains the correct substitution, pass the test and move on. OR....

2. Write out a truth table and come up with all possible combinations (those which are not logically inconsistent; sometimes the conditions can be mutually exclusive or certain truth value combinations are impossible) and test each one in succession, to make sure that the underlying code is able to handle all possible string generation scenarios.

Assuming that it is more costly to allow a mistake to get into production than to spend extra time in testing, which option makes the most sense? Should we cut it off after a certain number of predicates, because the time investment required for, say, 10 predicates (2^10 = 1024 test conditions) would be excessively high?

Oh, one proviso: this is for manual testing, not automated. Even an automated solution would probably take unbearably long time after a certain number of predicates due to the nature of the exponential function, but I'm trying to get an idea of what is a sane approach for manual testing, where the tester has to login to the app, run through a scenario to satisfy the conditions, then check the substitutions in the dynamic string.

Answer 1

allquixotic,

How important is it that you get these string correct in all situations?

For example, if this is a life-critical application that depends intrinsically on the correctness of your strings (the difference between "Send police to the Town Hall" and "Send fire equipment to the Town Hall" for example), then you must do a deeper dive. Consider automation, as otherwise you will likely have no chance.

If instead, this is an internal application, or an application where the strings aren't critical, then you could consider a single pass through each.

You might also do a bit of research for "allpairs" and "pairwise testing" to see if the assumptions and shortcuts embodied in this technique apply in your case or not.

There has been some research done which tends to show that random combinations are equally as effective as "allpairs".

Answer 2

Here is a link to a pdf article "Pairwise Testing: A Best Practice That Isn’t" (http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf) by Patrick J. Schroeder and James Bach that discusses pairs testing and random testing. In essence, it promotes having multiple techniques in your toolbox when approaching this type of problem. Hope that helps!