Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Situation:

  • Large enterprise web application
  • Formal test plan in HP Application Lifecycle Management (née Quality Center)
  • Many dynamic text strings in the web application with several variables

An example of a dynamic text string:

{X1} prefer {X2} when going to the {X3}?

X1: IF UserHasDog = 1 THEN "Does your dog" ELSE "Do you"

X2: IF PlanetOfResidence = "Earth" AND UserHasDog = 1 THEN "bones or treats"
IF PlanetOfResidence = "Mars" AND UserHasDog = 1 THEN "rocks or dust"
IF PlanetOfResidence = "Earth" AND UserHasDog = 0 THEN "ice cream or donuts"
IF PlanetOfResidence = "Mars" AND UserHasDog = 0 THEN "dried vegetables or reconstituted meats"

X3: IF UserHasDog = 1 THEN "park" ELSE "supermarket"

My question is: is it really necessary to test for all possible combinations of the predicates (the part after the "IF" statement)? If so, then by this example, we'd have to take the exponential function 2^n where n is the number of unique predicates, to determine the number of test conditions. And then write out a truth table basically testing each logically-consistent interleaving of predicate truth values.

We have a few dynamically constructed strings where we have six or more predicates, leading to a huge number of test conditions if we test all combinations.

There are basically two ways to test it:

  1. Navigate through the web app in such a way as to produce each predicate exactly once, in isolation, without worrying about the other predicates for the other dynamic variables. If the part of the text string that said predicate purports to modify contains the correct substitution, pass the test and move on. OR....

  2. Write out a truth table and come up with all possible combinations (those which are not logically inconsistent; sometimes the conditions can be mutually exclusive or certain truth value combinations are impossible) and test each one in succession, to make sure that the underlying code is able to handle all possible string generation scenarios.

Assuming that it is more costly to allow a mistake to get into production than to spend extra time in testing, which option makes the most sense? Should we cut it off after a certain number of predicates, because the time investment required for, say, 10 predicates (2^10 = 1024 test conditions) would be excessively high?

Oh, one proviso: this is for manual testing, not automated. Even an automated solution would probably take unbearably long time after a certain number of predicates due to the nature of the exponential function, but I'm trying to get an idea of what is a sane approach for manual testing, where the tester has to login to the app, run through a scenario to satisfy the conditions, then check the substitutions in the dynamic string.

share|improve this question
add comment

2 Answers

up vote 6 down vote accepted

allquixotic,

How important is it that you get these string correct in all situations?

For example, if this is a life-critical application that depends intrinsically on the correctness of your strings (the difference between "Send police to the Town Hall" and "Send fire equipment to the Town Hall" for example), then you must do a deeper dive. Consider automation, as otherwise you will likely have no chance.

If instead, this is an internal application, or an application where the strings aren't critical, then you could consider a single pass through each.

You might also do a bit of research for "allpairs" and "pairwise testing" to see if the assumptions and shortcuts embodied in this technique apply in your case or not.

There has been some research done which tends to show that random combinations are equally as effective as "allpairs".

share|improve this answer
    
Very detailed answer, thanks -- I had no idea that "allpairs" existed! In one of my real-life examples, I was able to take a truth table of 16 conditions down to 10 by eliminating impossible situations (sometimes the predicates are made up of conjunctions or other boolean logic that would lead to a contradiction), then down to 6 by using the allpairs tool to eliminate a few combinations. No, they aren't life-critical, but will lead to user confusion or support calls if it's wrong. –  allquixotic Feb 8 '13 at 20:21
    
You are welcome! Remember, the fact that it is sufficient to rely on just pairs (rather than just singles or higher multiples) is an assumption we make when we choose that path. You might want to validate that assumption for your particular context. –  Joe Strazzere Feb 8 '13 at 20:56
    
It depends on what I can convince my coworkers to do :) Maybe we'll do exhaustive testing for 1 to 3 variables, allpairs for 4 to 10, and random spot checks for more than 10? I don't know. We'll have to try it out. But the conceptual introduction of allpairs into my brain is a huge contribution to my knowledge and development as a software tester. –  allquixotic Feb 8 '13 at 21:04
    
If nothing slee, contemplating an allpairs strategy forces you to think about how variables related to each other: which are independent, which are dependent, and so on. Even if you eventually choose not to use allpairs, the analysis will help you to make smarter decisions about how you test. –  user246 Feb 8 '13 at 22:09
    
Joe, the "research" on random combinations is a random selection from all possible combinations, and not simply randomly selecting combinations and how the random selections are performed is also a factor. In general, a well-known combinatorial tool such as Hexawise, PICT, ACTS, etc. is a more viable solution in this case. I would also recommend reading csrc.nist.gov/groups/SNS/acts/Crosstalk0806KuhnLeiKacker.pdf –  Bj Rollison Feb 8 '13 at 22:55
show 1 more comment

Here is a link to a pdf article "Pairwise Testing: A Best Practice That Isn’t" (http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf) by Patrick J. Schroeder and James Bach that discusses pairs testing and random testing. In essence, it promotes having multiple techniques in your toolbox when approaching this type of problem. Hope that helps!

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.