3

I'm putting together a test plan to test a logic engine that takes input from a user in forms of answers to mainly closed questions. The engine then evaluates the answers against certain rules and concludes if the input makes them eligible or ineligible. Depending on the answer, there are further drill down questions which can go an extra 2 levels deep and can have up to 5 options for each. Essentially, there's a huge amount of routes and combinations that the user can go through and input.

The primary feature I want to test is the logic in that it evaluates the inputs correctly based upon the rules there are. The system works by taking the questions in three stages, and evaluates at each stage, so I also need to test that it evaluates correctly at each of these stages, and won't allow a user to progress further than they should, even if they are eventually evaluated correctly at a further stage.

I'm doing this manually, and don't have any testing software available to me so will be doing this in Excel.

I'm planning to use the pairwise method to get the most test coverage with the least input combinations, but it's still going to result in a huge amount of combinations.

Any guidance/ advice on this would be helpful.

Edit: Including drill down questions, there are 727 questions.

4
  • Can you assess what would be the number of routes without pairwise? I am not sure pairwise is a good technique for your task. – Alexey R. Jun 8 '18 at 11:18
  • I can't give an exact number as it would take time to work out in itself to due the diverse amount of routes that can be taken depending on options but it's at least 100,000, which isn't feasible to run through exhaustively. I'm happy to consider other options though, of course. – ahu77 Jun 8 '18 at 11:25
  • 1
    So the user starts from one question and if they fail to answer correctly then they are asked to answer some additional question right? and if they answer the additional question they proceed with the next question from the level-1, otherwise they get deeper to the level-3? is it correct? If yes, then we should at first bound the number of questions a single user might be asked. What would be the maximum number of questions the user will have to answer if they fail all the questions? what would be the maximum number of questions the user will have to answer if all the answers will be correct? – Alexey R. Jun 8 '18 at 11:35
  • The drill down are generally questions for further information, so if they answer "yes" to a question for example, may then be 5 options to choose from as a result. Picking one of these options may then result in an additional set of options. But essentially you're right with the logic you're applying. I don't have that info, but assuming I did, how could these be applied? To give some context to these they are health related questions assessing eligibility for certain plans. – ahu77 Jun 8 '18 at 11:58
1

How I would approach the task. I wouldn't use pairwise technique. I would rather use Equivalent Partitioning technique. Since we're testing a finite set of rules (we do not have 100000 rules one for each route) I would concentrate not on the combinations but on the states (the more abstract the states are - the better).

Treat you rule engine as finite state machine.

Hence I would analyze the rules (what the rule inputs are) and for each rule I would compose a state of 1 input from within the range and the value on the boundary.

For example if a rule brings you from level-2 to level-3 you should only test that transition. You shouldn't test what if you will get back to level-2 and then again to level-3 and then again to level-2 and then again to level-3.

5
  • Thanks for the response Alexey. What would be the best way of analsying, documenting and ultimately creating test cases from the analysis? The rules are documented in a spreadsheet that documents the possible options for each question in referenced sheets. It also states whether each answer will evaluate to, and links to a further drill down set of questions for each answer given if there is one. As it's a health related set of questions, some evaluations will take multiple answers into account e.g. a smoker may be allowed the product if their BMI is a certain value. – ahu77 Jun 8 '18 at 14:05
  • Also, it's worth noting that they are mostly closed questions, but some are open. – ahu77 Jun 8 '18 at 14:08
  • I've just done a count of the questions and there are 727 questions including drill down questions. – ahu77 Jun 8 '18 at 14:22
  • If we classify the questions what the classes would be? If the drill down question has more deep insight on the topic of the original question? – Alexey R. Jun 8 '18 at 14:26
  • The questions are in the form of: Select list > 2 options, checkbox, or boolean. Each of the questions evaluate eligibility or trigger further drill down questions which follow the same type but with the addition of open, but range validated fields e.g. 0-999. I would classify them as base questions: top level questions and drill down questions: which are questions that are only asked if certain answers are given. – ahu77 Jun 8 '18 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.