Where to start when testing web pages with large number of options

Question

I have inherited a web application that has no useful requirements/specifications, and no tests. The page that I am currently looking at sets the configuration options for the application and consists of 46 options. Testing validations, minimum/maximum/default values etc for each field would suggest that this one page would require several hundred tests.

How would you approach breaking this down into something more manageable? One test case with a huge table of inputs and expected results? Hundreds of test cases? Something in between? Especially if most of the testing is going to be manual to start with.....

Update: following user246's response, here's a little more information, which otherwise would take up too much space as a comment.

user246's assumption is correct - I am currently trying to work out how to document and execute the tests, rather than how to automate them, although that would, of course, be the ultimate goal.

The application itself is quite old, written with ASP.NET WebForms and uses the DevExpress control libraries, so the validation routines ought to be consistent, assuming that they have been configured properly by the developers. The application and the DX controls do not lend themselves to easily automated tests (I've tried Watin and Selenium without a great deal of success).

I'm still trying to figure out where the interactions are - the UI certainly does not expose any connection between form fields, in fact, the configuration fields appear pretty much in the order that they were added to the form - not much indication of any design/planning at all.

Answer 1

David,

I've thought about this exact question quite a lot and I'm optimistic that I'd be able to help share some tips that you'd find useful. Unfortunately, explanations in a few paragraphs may not be sufficient to answer your question in detail. I'd be happy to talk by Skype today if you want to talk; I'll send you my Skype details via LinkedIn.

What you're describing is a combinatorial explosion; when the configuration settings and user actions and data entered etc. makes it impossible to test everything, how can you be smart about selecting a subset of tests that will test the SUT as thoroughly as possible in a limited number of tests.

It is quite similar (whether it first appears to be similar or not) to the general type of combinatorial explosion faced by the testers who tested Google Maps.

An abbreviated version of a suggested approach that has worked for many teams (for web testing or virtually any kind of testing for that matter) is as follows:

While you are exploring the system and testing it out to find "where the weak points are," you will probably find it useful to vary things as much as possible, repeat your actions as little as possible. Those points are true whether you're doing relatively informal lightly documented Exploratory Testing or more heavily documented test scripts. It addition, since a large percentage of defects can be triggered by the interaction of just two test inputs, it would be nice, if you had time, to test every single possible combination involving two test inputs; that's the rationale behind allpairs, pairwise and orthogonal array-based test case prioritization methods.

I'd highly suggest looking at the Google Maps test plan example found here: https://app.hexawise.com/share/94JBFALL and then taking a crack at designing a similar set of draft tests for your SUT.

The draft tests you'd create would be good but imperfect. They'd cover a lot in a relatively few tests but you would be well advised to add other types of tests rather than rely on any one particular testing strategy.

Unfortunately, a login to our test design tool is required to look at that Google Maps example in detail. Accordingly, I've created a free account for you at david.keaveny@stackexchange.com (I know it's not your real address, but I had to use something). I've sent you the password to your account to you via LinkedIn.

To recreate a similar - very early draft - plan for yourself, I'd suggest going through the following steps to put together a relatively small number of highly informative end-to-end-ish tests:

1. Ask WHAT can change as users go through the system? Think about configuration settings, user actions, data formats, data ranges, etc. even throw in more "creative" ideas like user personas. Let your creativity and common sense guide you. Enter those in as Parameters.
2. Ask HOW those parameters can change? (for the Parameter "Browser" enter IE7, IE8, FF, etc.) Put those in as Values under each parameter (entering constraints as required)
3. Ask does that variation MATTER? (e.g., use Equivalence Classes and be biased towards fewer Values vs. more - at least for your early draft tests)
4. Ask what SPECIAL paths thorough the system do you want to be sure to include? (Most common happy path, paths to trigger certain business rules, etc.)

Click the Create Tests button and you'll instantly get a very nice DRAFT starter set of highly varied tests. If they look like they're relatively interesting and don't miss hugely important things, start informally executing them and you'll be sure to learn some more things as you do about the system's weak points that would result in you going back to those DRAFT tests and iterating them to make them stronger and cover more.

This is unfortunately a rather "abstracted" example, devoid of a practical example in this text. A couple examples that might make this approach a little more "real" can be found at http://hexawise.com/case-studies/insurance and http://hexawise.com/case-studies/mortgage If it has captured your interest and you'd like to talk more, I'd be happy to explain how you could try to apply this approach to your specific SUT if you'd be interested.

I hope this helps. Again, though, it may leave you with more questions than answers. I'd be happy to talk by Skype if you'd find it useful.

• Justin

Answer 2

Welcome to SQA, David. I suspect many of us have been in the same position -- I certainly have.

You did not ask for advice on what to automate, so I assume you are mostly interested in how to document and execute a test of so many options. One way to reduce the effort is to investigate how the 46 options are implemented. The more code they share, the easier the test effort could be. For example, if each option has a minimum and maximum numeric value, and the same piece of code performs range checking on all of the options, you may not need to test the range checking logic for every option.

As far as documenting is concerned, I would aim for eliminating as much boilerplate as possible -- as you put it, you should aim for "a huge table of inputs and expected results". If a set of options are of the same type, you should describe the process for testing that kind of option exactly once and then perhaps enumerate the options of that type.

I noticed you did not mention any tests of interactions between options. If you need to look for bugs that only arise when certain combinations of values occur, your testing problem becomes much bigger. Luckily, there are ways to deal with combinatorial testing problems; see for example Techniques for test selection, give a large suite of test cases.

Answer 3

I would go with the one test case, plus the huge table of inputs and expected results.

46 options on a single page sounds like a UI nightmare to me. But if that's the hand you are dealt, you may want to consider some data-driven test automation. Scripts are really good about accepting input and expected output from a table, and driving your page through all of them quickly.

Sometimes you can automate through the custom controls directly, and sometimes you cannot. In the latter case, try other less-direct methods like copying to the clipboard and reading results from there. Resort to image comparisons only if nothing else works.

Answer 4

Do you know how your users are using it ? Can you get real live data to see if all those options are used or is the Pareto number in affect where 80% of the users are using 20% of the options ?

What is your testing objective - why are you being asked to do it ? Is it live and buggy ? Are enhancements being made ?

Is it generally solid or flaky ? Do some exploratory test sessions and find out. Do you have to go the test case route or could you set up a number of charters and test this way ( do you know about charters ? )

Answer 5

David, Justin already tipped you about Hexawise but i wanted to add that there are other tools than hexawise that do pairwise combinations for you:

• "ALLPAIRS Test Case Generation Tool" http://www.satisfice.com/tools.shtml
• PICT by microsoft
• NIST by the US government

These are to my knowledge all free and modifiable by the user and there are a few others as well.

Also, i'd like to share some strengths and weaknesses of orthogonal arrays and why it is enough(in most cases) to only test all pairs and not triple(or more) combinations.

90%+ of bugs are caused by by pair combinations, it is very rare that triple combinations cause an error that would not appear in pair combination and if you do quadruple or quintuple combinations the likelihood falls even lower. The bigger the combination the less likely the bug is too happen. Also to test all triple(or more) combinations takes a long time and in 99% of the cases it is not worth spending the time to find a bug that most probably is going to turn out to be trivial. So in summary pairwise does not cover triple or more combinations but unless you're testing medical equipment or other critical software like space rocket flight computers it's not going to matter.

But orthagonal arrays will only get you part of the way. Since you were not aware of orthagonal arrays i'll assume you aren't aware of boundary value testing and equivalence partitioning techniques either. But for fields where you have a range of values that can be entered this will help you. I'm not going to go into details about these techniques as they are easy to understand quickly and you can just google it now that you know that they exist.Now remember these two techniques aren't foolproof and there are pitfalls with all techniques but they should definitely help you.