Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I have inherited a web application that has no useful requirements/specifications, and no tests. The page that I am currently looking at sets the configuration options for the application and consists of 46 options. Testing validations, minimum/maximum/default values etc for each field would suggest that this one page would require several hundred tests.

How would you approach breaking this down into something more manageable? One test case with a huge table of inputs and expected results? Hundreds of test cases? Something inbetween? Especially if most of the testing is going to be manual to start with...

Update: following user246's response, here's a little more information, which otherwise would take up too much space as a comment.

user246's assumption is correct - I am currently trying to work out how to document and execute the tests, rather than how to automate them, although that would of course be the ultimate goal.

The application itself is quite old, written with ASP.NET WebForms and uses the DevExpress control libraries, so the validation routines ought to be consistent, assuming that they have been configured properly by the developers. The application and the DX controls do not lend themselves to easily automated tests (I've tried Watin and Selenium without a great deal of success).

I'm still trying to figure out where the interactions are - the UI certainly does not expose any connection between form fields, in fact the configuration fields appear pretty much in the order that they were added to the form - not much indication of any design/planning at all.

share|improve this question
    
Where validation is made? On UI or backend? If UI does not lend well to automation and you are still concerned with the number of test cases it will take, automated tests of the backend would be an option. –  dzieciou Dec 17 '12 at 9:47

4 Answers 4

up vote 5 down vote accepted

David,

I've thought about this exact question quite a lot and I'm optimistic that I'd be able to help share some tips that you'd find useful. Unfortunately, explanations in a few paragraphs may not be sufficient to answer your question in detail. I'd be happy to talk by Skype today if you want to talk; I'll send you my Skype details via LinkedIn.

What you're describing is a combinatorial explosion; when the configuration settings and user actions and data entered etc. makes it impossible to test everything, how can you be smart about selecting a subset of tests that will test the SUT as thoroughly as possible in a limited number of tests.

It is quite similar (whether it first appears to be similar or not) to the general type of combinatorial explosion faced by the testers who tested Google Maps.

An abbreviated version of a suggested approach that has worked for many teams (for web testing or virtually any kind of testing for that matter) is as follows:

While you are exploring the system and testing it out to find "where the weak points are," you will probably find it useful to vary things as much as possible, repeat your actions as little as possible. Those points are true whether you're doing relatively informal lightly documented Exploratory Testing or more heavily documented test scripts. It addition, since a large percentage of defects can be triggered by the interaction of just two test inputs, it would be nice, if you had time, to test every single possible combination involving two test inputs; that's the rationale behind allpairs, pairwise and orthogonal array-based test case prioritization methods.

I'd highly suggest looking at the Google Maps test plan example found here: https://app.hexawise.com/share/94JBFALL and then taking a crack at designing a similar set of draft tests for your SUT.

enter image description here

The draft tests you'd create would be good but imperfect. They'd cover a lot in a relatively few tests but you would be well advised to add other types of tests rather than rely on any one particular testing strategy.

Unfortunately, a login to our test design tool is required to look at that Google Maps example in detail. Accordingly, I've created a free account for you at david.keaveny@stackexchange.com (I know it's not your real address, but I had to use something). I've sent you the password to your account to you via LinkedIn.

To recreate a similar - very early draft - plan for yourself, I'd suggest going through the following steps to put together a relatively small number of highly informative end-to-end-ish tests:

  1. Ask WHAT can change as users go through the system? Think about configuration settings, user actions, data formats, data ranges, etc. even throw in more "creative" ideas like user personas. Let your creativity and common sense guide you. Enter those in as Parameters.
  2. Ask HOW those parameters can change? (for the Parameter "Browser" enter IE7, IE8, FF, etc.) Put those in as Values under each parameter (entering constraints as required)
  3. Ask does that variation MATTER? (e.g., use Equivalence Classes and be biased towards fewer Values vs. more - at least for your early draft tests)
  4. Ask what SPECIAL paths thorough the system do you want to be sure to include? (Most common happy path, paths to trigger certain business rules, etc.)

Click the Create Tests button and you'll instantly get a very nice DRAFT starter set of highly varied tests. If they look like they're relatively interesting and don't miss hugely important things, start informally executing them and you'll be sure to learn some more things as you do about the system's weak points that would result in you going back to those DRAFT tests and iterating them to make them stronger and cover more.

This is unfortunately a rather "abstracted" example, devoid of a practical example in this text. A couple examples that might make this approach a little more "real" can be found at http://hexawise.com/case-studies/insurance and http://hexawise.com/case-studies/mortgage If it has captured your interest and you'd like to talk more, I'd be happy to explain how you could try to apply this approach to your specific SUT if you'd be interested.

I hope this helps. Again, though, it may leave you with more questions than answers. I'd be happy to talk by Skype if you'd find it useful.

  • Justin
share|improve this answer
    
Many thanks for such a detailed answer; I've had a look at Hexawise and it looks very useful. I plugged in some initial input values - my 46 fields yielded 2.1E18 combinations, although Hexawise boiled that down to 88 tests! Digging through the source code, it's looking like a lot of the configuration options are orthogonal, so I can probably reduce that still further. –  David Keaveny Dec 18 '12 at 23:01

Welcome to SQA, David. I suspect many of us have been in the same position -- I certainly have.

You did not ask for advice on what to automate, so I assume you are mostly interested in how to document and execute a test of so many options. One way to reduce the effort is to investigate how the 46 options are implemented. The more code they share, the easier the test effort could be. For example, if each option has a minimum and maximum numeric value, and the same piece of code performs range checking on all of the options, you may not need to test the range checking logic for every option.

As far as documenting is concerned, I would aim for eliminating as much boilerplate as possible -- as you put it, you should aim for "a huge table of inputs and expected results". If a set of options are of the same type, you should describe the process for testing that kind of option exactly once and then perhaps enumerate the options of that type.

I noticed you did not mention any tests of interactions between options. If you need to look for bugs that only arise when certain combinations of values occur, your testing problem becomes much bigger. Luckily, there are ways to deal with combinatorial testing problems; see for example Techniques for test selection, give a large suite of test cases.

share|improve this answer
    
Thanks for your answer; I've added some more detail to the original question, the comment field doesn't really give enough space. –  David Keaveny Dec 17 '12 at 2:48

Do you know how your users are using it ? Can you get real live data to see if all those options are used or is the Pareto number in affect where 80% of the users are using 20% of the options ?

What is your testing objective - why are you being asked to do it ? Is it live and buggy ? Are enhancements being made ?

Is it generally solid or flaky ? Do some exploratory test sessions and find out. Do you have to go the test case route or could you set up a number of charters and test this way ( do you know about charters ? )

share|improve this answer
    
I'm not aware of any instrumentation being in place to track how the form is being used. The application is live and gets updated from time to time, but management want to increase the frequency of roll-outs. To date, there is no formalised testing as part of the release process, and no dedicated test team, so it has fallen to me (a developer, not the one responsible for the application) to come up with something. And I know nothing about charters! –  David Keaveny Dec 17 '12 at 4:45

I would go with the one test case, plus the huge table of inputs and expected results.

46 options on a single page sounds like a UI nightmare to me. But if that's the hand you are dealt, you may want to consider some data-driven test automation. Scripts are really good about accepting input and expected output from a table, and driving your page through all of them quickly.

Sometimes you can automate through the custom controls directly, and sometimes you cannot. In the latter case, try other less-direct methods like copying to the clipboard and reading results from there. Resort to image comparisons only if nothing else works.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.