I've thought about this exact question quite a lot and I'm optimistic that I'd be able to help share some tips that you'd find useful. Unfortunately, explanations in a few paragraphs may not be sufficient to answer your question in detail. I'd be happy to talk by Skype today if you want to talk; I'll send you my Skype details via LinkedIn.
What you're describing is a combinatorial explosion; when the configuration settings and user actions and data entered etc. makes it impossible to test everything, how can you be smart about selecting a subset of tests that will test the SUT as thoroughly as possible in a limited number of tests.
It is quite similar (whether it first appears to be similar or not) to the general type of combinatorial explosion faced by the testers who tested Google Maps.
An abbreviated version of a suggested approach that has worked for many teams (for web testing or virtually any kind of testing for that matter) is as follows:
While you are exploring the system and testing it out to find "where the weak points are," you will probably find it useful to vary things as much as possible, repeat your actions as little as possible. Those points are true whether you're doing relatively informal lightly documented Exploratory Testing or more heavily documented test scripts. It addition, since a large percentage of defects can be triggered by the interaction of just two test inputs, it would be nice, if you had time, to test every single possible combination involving two test inputs; that's the rationale behind allpairs, pairwise and orthogonal array-based test case prioritization methods.
I'd highly suggest looking at the Google Maps test plan example found here: https://app.hexawise.com/share/94JBFALL and then taking a crack at designing a similar set of draft tests for your SUT.
The draft tests you'd create would be good but imperfect. They'd cover a lot in a relatively few tests but you would be well advised to add other types of tests rather than rely on any one particular testing strategy.
Unfortunately, a login to our test design tool is required to look at that Google Maps example in detail. Accordingly, I've created a free account for you at email@example.com (I know it's not your real address, but I had to use something). I've sent you the password to your account to you via LinkedIn.
To recreate a similar - very early draft - plan for yourself, I'd suggest going through the following steps to put together a relatively small number of highly informative end-to-end-ish tests:
- Ask WHAT can change as users go through the system? Think about configuration settings, user actions, data formats, data ranges, etc. even throw in more "creative" ideas like user personas. Let your creativity and common sense guide you. Enter those in as Parameters.
- Ask HOW those parameters can change? (for the Parameter "Browser" enter IE7, IE8, FF, etc.) Put those in as Values under each parameter (entering constraints as required)
- Ask does that variation MATTER? (e.g., use Equivalence Classes and be biased towards fewer Values vs. more - at least for your early draft tests)
- Ask what SPECIAL paths thorough the system do you want to be sure to include? (Most common happy path, paths to trigger certain business rules, etc.)
Click the Create Tests button and you'll instantly get a very nice DRAFT starter set of highly varied tests. If they look like they're relatively interesting and don't miss hugely important things, start informally executing them and you'll be sure to learn some more things as you do about the system's weak points that would result in you going back to those DRAFT tests and iterating them to make them stronger and cover more.
This is unfortunately a rather "abstracted" example, devoid of a practical example in this text. A couple examples that might make this approach a little more "real" can be found at http://hexawise.com/case-studies/insurance and http://hexawise.com/case-studies/mortgage If it has captured your interest and you'd like to talk more, I'd be happy to explain how you could try to apply this approach to your specific SUT if you'd be interested.
I hope this helps. Again, though, it may leave you with more questions than answers. I'd be happy to talk by Skype if you'd find it useful.