Both the product I used to work on and the one I'm working on now have a fairly large and complex matrix of possible configurations, work-flows, and data scenarios. The one product had, no joke, close to (if not more than) 100 boolean configuration flags, not all necessarily in one place, but many of them mutually incompatible (if I set this bit to true, I cannot use the feature activated when I set that bit to false) without any real internal validation in the application to prevent invalid configurations. Additionally, while most of the users of this product used a standard configuration, all the users had some very unique combinations of configurations.
Many new feature developments for the product were client driven in that the client submitted a change request with requirements and development was performed against those requirements. But in those requirements for the change order, there was not always a transparency as to what the existing configurations were for the customer and so not a lot of clarity as to what the risk factors were for those non-standard configurations. Because new features, once released, were released to all existing clients with current maintenance contracts, risk management dictated that we focus most time and effort on those commonly used configurations.
However, back then, and now, I've experienced a number of situations where those non-standard configurations have bitten us badly. And the only way, really, to be aware of these is to have an exact copy of the client's current configurations. But with a client base of hundreds of clients, each with a different configuration, it does not seem practical to run tests through ALL clients data for EVER development effort.
So, there's a need at times to use client data during testing processes. But there's also a need to prioritize test cases to those with the broadest impact on the most clients. These seem to be, in part, conflicting needs.
The question then is: How does a software development organization building and modifying such a highly complex project determine when to use client data during pre-release testing processes while still maintaining time-boxed risk assessment necessary to meet delivery deadlines?