What criteria do I apply to a highly complex development project to warrant getting "real world" data?

Question

Both the product I used to work on and the one I'm working on now have a fairly large and complex matrix of possible configurations, work-flows, and data scenarios. The one product had, no joke, close to (if not more than) 100 boolean configuration flags, not all necessarily in one place, but many of them mutually incompatible (if I set this bit to true, I cannot use the feature activated when I set that bit to false) without any real internal validation in the application to prevent invalid configurations. Additionally, while most of the users of this product used a standard configuration, all the users had some very unique combinations of configurations.

Many new feature developments for the product were client driven in that the client submitted a change request with requirements and development was performed against those requirements. But in those requirements for the change order, there was not always a transparency as to what the existing configurations were for the customer and so not a lot of clarity as to what the risk factors were for those non-standard configurations. Because new features, once released, were released to all existing clients with current maintenance contracts, risk management dictated that we focus most time and effort on those commonly used configurations.

However, back then, and now, I've experienced a number of situations where those non-standard configurations have bitten us badly. And the only way, really, to be aware of these is to have an exact copy of the client's current configurations. But with a client base of hundreds of clients, each with a different configuration, it does not seem practical to run tests through ALL clients data for EVER development effort.

So, there's a need at times to use client data during testing processes. But there's also a need to prioritize test cases to those with the broadest impact on the most clients. These seem to be, in part, conflicting needs.

The question then is: How does a software development organization building and modifying such a highly complex project determine when to use client data during pre-release testing processes while still maintaining time-boxed risk assessment necessary to meet delivery deadlines?

Answer 1

On my team, the process is simple. We don't have automated tests, so pretty much all we get are using the real-world data, which is usually a week, month, or more out of date. It gets refreshed when someone either screws it up or they need to illustrate something that happened on life.

You attempt to reproduce it on a system you use all the time.

• If you can't there, then you find out more about their settings, and go to a system that has that configuration.
• If it still isn't able to be reproduced, we request a refresh of one of the systems from the live system.
• If we still can't reproduce it from the live configuration, then we assume the users were lying (which actually has happened!) or that they didn't really understand the procedure/error/scenario and only thought there was a problem. This is rare, as the analysts usually squash these "false bugs" before they ever get to the development staff.

Bottom line, you need a good way to export their configuration and load it in yours, if you don't have one already. You also need a good way of finding what configuration flags have the potential to cause problems in the areas they claim the error lies in. Then you don't have to use their exact configuration but can just tweak a couple on a system you use a lot.

Doing an analysis of the most popular and most problematic combinations of non-standard configurations will help you a lot. Don't test every single one (or you'll never finish a project) but getting the ones that are fairly common, or have a history of causing problems, will net you great results in testing.

Answer 2

I once worked on a financial services product with a large number of configuration settings -- perhaps as many as the questioner described. We had many customers, each of whom had their own configuration. All customers ran out of the same installation of the product, which we hosted in our data centre. Our database contained every customer's configuration settings and all of their data, so we knew a lot about how the product was used.

It was unfeasible to test every feature against every customer's configuration. However, in our product, it was never the case that product feature depended on every configuration setting. I researched each feature to come up with a list of configuration settings that the feature depended on. The list probably wasn't perfect -- I probably omitted something or unnecessary included something now and then -- but it was close. I analyzed the variability in those settings to figure out which configurations were actually used and which were supported by unused.

Given those lists, I wrote automated tests that tested high-priority features against the cross-product of those settings or, when that wasn't feasible, I tried to use a pair-wise set of configurations.

This approach wasn't perfect, but it let us get out of the trap of deciding which company configurations were the most important. The automated tests found bugs, and it seemed like our quality improved.

Answer 3

Configuration testing is a common software testing challenge. Configuration matrices can help outline common customer configurations and potentially even help identify some 'interesting' combinations.

For highly complex configurations a good approach is a combinatorial testing approach using a pair-wise or other n-way analysis of the possible configuration settings.

I would recommend taking a look at PICT tool documentation which provides rich examples for configuration testing. http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi