Organizing testing for product with multiple client customization versions

Question

My problem is as follows. Currently I am on a project, where we have a standard product that gets changed/bespoke for each client individually. My task is to create a test approach, in order to test the product, but also test for each client individually. As we are a small testing team, testing for each client individually will become impossible if we get to a large amount of clients.

So, please can I ask if someone has been in a similar situation to give guidance?

Answer 1

My recommendation would be to build an automation suite, on the base of the product. Then, to have your testers manually test the differences in the product based off of your client requirements. Unless you hire other testing resources, automation will do a lot of the testing for you, in the base product.

Answer 2

One of the approaches that can be used to aid in automated testing of all the client specific builds is to adjust the design to simplify both test and the production of client specific builds.

1. Influence the design such that all the clients have the same code, with all the functionality present, but with which functionality is available to the specific client being determined by one, or more, of licence file options, licence server replies, registry entries, configuration file contents. You can then have a test suite that tests all of the product functionality with all of the functionality enabled and a client specific test that simply has to confirm which features are available and unavailable.
2. Another option is to have the base product structured into core items and customer specific items and a configuration tool that reads the client specific details and outputs them to both a set of compile time options and a set of test options/limits. Then your test suite will consist of the core functionality tests plus those tests enabled/tailored for the client.

Either, or both, of these options will probably require some restructuring and rework of the product code base but in addition to improved test coverage and reduced test time, both well worth the investment by themselves, they will also result in the following improvements for the coding department:

• Some clear delineation between core code and client specific code
  • this will speed up the generation of client specific code,
  • simplify source code version control,
  • possibly allow some division of responsibility between programmers,
  • ease the roll out of bug fixes, etc., e.g. an important change in the core code section should possibly result in notifications or updates for all customers major changes in customer specific code should not but can you currently easily tell which is which?
• Either approach will tend to make the generation of the customer specific build, and the associated tests, more of a configuration exercise than a programming one possibly allowing the support or project staff to do it freeing up your talented programmers for more productive work and speeding the response to customer specific requests.
• It will often allow the identification of the code that should possibly be refactored into generic functions rather than a multitude of customer specific functions. This in turn will often allow testing and bug fixing before the customer has made up their minds about something. e.g. If alarm sounds and duration vary between customers then a generic alarm module can be specified, developed and tested that takes an input file name identifying one, or more, specific format of audio file plus reads a specific setting from a configuration source. This can then be adjusted, without code changes with just configuration changes at any time up to, and possibly beyond, customer delivery. Since you have, hopefully, already tested all the valid inputs and the fall backs for erroneous inputs last minute customer changes only require testing that the customer is happy with the settings that they have selected.

As both a software engineer and a tester I would always rather work a little harder and smarter once, then be able to move on to something else, than use a quick hack over and over again and possibly risk forgetting for one, or more, customers/occasions.

Answer 3

An automated test suite for the primary product before it gets customized is a good way to go.

You could go a step further and automate the creation and testing of the bespoke components too. If the group that handles the bespoke part is also your testing team then simplifies the politics. If your automation scripts are checked into source control then you can always go back and recreate a particular bespoke component. Knowing what was done is so valuable when testing and troubleshooting. And! There's great satisfaction in saying, "I ran the script the way I always do. If there are problems, it wasn't me."

You didn't state if the bespoke parts are compiled differences or configuration differences. Either way, scripting the compiling, composition, and testing of all components will make your life much easier.