One of the approaches that can be used to aid in automated testing of all the client specific builds is to adjust the design to simplify both test and the production of client specific builds.
- Influence the design such that all the clients have the same code, with all the functionality present, but with which functionality is available to the specific client being determined by one, or more, of licence file options, licence server replies, registry entries, configuration file contents. You can then have a test suite that tests all of the product functionality with all of the functionality enabled and a client specific test that simply has to confirm which features are available and unavailable.
- Another option is to have the base product structured into core items and customer specific items and a configuration tool that reads the client specific details and outputs them to both a set of compile time options and a set of test options/limits. Then your test suite will consist of the core functionality tests plus those tests enabled/tailored for the client.
Either, or both, of these options will probably require some restructuring and rework of the product code base but in addition to improved test coverage and reduced test time, both well worth the investment by themselves, they will also result in the following improvements for the coding department:
- Some clear delineation between core code and client specific code
- this will speed up the generation of client specific code,
- simplify source code version control,
- possibly allow some division of responsibility between programmers,
- ease the roll out of bug fixes, etc., e.g. an important change in the core code section should possibly result in notifications or updates for all customers major changes in customer specific code should not but can you currently easily tell which is which?
- Either approach will tend to make the generation of the customer specific build, and the associated tests, more of a configuration exercise than a programming one possibly allowing the support or project staff to do it freeing up your talented programmers for more productive work and speeding the response to customer specific requests.
- It will often allow the identification of the code that should possibly be refactored into generic functions rather than a multitude of customer specific functions. This in turn will often allow testing and bug fixing before the customer has made up their minds about something. e.g. If alarm sounds and duration vary between customers then a generic alarm module can be specified, developed and tested that takes an input file name identifying one, or more, specific format of audio file plus reads a specific setting from a configuration source. This can then be adjusted, without code changes with just configuration changes at any time up to, and possibly beyond, customer delivery. Since you have, hopefully, already tested all the valid inputs and the fall backs for erroneous inputs last minute customer changes only require testing that the customer is happy with the settings that they have selected.
As both a software engineer and a tester I would always rather work a little harder and smarter once, then be able to move on to something else, than use a quick hack over and over again and possibly risk forgetting for one, or more, customers/occasions.