I have a problem on deciding how to guarantee good testing of a complex distributed system before release. We have an automated testing infrastructure in place. The problem is that running all the tests takes around 5 days (including stress, longhaul etc.).
Let us assume that we start with build V1 of the system and we start testing on that. After 5 days, we look at the results and notice that there are some failures, either due to dev code changes, some due to breakage in the automation. In the meantime, devs continue work on their tasks and after 1 day, we fix all these issues, we get a new build out (V1.1 - which contains more dev changes + changes for the current testing) and we want to start testing again.
Now, the question is, what do we test? Do we spend 5 days running the full bunch of tests on V1.1 or do we run only specific tests for the affected code and pray/hope the components do not break? Can you please suggest some methodology around testing such systems?
One idea that I had was: create a branch in our repository that starting tests "snapshots" the dev branch and starts running the tests on the snapshot. If changes are required to either the dev or automation, those changes are made in the dev branch and cross-integrated to the snapshot branch. This guarantees that only changes for the test pass are in the snapshot branch, but the second problem is still not solved.
Thanks,
/cd