Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

Our system under test have schema that differs between system versions 1.0 and 2.0: new entities have been added, some have columns added. We used Hibernate to generate Java entities for both schema versions. Then, depending on which version we are testing, we are switching the classpath to those Java entities. This way we can use run the same test cases and use the same DB assertions in both 1.0 testing and 2.0.

This approach resolves the problem at runtime. However, at compilation time, we got a problem:

  • for entities that are specific only for schema 1.0, tests/assertions for schema 2.0 fail to compile; and vice versa
  • for entities that are common and same for both schema, there is no problem.

Is branching our test cases and test framework in SVN the only approach to this problem? The same way devs branches the system?

That will be put additional effort on us to maintain each branch.

share|improve this question
add comment

4 Answers

up vote 2 down vote accepted

In my experience branching is the simplest way to handle multiple versions of the system in test. I'd suggest taking a look at this question and the answers, too: Strategies for automating testing on features that are only on certain environments

In that one I've given a list of strategies that can be used, which also applies to versioning.

My experience is that while the company is maintaining more than one version of their software, it's necessary for test automation to maintain more or less matching versions of the automation. Depending on how much changes between versions you can run n+1 automation against application version n (if there are no additions to the schema and nothing has been deprecated). If nothing has been deprecated, you can usually run n-1 automation against application version n.

Typically I've found that in the standard release schema of Release.MajorVersion.MinorVersion.BuildNumber, automation versioning needs to happen at the Release.MajorVersion level. Managing keeping versions in line is part of the automation harness: whatever you use to automatically retrieve your automation code for the run has the version to pull in its code (named branches work well for this). That same task is the one that indicates the version of the application to retrieve for the runtime tests.

The versioned artifacts should also include your baselines and other reference data so that when that changes you aren't failing tests because of it.

There is some additional effort to maintain multiple branches, but typically maintenance required on older branches should be minimal and most of your effort will go into your current development branch.

share|improve this answer
    
Cost of maintaining branches will be higher, if test suites for each release/major version aren't stable (e.g., we are currently increasing coverage in end-to-end tests), so every change in one branch must be merged into another. I think a good moment to create branch would be when test suites are relatively stable for one release/major version, so only bug fixes are merged. –  dzieciou Aug 18 '13 at 10:29
    
@dzieciou - That's usually when it's done, yes. –  Kate Paulk Aug 18 '13 at 13:20
add comment

I am not a Hibernate expert, so I can't offer an advice on that front. There are ways to use Java reflection to convert compile-time dependencies into run-time dependencies, at the cost of additional code complexity. You could accomplish the same thing by reimplementing the relevant tests in a JVM-based scripting language.

I can tell you that in every test job I've had, it was necessary to branch the test code to account for differences in the product code. We didn't need to branch every time the product branched, but more often than not it was necessary. Of course, over time the number of test branches increased, but we didn't need to maintain most of the older ones, so the maintenance cost was acceptable.

share|improve this answer
add comment

Store your test code in the same repository as the production code. That way they are always in sync, and you avoid the version-matching headaches.

share|improve this answer
    
That's a good idea, but will require QAs and development work more closely to immediately resolve sync problems (e.g. compilation failures). We do so for unit and integration tests, but end-to-end tests we versioned in a separate repository. Anyway, might be worth considering merging both repositories together, thanks! –  dzieciou Aug 18 '13 at 10:15
add comment

In the past I have built a system whereby the tests know what version of code they are running against. This allowed me to have a single test code base, but have tests that are specific to V1, common to both, or specific to V2. I used the same mechanisms for the code to run in different locales where the business rules were different as, in my case the accounting treatment differed between three countries for a single, global in-house application. I could set the default in a config file, and had custom attributes on test cases that were specific to an application version.

In my general experience I have found that ideally the strategy for managing your test code should mirror your production code. Did they branch the production code? If yes than you should branch too. Do they have the same code base with conditional logic and compile time switches? Talk to your developers to find out their approach and you should probably follow suit.

share|improve this answer
    
In your 1st approach (1st paragraph) didn't you have test fixtures specific for a given version of a system? If so, then I guess those fixtures weren't tightly coupled to the system API, classes, DB so you had no compilation, runtime problems, correct? –  dzieciou Aug 18 '13 at 10:27
1  
No, it was all at the test level, my libraries could handle all the versions. Yes they were not tightly coupled. –  Bruce McLeod Aug 18 '13 at 10:52
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.