How to approach running automated tests "from the head"?

Question

I'm currently one of a group of developers developing tests for a web-based line of products. These products are separate but related, and are periodically released (ie versioned).

We are trying to make our tests all "run from the head" against all different versions, meaning that the current state of our master test code branch can run meaningfully against arbitrary versions of the production builds. At first, this seems a bit intimidating, particularly since our tests are generally UI (Selenium) based, and hence, quite dependent on particular versions.

Any tips for making this process easier? Or at least, more doable?

Answer 1

You are astute to recognize that change is the Achilles Heel of automated UI tests. This is no different from developing/maintaining a shared library that needs to work with multiple versions of an application.

There are practices you can follow to reduce (but not eliminate) the pain. On the product side:

• Use predictable element IDs in your UI so that tests are less reliant on the DOM layout.
• Design your pages using reusable components so that the page design is more consistent; this reduces the number of patterns the test needs to understand.
• Get your UI developers involved in the test design so that everyone understands the impact of changes.

On the test side:

• Take advantage of element IDs when they are available (and when they are consistent; dynamic element IDs are not particularly useful to a tester).
• Structure your tests to take advantage of page design patterns.
• Allow your tests to be aware of what product version they are running against, and tag your tests with version information.
• Consider negotiating the number of production versions you intend to support. Perhaps there is some middle ground between compatibility with only the latest code and compatibility with arbitrary versions.
• Consider how much of test suite needs to remain compatible. Some tests may be more crucial than others; it may be enough to commit to compatibility for only a subset of the full test suite.

Answer 2

I'm going to presume that you have sound reasons, whether business or otherwise, to do this. It seems simpler to me to pull the appropriate versions of your test code from the source repository (you do have your test code in a source repository and flag the versions, right?), but anyway...

I see two main options here. One is to include version detection and an internal listing of features per build. Benefits here include making the test simpler because if the version list says feature X isn't part of it, the script never checks feature X, and the presence of a 'feature oracle'. Drawbacks are that it's more complex to implement and maintain, since the feature/version matrix has to be correct and up to date or you could have features not being tested (if the oracle says the feature is from a later version or is missing the feature entirely) or tests failing (if the oracle says the feature is from too early a version).

The other is to build feature detection into your scripts instead: check for the existence of the link to feature X and act on it only if it exists. Advantages include - if the feature is there it will be tested, tests will never fail on trying to exercise a feature that doesn't exist. Disadvantages include - this will be slower because there will always be the necessary checking time to determine whether the feature exists, you won't know if a feature is missing from a version (unless you use both approaches, which of course gets you all the advantages and drawbacks of both).

Regardless, changes to application objects will get you some pretty ugly code: instead of pulling versioned code that refers to the specific object, your script code will end up with a decision tree based on version or whether the object can be found under any of its known names.

I'm honestly not sure that the advantages of having a single branch and version of your script codebase outweigh the disadvantages of this approach, but if I had to, I'd approach it with a hybrid approach as I described above.

Good luck.