We're running our own CI/performance tests system for some other team's software library as they've yet to decide on something beyond nightly tarballs and our requirements are much more stringent. Due to a notoriously segregated company structure, we have zero influence over the situation.

As part of our performance tests, we check a (public and documented) statistics variable so we don't have to monkey-patch our own frames-per-second counters, memory trackers, etc.
They now have a pull request open that would slightly change this stats variable, just enough that we have to adapt our test scripts. However, due to inertia and business protocols, them actually merging this PR is still in the future - requiring us to run two versions of our tests, one for 'old' code/pull requests and one for code already based on the 'new' interface.

My question now: Is there an accepted best-practices way of handling multiple test pipelines or does the whole thing boil down to us doing a if (revision predates commit X) { old_tests(); } else { new_tests(); }? What if we have to support more than just two versions?

  • Will the statistics be numerically different after this change or is it just a formatting change?
    – sphennings
    Commented Feb 22, 2018 at 20:08
  • @sphennings the majority will be numerically different most likely Commented Feb 22, 2018 at 20:30
  • Are you just expecting new values for the same statistics, or are you expecting to have new statistics that will require a completely new assessment?
    – sphennings
    Commented Feb 22, 2018 at 20:37

1 Answer 1


If the statistics are formatted differently you only need to modify how your get_statistics() methods parse their data sources.

If the statistics are numerically different, requiring you to set different target values. It's probably a good idea start storing those statistics in a config file. Then you can isolate version specific logic into the code that selects which set of target values to load.

Some test frameworks have features that make it easy to conditionally skip or expect the failure of specific tests. This is useful when there are only specific tests that you want to treat differently. Pytest's mark.skip(), mark.skipif() and mark.xfail() decorators are a good example of this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.