What would be the most maintainable way of defining your tests to separate them between environments (dev, test staging) using pytest?

Question

Currently you have a large test-suite in the scale of hundreds of test cases which is getting bigger every day

The requirement from the rest of the team members is simple: Let us execute tests on Dev, Test and Staging environments

The question becomes: Which tests ? You need to be able to separate the tests between those applicable to Dev, Test and Staging and of course this can be overlapping. You can have tests applicable both in Dev and Test but not in Staging. Or vice versa you can have tests applicable in Test and Staging but not applicable (any more) in Dev (because the functionality is removed or disabled). You get the point

In Pytest I can imagine three possible ways to achieve this:

• One is with tags/markers. You put on top of tests multiple markers like @pytest.mark.staging, @pytest.mark.dev etc. I see this become hard work for one to maintain that these markers are going to reflect the truth always
• Folders. You move your entire test modules inside folders named test_dev_env, test_env and test_staging_env correspondingly. And then you have to move files from one folder to another to signify what belongs where Here you might be tempted to duplicate code, because yes you might endup in a weird situation where a testcase is applicable in Dev and Staging environments but not in the middle Test environment (because good old business decisions change all the time). Drawback here is that you bind a testcase with an environment and might be trickier for complex scenarios
• Git Branching. You have three separate git branches for each environment. You create tests on your main branch e.g. master and then you push your changes to dev, test and staging branches accordingly

Which one would you recommend as the most maintainable solution ?
Or would you recommend a different way of managing this problem ?

Many thanks

Answer 1

I recommend

One set of tests

You've clearly defined a common problem that many of us face.
I have tried all of the solutions you outlined but they all seem to run into the issue of a lot of maintenance and knowledge and updating code based on environment.

My recommendation is one set of tests. If there are dependencies, e.g. database you create a connection string that is based on environment and then define that in one place such as a .config file. You may need to exclude this from version control so that each area (development, CI, production, etc) can set it without conflict. Keeping it separate and out of source control means that it is also a good solution for storing passwords which usually comes up at some point. Environmental variables are usually part of this solution. https://linuxhint.com/bash-environment-variables/

If there are dependencies on other systems, make sure your unit tests stub out all dependencies. In some shops this is the norm. In other it is presented as 'impossible', e.g. vendor does not have a test system, etc.

If you have tests that are only applicable to one environment, I would suggest working closely with developers to identify why that is the case and to come up with a way to address it - mocks, stubs, proxy servers for auth, etc.

Tags and flags are certainly an option but as you've guessed the maintenance is a big issue. Often quick hacks become embedded deeply in development and the original intent to keep them limited is lost.

Answer 2

I work in an environment where we have many environments but they're all part of a single release pipeline, i.e., features are first deployed to dev and they progress through all environments until they reach production. I can imagine some organisations might have different environments that are connected to different downstream systems, which possibly sounds like your environment.

The way I'd do this is with tags, but not the way you suggest. With our tests, we tag them based on a number of things: the feature they're testing, the type of test, the bug/work item number, etc. When our CI/CD pipeline deploys our code to a particular environment, we can control which tests are executed against that environment using those tags. In your example, I would set up each of these environments to execute the test suite using a combination of tags that only execute the tests that make sense for that environment. That way, you'll keep all of your tests together in the same branch of the same repository and let the pipeline dictate which tags need to run.