Currently you have a large test-suite in the scale of hundreds of test cases which is getting bigger every day
The requirement from the rest of the team members is simple: Let us execute tests on Dev, Test and Staging environments
The question becomes: Which tests ? You need to be able to separate the tests between those applicable to Dev, Test and Staging and of course this can be overlapping. You can have tests applicable both in Dev and Test but not in Staging. Or vice versa you can have tests applicable in Test and Staging but not applicable (any more) in Dev (because the functionality is removed or disabled). You get the point
In Pytest I can imagine three possible ways to achieve this:
- One is with tags/markers. You put on top of tests multiple markers like
@pytest.mark.devetc. I see this become hard work for one to maintain that these markers are going to reflect the truth always
- Folders. You move your entire test modules inside folders named
test_staging_envcorrespondingly. And then you have to move files from one folder to another to signify what belongs where Here you might be tempted to duplicate code, because yes you might endup in a weird situation where a testcase is applicable in Dev and Staging environments but not in the middle Test environment (because good old business decisions change all the time). Drawback here is that you bind a testcase with an environment and might be trickier for complex scenarios
- Git Branching. You have three separate git branches for each environment. You create tests on your main branch e.g.
masterand then you push your changes to
Which one would you recommend as the most maintainable solution ?
Or would you recommend a different way of managing this problem ?