What kind of pattern can we follow when we have a lot of system wide settings in the system?

Question

Maybe not massively common, but what type of approach and or patterns can we apply when we are facing quite a lot of system wide settings that interact heavily with our users permissions and state etc, think global settings applied to all users.

When running our tests in a parallelised manner, how best to achieve this without major headaches?

Scenario: Consider a setting when enabled applies to ALL users, this setting may be something like:

Enforces two factor auth for every user: **true**

How do we not cause conflicts when running a large suite in parallel where say 15-20% encompass the setting to be false ?

I'm thinking maybe run the ones which rely on it after, almost like 2x sequential-parallelised executions if you will.

Curious on some input.

Answer 1

My personal opinion:

I am not sure if there is a fancy name for this, but I would build a table:

• Trying to identify tests that share the most common settings and group them together.
• Order them by the number of common settings they share, the less they share, they higher they are ordered. E.g. if they share no settings in common, they should be ranked as No. 1; if they share every setting, they should be ranked the lowest.
• Execute them in parallel in this order you get from the previous step.

Question:

• Can you not use virtual environments? If you set up multiple virtual environments, your tests would not be interfering with each other as much.

Answer 2

You can't, easily. You can potentially divide your tests up into buckets, or whatever you want to call them, and coordinate that "This is the MFA bucket," and "this is the security disabled bucket," and so on. You then have to go through all of your test cases and figure out which ones go into which bucket(s), where you may well need to run a single test case multiple times, if it would be relevant for multiple buckets.

As Yu Zhang said, the real solution to this is virtual/multiple environments. By definition, if you have settings that impact the entire system, you can't test changes in isolation.

Answer 3

I have been in similar situation as you.

I have been testing authentiation scenarios for users with very different setup. Scenarios can change the state of the user (e.g., lock user out when invalid password is given 5 times in a row), so it was important to use different users for different scenarios. First idea was to create test users for each scenario, but in our case the process of creating a user and granting certain permission is often time-consuming, increasing test execution time significantly, and hard to automate. Instead, we have decided to find users matching prerequisities for certain tests in existing database.

Here's is how this works:

1. Test framework finds a group of X users from DB and caches them in a pool available for all tests
2. 1st test gets 1st user from the group, 2nd test gets 2nd user from the group, etc.. X-th test gets X-th user from the group, and then (X+1)-th test gets agains 1st user from the group. If you assume that each test uses only one user and number of tests that you can run in parallel is Y, then everything works fine only if Y <=X. Otherwise, two tests running in parallel can accidentally reuse same user at same time.
3. It is very import that the test leaves the user in clear state after completion.
4. There can be mutiple pools for different groups of tests, e.g., a pool of users with TFA, a pool of external users, a pool of users who can login only through SSO, etc.

Obviously, this work if pool of users can be coordinated between tests, e.g., all tests are run in the same JVM. However, in Jenkins you can have additional parallelism, where tests from job A and running in parallel to tests from job B but there is no simple way to share a coordinate user pools between those jobs. The way I solved it was grant each job a specific groupId. This way the pool of users for job A gets first group of matching users from DB (1..X matching users from DB), the pool of users for job B gets second group of matching users from DB (X+1..2X matching users from DB), etc.