Managing exclusive resources across tests

Question

I have a serial E2E test suite that's growing and getting slower.

To solve this issue, I have to run the tests in parallel. But creating a user and all the required configurations is very complicated in this project.

Having one user per test is also not ideal, as some pieces of the configuration cost money.

So I was thinking of using something similar to Jenkins Lockable Resources or, where each user is a resource that can be "locked" per test and "unlocked" at the end.

Is there a simple API for achieving this? (Or another solution)

Answer 1

Yes, there are several ways to manage exclusive resources across tests, including using a test runner with parallel execution capabilities or implementing custom locking mechanisms in your code.

One popular option is to use a test runner like NUnit or TestNG that supports parallel test execution. These runners typically provide various options for running tests in parallel, such as by method, class, or assembly. By configuring the runner to use a fixed number of threads, you can ensure that only a limited number of tests run concurrently, which can help reduce the load on your system and prevent resource contention.

Another approach is to implement custom locking mechanisms in your code. For example, you could use a database or a distributed lock manager like Redis or ZooKeeper to coordinate access to shared resources. You could also implement a simple locking scheme using file system locks or named mutexes.

Here's an example of how you might implement custom locking using C# and Redis:

1 Install the StackExchange.Redis NuGet package:

Install-Package StackExchange.Redis

2 Create a Redis client instance:

var redis = ConnectionMultiplexer.Connect("localhost");

3 Create a lock manager class that provides a Lock and Unlock method:

public class RedisLockManager
{
    private readonly IDatabase _redis;

    public RedisLockManager(IDatabase redis)
    {
        _redis = redis;
    }

    public bool Lock(string resource, TimeSpan timeout)
    {
        var key = $"lock:{resource}";
        var value = Guid.NewGuid().ToString();
        return _redis.StringSet(key, value, timeout, When.NotExists);
    }

    public void Unlock(string resource)
    {
        var key = $"lock:{resource}";
        _redis.KeyDelete(key);
    }
}

4 In your test code, use the lock manager to acquire a lock on the resource you need to use:

[Test]
public void MyTest()
{
    var manager = new RedisLockManager(redis.GetDatabase());
    var locked = manager.Lock("my-resource", TimeSpan.FromSeconds(10));
    if (locked)
    {
        try
        {
            // Run test code that uses the resource
        }
        finally
        {
            manager.Unlock("my-resource");
        }
    }
    else
    {
        Assert.Fail("Could not acquire lock on resource");
    }
}

This approach provides a simple way to ensure that only one test is using a shared resource at a time. However, it does require some additional setup and may have performance implications depending on the number of resources and the frequency of contention.

Answer 2

The answer from @IAmMilinPatel is a great start, but it needs some improvements to be really fully distributed, be able to lock multiple resources and be flexible enough.

Multiple resources refers to tests requiring more than one resources where resource can be a physical thing or a feature of the system under test, some examples could be Android AND Android Version > 12 AND Pixel 7 phone, Chrome Browser AND Linux Red Hat or Database Was Recently Updated AND Software Version N Installed AND User of of type A.

An extension can be having a lockable resource of a certain type. To fulfil "I want a user that can add files to the system" you can lock one existing user, but also assign one from a list or create one on the fly and then lock it.

Why is that important? Efficiency. You don't want test to wait long for a resource to be randomly released, or even worst test can be deadlocked waiting for multiple resources- two tests waiting for resources A & B, test 1 holds A and test 2 holds B- they will never run.

The idea, shown here from King's tech blog and implemented in somewhat similar way in my team using The Actor Model, is:

• A job queue manager holds and dispatches tests according to available resource and an algorithm about allocating them. The queue manager should have an API to add tests/jobs and report the results synchronously/asynchronously.

• A resource manager communicate with the resources' device manager and can mark them as available, in use or "broken".

• A device manager. This can be a computer connected via USB to phones (the phone is the SUT), a computer running VMs (the VM is the SUT) or the device itself running a small manager software (the typical use case are Apple PCs that don't have good VM solution). The device manager stores and follows the device's state and configurations and communicates them to the resource manager upon request, this is needed so the resource manager don't have to know the device's internals.

• Some resource might have a special kind of manager, a good example are test users. The use manager can create them on the fly and remove them when needed, it can also clean up or load data into the user's data.

• A logical-test runner. This is another special resource to be allocated, in some cases the logic of the test needs to run on an agent other than the SUT and trigger action on the device. For example when you test a mobile device or test using multiple devices, think of an end to end video call. Again, the runners can be of different types and have various features.

• Finally you need to set up a communication methods and protocols between the parts of the system

To answer the original question, I am not aware of any such system ready "out of the box" and it is certainly not easy to develop and maintain.