Managing repeatable test data when using cucumber and a BDD approach [Java, Selenium]

Question

When dealing with the highest level of the stack, functional end 2 end tests under no isolation, I am writing my selenium driven tests using the cucumber / BDD syntax and running each scenario in complete isolation on a forked JVM process.

Could I ask for some help on approaching the management of test data, currently I am using a randomisation technique, so for example lets say I have a user which has an expired attribute of true and they cannot login, I may retrieve such user via the following:

@Given("^\"([^\"]*)\" is an expired \"([^\"]*)\"$")
    public void is_an_expired_disabled(String userAlias, String userType) {
        switch (userType.toLowerCase()) {
            case "user":
                cucumberWorld.addNewAccount(userAlias, XploreAccountProvider.expiredUser());
                break;
            case "manager":
                cucumberWorld.addNewAccount(userAlias, XploreAccountProvider.expiredManager());
                break;
        }
        addUserViaApi(cucumberWorld.getAccountByAlias(userAlias));
    }

Now, this means I need to keep referencing such aliased user throughout the feature file, for example:

Given "Bob" is an expired "user"
When  They try to login
Then  They see the "expired.account" error message

What kind of approaches can we take so that we can run the same scenario over and over again, using the exact same data without the randomisation on each run, "Bob"'s information will vary on every execution right now because the User obj is populated with random data (that would meet the criteria for a valid test).

What approaches are viable here, do we create .json files of a user known as BoB and hardcode this scenario to go off and fetch such a user? concerns here is another layer of maintainance, if the user obj required changing in future, then all of these json blobs will require modifications.

What other approaches can we use? is the randomization a viable strategy?

Answer 1

It's tempting to hardcode user data but that can quickly be misleading if you use the same values each time as workarounds get applied. It's better to be able use arbitrary data that reflects the range of inputs your system must handle.

I would create some classes that generate valid user data for your tests instead of just using a file. Then use them to set defaults in your steps so that if your feature files don't specify values you will have sane ones to use. These classes could also be capable of altering that data to allow for injecting data to provoke edge case data as well as the happy paths.

If you then need to ensure this data is valid for your backend you can use the same classes to populate the backend data stores with the appropriate data before you execute the steps. You can also make subclasses of these that generate mocks of services instead of configuring them.

With this approach you have system level test that can be driven by arbitrary data from what ever source you care to add to the class (e.g. csv, xml , JSON ).

The key here is having classes so that you can trigger events when data is fetched to ensure it will be valid for your test and to fail early if that is not going to happen. This can save lots of tedious debugging and error handling by working as a gaurd clause.

Answer 2

Do you modify Bob's data during the run? If Bob is effectively read only, then you can create him once as you are doing. I would not randomize the data, I would use default values for the attributes that are not being tested for that account.

If you modify the user, then you should create a new user for each test or reset the state of the data before a test.

Using random values adds ambiguity to why a test failed, you should write explicit tests for the cases that a random value could expose (long strings, special characters, etc).