Should end-to-end tests be self-contained?

Question

I am automating web UI tests using Selenium. These tests are written partly in the spirit of Unit tests, that is they try their best to leave the application in the same state in which they found it. For instance, if a test registers a support ticket using the web UI, it is expected to remove it and the end of the test using the web UI.

Compared to unit tests, these kinds of tests do much more, not in terms of actual code executed or processes triggered, but in terms of tested actions. One such hypothetical test would be:

1. (Actor 1) register support ticket;
2. (Actor 1) comment with initial data;
3. (Actor 2) reply with request for more information
4. (Actor 1) upload more information
5. (Actor 2) comment that issue is fixed
6. (Actor 1) close issue

All of these actions are verified during a test, and should one fail, the application data is inconsistent.

Is there a better way of making end-to-end tests self-contained and idempotent?

Answer 1

Your best bet is to make tests that are idempotent. We use a tool called nosedjango to run all of our tests that handles the database, cache, etcs. This would only work if you are using django however. I am sure there are equivalent tools for whatever it is you might be using.

Answer 2

Personally I think that there are a few key principles.

1. Tests should, where possible, assume that they have run previously and failed.

2. Some tests will want to validate a feature, whilst others will need to just "drive through" a feature on the way to test something else.

3. I can scale out and use many machines, so that I can run a lot of tests in parallel, and I am not time limited on a run.

With those in mind, I would, where possible:

• Have each test generate unique data for itself that does not rely on a previous test passing
• Have the logic for execution separate from validation code

so for your example I would create and execute six separate "objects" for each of your actions:Registering a support ticket, adding a comment, replying to a request for more information, uploading additional information and close a ticket.

Then I would string them together in six test cases as follows:

1. Register support ticket and validate
2. Register a support ticket, then add a comment and validate
3. Register a support ticket, then add a comment, reply with request for more information and validate
4. Register a support ticket, then add a comment, reply with request for more information, upload more information and validate
5. Register a support ticket, then add a comment, reply with request for more information, upload more information comment that issue is fixed validate
6. Register a support ticket, then add a comment, reply with request for more information, upload more information comment that issue is fixed, close and validate

As I test registration 6 times, I would build up a test data object, and use different test data each time so I can test a different scenario each time the registration occurs as well.

So the main "cost" is execution time as I execute 6 long tests as opposed to one (but using open source tools means I can scale out easily) The use of objects means that I don't have lots of duplicated code everywhere. This also means that if the register a ticket part of the application is changed, I only change one object in one place and all the tests keep running.

Here is this approach in code for a single "object" using "Parkcalc".

    [TestMethod]
    public void EconomyLessThanOneHour()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "10:59", "AM", "today");
    }

    [TestMethod]
    public void EconomyExactlyOneHour()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "11:00", "AM", "today");
    }

    [TestMethod]
    public void EconomyMoreThanOneHour()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "11:01", "AM", "today");
    }

    [TestMethod]
    public void EconomyJustLessThanOneDay()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "9:59", "AM", "today+1");
    }

    public void EconomyExactlyOneDay()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "10:00", "AM", "today+1");
    }

    [TestMethod]
    public void EconomyJustMoreThanOneDay()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "10:01", "AM", "today+1");
    }

    [TestMethod]
    public void EconomyJustLessThanOneWeek()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "9:59", "AM", "today+7");
    }

And the implementation of CalculateAndVerify ...

 namespace Parkcalc.Logical
 {
    public class Parking
    {
        public static void OpenHome()
        {
             Physical.NavigateTo.Homepage();
        }

         public static void Calculate(ParkingType parkingType, string inTime, string inAMPM, string inDate, string outTime, string outAMPM, string outDate)
         {
            inDate = Utility.CalculateDate(inDate);
            outDate = Utility.CalculateDate(outDate);
            Physical.Parking.Calculate(parkingType, inTime, inAMPM, inDate, outTime, outAMPM, outDate);
         }

         public static void CalculateAndVerify(ParkingType parkingType, string inTime, string inAMPM, string inDate, string outTime, string outAMPM, string outDate)
         {
             Calculate(parkingType, inTime, inAMPM, inDate, outTime, outAMPM, outDate);
             Verification.Verify.VerifyResult(parkingType, inTime, inAMPM, inDate, outTime, outAMPM, outDate);
         }
     }
 }

and the next layer down

namespace Parkcalc.Physical
{
    public static class Parking
    {

        public static void Calculate(ParkingType parkingType, string inTime, string inAMPM, string inDate, string outTime, string outAMPM, string outDate)
        {
            NavigateTo.Homepage();
            Browser.SetValue(Controls.ParkingCalculator.ddlLot,EnumValues.GetParkingType(parkingType));
            if (!string.IsNullOrEmpty(inTime))
            {
                Browser.SetValue(Controls.ParkingCalculator.txtEntryTime, inTime);
            }

            if (!string.IsNullOrEmpty(inAMPM))
            {
                Browser.SetValue(Controls.ParkingCalculator.rdoEntryTimeAMPM, inAMPM);
            }

            if (!string.IsNullOrEmpty(inDate))
            {
                Browser.SetValue(Controls.ParkingCalculator.txtEntryDate, inDate);
            }

            if (!string.IsNullOrEmpty(outTime))
            {
                Browser.SetValue(Controls.ParkingCalculator.txtExitTime, outTime);
            }

            if (!string.IsNullOrEmpty(outAMPM))
            {
                Browser.SetValue(Controls.ParkingCalculator.rdoExitTimeAMPM, outAMPM);
            }

            if (!string.IsNullOrEmpty(outDate))
            {
                Browser.SetValue(Controls.ParkingCalculator.txtExitDate, outDate);
            }

            Browser.Invoke(Controls.ParkingCalculator.btnCalculate);
        }

        public static string GetResult()
        {
           return Browser.GetValue(Controls.ParkingCalculator.txtResult);
        }
    }
}

I have a full working C# example using WatiN here ... http://testingstax.codeplex.com/SourceControl/changeset/view/7938#

For test data I generally build a data generator and add unique random strings so that there is no data collisions.

Answer 3

I've seen tests written that way, but if your test suite depends on each test always cleaning up after itself, your suite will be fragile; a bug in one test may cause a propagation of false positives (or false negatives) in subsequent tests.

Sometimes you can make tests independent by partitioning the environment. For example, if you are testing a application that supports multiple users, you might want to run each test (or each group of tests) as a different user.

Answer 4

I've always operated under the philosophy that the tests will give you the best information when they are run on a fresh install, complete with a fresh (empty/default populated) database. I have even seen discussions about spawning a new VM programmatically from a saved image to ensure the least possible interaction between two test runs.

Consider the classic bug of "If I add a user and remove him, it works. If I add a user and remove him 1000 times, it fails." We've all had some kind of bug like that. When I run a test, more than once, I expect to get the EXACT same results. There may be some minor differences in speed execution due to the nature of running programs not always coming out exact, but the closer they are to exact, the better.

This means if it fails, and I get a debug output of the Object that failed, and it has id = 14 on one run, it had better have id = 14 on another run. Whatever logic decided to make it 14 once should make it 14 again unless the program changed. If it ends up being 28 the second time, how do I know the first run didn't influence the second run?

We live in an edge where our products have to run perfect, every time. Often we're legally/contractually obligated to do so. And that means our tests have to give us as much information as possible.

That's not to say there might not be benefit in running the test AGAIN. Essentially, yes it should run exactly the same (obviously it may have different universal IDs, and other small differences) and if you have the spare CPU cycles, might be worth doing again. But I also kind of feel that anything that would be covered by a second test SHOULD be a test case normally anyway.

tl;dr I perfer not just resetting the database, but installing a fresh application, on a fresh machine if possible. Scripts can automate this like a breeze.

Answer 5

Cleaning after you is a the best way to write automated tests, although it does not guarantee the tests are 100% idempotent, for example the cache might be changed, register values changed etc.

Answer 6

In similar scenario, we designed & implemented

E2E tests which create & consume their own data. Even though tests are very long but they are absolutely independent without any side effects.

In the worst case scenario, a test might fail due to any reason without impacting any other single test in the suite in any way.This ensures that we don't have any cascading effect in between tests.