Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I am automating web UI tests using Selenium. These tests are written partly in the spirit of Unit tests, that is they try their best to leave the application in the same state in which they found it. For instance, if a test registers a support ticket using the web UI, it is expected to remove it and the end of the test using the web UI.

Compared to unit tests, these kinds of tests do much more, not in terms of actual code executed or processes triggered, but in terms of tested actions. One such hypothetical test would be:

  1. (Actor 1) register support ticket;
  2. (Actor 1) comment with initial data;
  3. (Actor 2) reply with request for more information
  4. (Actor 1) upload more information
  5. (Actor 2) comment that issue is fixed
  6. (Actor 1) close issue

All of these actions are verified during a test, and should one fail, the application data is inconsistent.

Is there a better way of making end-to-end tests self-contained and idempotent?

share|improve this question
    
Do you need to do this or can you reset the database at the end? I can imagine cases where you want to compare the DB at the end of a test and reset it to start from a known state before you begin. I have thought of doing similar tests, but that is not always practical. –  MichaelF May 5 '11 at 12:35
    
I do not need or want to do this but I don't currently have a way of resetting the database. And basically that would also mean restarting the application since caches and sessions would need to be invalidated as well, and that would get messy soon. –  Robert Munteanu May 5 '11 at 12:58
    
The only issue I see with this is that if your test fails for some reason, you may be left with inconsistent data that you cannot easily clean up. But if you can make it so, then I'd do it to the best of your ability, or make sure you have the tools that do so. –  MichaelF May 5 '11 at 13:08
add comment

5 Answers

up vote 3 down vote accepted

Your best bet is to make tests that are idempotent. We use a tool called nosedjango to run all of our tests that handles the database, cache, etcs. This would only work if you are using django however. I am sure there are equivalent tools for whatever it is you might be using.

share|improve this answer
add comment

Personally I think that there are a few key principles.

  1. Tests should, where possible, assume that they have run previously and failed.

  2. Some tests will want to validate a feature, whilst others will need to just "drive through" a feature on the way to test something else.

  3. I can scale out and use many machines, so that I can run a lot of tests in parallel, and I am not time limited on a run.

With those in mind, I would, where possible:

  • Have each test generate unique data for itself that does not rely on a previous test passing
  • Have the logic for execution separate from validation code

so for your example I would create and execute six separate "objects" for each of your actions:Registering a support ticket, adding a comment, replying to a request for more information, uploading additional information and close a ticket.

Then I would string them together in six test cases as follows:

  1. Register support ticket and validate
  2. Register a support ticket, then add a comment and validate
  3. Register a support ticket, then add a comment, reply with request for more information and validate
  4. Register a support ticket, then add a comment, reply with request for more information, upload more information and validate
  5. Register a support ticket, then add a comment, reply with request for more information, upload more information comment that issue is fixed validate
  6. Register a support ticket, then add a comment, reply with request for more information, upload more information comment that issue is fixed, close and validate

As I test registration 6 times, I would build up a test data object, and use different test data each time so I can test a different scenario each time the registration occurs as well.

So the main "cost" is execution time as I execute 6 long tests as opposed to one (but using open source tools means I can scale out easily) The use of objects means that I don't have lots of duplicated code everywhere. This also means that if the register a ticket part of the application is changed, I only change one object in one place and all the tests keep running.

Here is this approach in code for a single "object" using "Parkcalc".

    [TestMethod]
    public void EconomyLessThanOneHour()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "10:59", "AM", "today");
    }

    [TestMethod]
    public void EconomyExactlyOneHour()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "11:00", "AM", "today");
    }

    [TestMethod]
    public void EconomyMoreThanOneHour()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "11:01", "AM", "today");
    }

    [TestMethod]
    public void EconomyJustLessThanOneDay()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "9:59", "AM", "today+1");
    }

    public void EconomyExactlyOneDay()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "10:00", "AM", "today+1");
    }

    [TestMethod]
    public void EconomyJustMoreThanOneDay()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "10:01", "AM", "today+1");
    }

    [TestMethod]
    public void EconomyJustLessThanOneWeek()
    {
        Parking.CalculateAndVerify(ParkingType.EconomyParking, "10:00", "AM", "today", "9:59", "AM", "today+7");
    }

And the implementation of CalculateAndVerify ...

 namespace Parkcalc.Logical
 {
    public class Parking
    {
        public static void OpenHome()
        {
             Physical.NavigateTo.Homepage();
        }

         public static void Calculate(ParkingType parkingType, string inTime, string inAMPM, string inDate, string outTime, string outAMPM, string outDate)
         {
            inDate = Utility.CalculateDate(inDate);
            outDate = Utility.CalculateDate(outDate);
            Physical.Parking.Calculate(parkingType, inTime, inAMPM, inDate, outTime, outAMPM, outDate);
         }

         public static void CalculateAndVerify(ParkingType parkingType, string inTime, string inAMPM, string inDate, string outTime, string outAMPM, string outDate)
         {
             Calculate(parkingType, inTime, inAMPM, inDate, outTime, outAMPM, outDate);
             Verification.Verify.VerifyResult(parkingType, inTime, inAMPM, inDate, outTime, outAMPM, outDate);
         }
     }
 }

and the next layer down

namespace Parkcalc.Physical
{
    public static class Parking
    {

        public static void Calculate(ParkingType parkingType, string inTime, string inAMPM, string inDate, string outTime, string outAMPM, string outDate)
        {
            NavigateTo.Homepage();
            Browser.SetValue(Controls.ParkingCalculator.ddlLot,EnumValues.GetParkingType(parkingType));
            if (!string.IsNullOrEmpty(inTime))
            {
                Browser.SetValue(Controls.ParkingCalculator.txtEntryTime, inTime);
            }

            if (!string.IsNullOrEmpty(inAMPM))
            {
                Browser.SetValue(Controls.ParkingCalculator.rdoEntryTimeAMPM, inAMPM);
            }

            if (!string.IsNullOrEmpty(inDate))
            {
                Browser.SetValue(Controls.ParkingCalculator.txtEntryDate, inDate);
            }

            if (!string.IsNullOrEmpty(outTime))
            {
                Browser.SetValue(Controls.ParkingCalculator.txtExitTime, outTime);
            }

            if (!string.IsNullOrEmpty(outAMPM))
            {
                Browser.SetValue(Controls.ParkingCalculator.rdoExitTimeAMPM, outAMPM);
            }

            if (!string.IsNullOrEmpty(outDate))
            {
                Browser.SetValue(Controls.ParkingCalculator.txtExitDate, outDate);
            }

            Browser.Invoke(Controls.ParkingCalculator.btnCalculate);
        }

        public static string GetResult()
        {
           return Browser.GetValue(Controls.ParkingCalculator.txtResult);
        }
    }
}

I have a full working C# example using WatiN here ... http://testingstax.codeplex.com/SourceControl/changeset/view/7938#

For test data I generally build a data generator and add unique random strings so that there is no data collisions.

share|improve this answer
add comment

I've seen tests written that way, but if your test suite depends on each test always cleaning up after itself, your suite will be fragile; a bug in one test may cause a propagation of false positives (or false negatives) in subsequent tests.

Sometimes you can make tests independent by partitioning the environment. For example, if you are testing a application that supports multiple users, you might want to run each test (or each group of tests) as a different user.

share|improve this answer
add comment

I've always operated under the philosophy that the tests will give you the best information when they are run on a fresh install, complete with a fresh (empty/default populated) database. I have even seen discussions about spawning a new VM programmatically from a saved image to ensure the least possible interaction between two test runs.

Consider the classic bug of "If I add a user and remove him, it works. If I add a user and remove him 1000 times, it fails." We've all had some kind of bug like that. When I run a test, more than once, I expect to get the EXACT same results. There may be some minor differences in speed execution due to the nature of running programs not always coming out exact, but the closer they are to exact, the better.

This means if it fails, and I get a debug output of the Object that failed, and it has id = 14 on one run, it had better have id = 14 on another run. Whatever logic decided to make it 14 once should make it 14 again unless the program changed. If it ends up being 28 the second time, how do I know the first run didn't influence the second run?

We live in an edge where our products have to run perfect, every time. Often we're legally/contractually obligated to do so. And that means our tests have to give us as much information as possible.

That's not to say there might not be benefit in running the test AGAIN. Essentially, yes it should run exactly the same (obviously it may have different universal IDs, and other small differences) and if you have the spare CPU cycles, might be worth doing again. But I also kind of feel that anything that would be covered by a second test SHOULD be a test case normally anyway.

tl;dr I perfer not just resetting the database, but installing a fresh application, on a fresh machine if possible. Scripts can automate this like a breeze.

share|improve this answer
add comment

Cleaning after you is a the best way to write automated tests, although it does not guarantee the tests are 100% idempotent, for example the cache might be changed, register values changed etc.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.