How to set up selenium tests so that if one fails, the others still run?

Question

To give some context I am building a testing system with Selenium, nUnit and Visual Studio in c# that will allow me to test a number of websites one after another.

The issue I have is that all of the tests are stored in separate classes and are called one after another. This works great, but if any tests are to fail all of the other tests following them will not be executed.

I tried to work around this by using a verify instead of an assert, this does work but any data about the failure is not shown.

In an ideal situation, the Site A tests would run one after another, if any tests are to fail it would store that data and continue with the next test until it reaches the last test and will start the tests for Site B.

Below is an example of how I call the tests:

    [Test]
    public void Tests()
    {
        foreach (var baseURL in baseURLS)
        {
            // Frontend Failed login
            var Frontend_FailedLogin = new UseCases.Frontend_Tests.Account.Frontend_FailedLogin(this._driver, baseURL);
            Frontend_FailedLogin.FailedLogin();

            // Frontend Login test
            var Frontend_Login = new UseCases.Frontend_Tests.Account.Frontend_Login(this._driver, baseURL);
            Frontend_Login.Login();
        }
     }

Does anyone have any suggestions on how to handle this? The main issue is that I need to run these tests one after another but still be able to see what tests have failed.

I am pretty new to this so please excuse my lack of knowledge on the subject.

Answer 1

I'd suggest you data drive your tests. There are two major ways you can do this with minimal repetition.

Option 1 - Data-Driven URL List - Minimal Test Code Logic Changes (changes below are pseudo-codey - I don't know the nUnit syntax for data source attributes):

[Test DataSource="URLList.csv"] // syntax of the data source attribute will depend on your engine and where the data is stored
public void FailLogin()
{
    // Frontend Failed login
    var Frontend_FailedLogin = new UseCases.Frontend_Tests.Account.Frontend_FailedLogin(this._driver, baseURL);
        Frontend_FailedLogin.FailedLogin();
    Assert(IsFalse, User.IsLoggedIn); // Again, syntax depends on your plugins
}

[Test DataSource = "URLList.csv"] // Syntax depends on engine
public void SuccessLogin()
{
        // Frontend Login test
        var Frontend_Login = new UseCases.Frontend_Tests.Account.Frontend_Login(this._driver, baseURL);
        Frontend_Login.Login();
        Assert(IsTrue, User.IsLoggedIn); // Syntax depends on engine
 }

You may want to add a test cleanup method along the lines of (more pseudocode because I suspect the nUnit cleanup attribute isn't the same as the MS unit test one than actual code):

[TestCleanup]
public void Cleanup()
{
    if (User.IsLoggedIn) 
    {
        User.LogOff();
    }
    BrowserWindow.Close();
}

This helps keep your code modular and minimizes repeated code. If there's too much of a hit from using an external data source multiple times you can set up an internal source instead in class initialize or assembly initialize routine and pull the data from that - but you would then lose the automatic cycling through all rows in the data source.

Option 2 - Also Data-Driven

This option requires more logic - you would have your data source contain not only the URLs but whether or not the login should pass/fail (alternatively you could pass in the user name & password and the expected outcome)

Assuming this structure in the input file (CSV):

URL,IsLoggedOn
www.something.com,false
www.somethingelse.org,true

The code would be something like the pseudo-code below:

[Test DataSource="URLList.csv"]
public void LoginTest()
{
    bool ExpectSuccess = TestContext.DataSource.Param("IsLoggedOn");
    var Frontend_Login = new UseCases.Frontend_Tests.Account.Frontend_Login(this._driver, TestContext.DataSource.Param("URL"));

    if(ExpectSuccess)
    {
        Frontend_Login.Login();
    }
    else
    {
        Frontend_Login.FailedLogin();
    }

    Assert(ExpectSuccess, User.IsLoggedOn);
}

Answer 2

Get used to the thought of separation of concerns.

Some good ideas are to:
have separate projects for separate websites.
have separate frameworks for separate suites.
have separation between unit, integration, and user acceptance level tests.

In NUnit with C# you would create a framework like below and have the tests inherit it:

[TestFixture]
public class Framework
{

    [TestFixtureSetUp]
    public void BeforeEverything()
    {
    }

    [TestFixtureTearDown]
    public void AfterEverything()
    {
    }

    [SetUp]
    public void BeforeEachTest()
    {
    }

    [TearDown]
    public void AfterEachTest()
    {
    }
}