2

I'm using unit tests in VS to create my tests, and naturally I ran into the challenge that unit tests should not be dependant on the run order. However, since my tests are actually functional tests, the order is important.

So I thought ordered test would be the solution. However, I'm running into some problems which leads me to believe that an ordered test is not as easy to use as I thought.

Here is an example of the tests themselves:

The "master test class", which all others inherit from:

[TestClass]
public class MyTest {

    [TestInitialize]
        public void Init () {
        GCDriver.Initialize();

        MyDropStartPage.GoTo();
        Assert.IsTrue(MyDropStartPage.IsAt, "Something went wrong. The browser didn't open start page.");
    }

    [TestCleanup]
    public void Cleanup() {
        GCDriver.Close();
    }
}

A test class:

[TestClass]
public class Feature149304 : MyTest {

    [TestMethod]
    public void Navigate_to_a_Locations_page () {

    LeftMenu.Locations.Select();                                                         
        UITest(() => {
            Assert.IsTrue(LocationsPage.IsAt, "Something went wrong. The URL seems to be wrong.");                  // - and make sure we're really on the Locations page.
         });         
    } 
}

Just one test, and all it does is open the browser and navigate to a location. Run directly, it works fine.

However, if I put this (and only this) into an ordered test, and try to run that, I get this error:

Test Name:  feature_149304
Test FullName:  c:\users\149999frho\source\workspaces\qa\myautomation\mytests\regression test suite 1\feature 149304\feature_149304.orderedtest
Test Source:    c:\users\149999frho\source\workspaces\qa\myautomation\mytests\regression test suite 1\feature 149304\feature_149304.orderedtest :line -1
Test Outcome:   Failed
Test Duration:  0:00:00,0648434

Result StackTrace:  
at OpenQA.Selenium.DriverService.FindDriverServiceExecutable(String     executableName, Uri downloadUrl)
at OpenQA.Selenium.Chrome.ChromeDriverService.CreateDefaultService()
at OpenQA.Selenium.Chrome.ChromeDriver..ctor(ChromeOptions options)
at OpenQA.Selenium.Chrome.ChromeDriver..ctor()
at MyAutomation.GCDriver.Initialize() in C:\Users\149999frho\Source\Workspaces\QA\MyAutomation\MyAutomation\Selenium\GCDriver.cs:line 17
at MyTests.MyTest.Init() in C:\Users\149999frho\Source\Workspaces\QA\MyAutomation\MyTests\Base\MyTest.cs:line 12
Result Message: Initialization method MyTests.Feature149304.Init threw exception. 
OpenQA.Selenium.DriverServiceNotFoundException:     
OpenQA.Selenium.DriverServiceNotFoundException: 
The chromedriver.exe file does not exist in the current directory or in a
directory on the PATH environment variable. The driver can be downloaded at
http://chromedriver.storage.googleapis.com/index.html..

Now, not being completely stupid, I see that this error complains about the Selenium ChromeDriver not being found.

But how do I remedy this? Neither the "master" test class nor the other is using any Selenium references. The Selenium work is done by a class library, whose methods are called by the test classes (LeftMenu.Locations.Select(); etc.)

What makes this not work when the test is run from an orderedtest, when it works perfectly otherwise?

UPDATE:

The project folder structure:

Project: MyAutomation (class library)
- Folder: Selenium
- - Class: ChromeDriver
- Folder: Pages
- - Class: StartPage
- - Class: LocationsPage
- - ...

The ChromeDriver class is instantiated as a singleton, and all the different classes in the class library have methods which use an instance of this.

Project: MyTests (unit test project)
- Folder: Base
- - Unit test class: MyTest
- Folder: TestSuite1
- - Unit test class: FirstTest
- - Unit test class: SecondTest
- - Ordered test: FirstAndSecondTests

The unit test project references the class library project.

The MyTest unit test class has the Instantiate and TestCleanup functions, and it instantiates the ChromeDriver singleton in the class library project.

The other unit test classes (FirstTest, SecondTest ...) inherit from the MyTest unit test class, in addition to calling static methods in the StartPage and LocationsPage classes in the class library project, to do the actual testing.

The ordered test has FirstTest and SecondTest added.

UPDATE2

The directory structure on the disk:

Folder: C:\Users\149999frho\Source\Workspaces\QA\MyAutomation\MyTests\Regression Test Suite 1\Feature 149304

Files: Feature_149304.cs - this is the file with the individual unit tests
Feature_149304.orderedtest - this is the ordered test, containing one of the unit tests in the above file.

When logging the current working directory, when running the ordered test and the individual test respectively, I get this

Individual unit test:
Current dir: C:\Users\149999frho\Source\Workspaces\QA\MyAutomation\MyTests\bin\Debug

The ordered test (containing the above individual test):
Current dir: C:\Users\149999frho\Source\Workspaces\QA\MyAutomation\TestResults\149999frho_149999FRHO 2017-04-03 20_20_11\Out

Clearly, the working directories are different. But how come, when both tests are in the same project, project folder and folder on the disk?

Looking at the properties for the test classes, they're identical:

C:\Users\149999frho\Source\Workspaces\QA\MyAutomation\MyTests\Regression Test Suite 1\Feature 149304\Feature_149304.orderedtest
C:\Users\149999frho\Source\Workspaces\QA\MyAutomation\MyTests\Regression Test Suite 1\Feature 149304\Feature_149304.cs
  • Really? No one has any ideas/suggestion as to why this "inconsistency" occurs? – Hfrav Apr 5 '17 at 8:01
1

UPDATE to account for updates to the question:

This is a function of the way Microsoft test suites work (there is more information available at https://msdn.microsoft.com/en-us/library/ms182475.aspx).

According to the linked page, for your test suites to work, you will need to follow the instructions in that page to force the Chrome Driver to be moved to the working directory for your test suite.

You don't want to manually copy, because each run creates a new time-stamped directory within \TestResults\.


Original answer

Your problem is that your current working directory is not the same for your test suite as it is for your isolated test case.

Your error message shows the test suite directory as c:\users\149999frho\source\workspaces\qa\myautomation\mytests\regression test suite 1\feature 149304\

The test class directory is at C:\Users\149999frho\Source\Workspaces\QA\MyAutomation\MyTests\Base\

The easiest way to resolve this is to use the Visual Studio IDE and flag the ChromeDriver executable as both part of the project, and set it to Always Copy Local. That will put it in the bin directory along with the compiled files, and make it available to your test code regardless of whether you are calling from individual tests or from a test suite.

This problem has nothing to do with using ordered tests: it would have happened no matter what kind of test suite you used.

  • I don't understand that. The \Base\ class is a folder where ONLY the Base class i located; All the other, isolated test cases, are in different folders. These inherit from the \Base\Mytest class without problems. The point of separating the class library from all the test classes is to avoid any direct interaction with Selenium, and ChromeDriver, in the test classes. Only the class library project works with the selenium classes. – Hfrav Apr 3 '17 at 15:27
  • Your class library has its own working directory. When you run your test directly from the library, the library's working directory is used. When you run it from a test suite, the test suite's working directory is used. They are not necessarily the same directory, which means that the test suite doesn't know where to find your chrome driver (because it's somewhere that only the test library knows about). I'd have to look at your project structure to say more than that - but this is pretty basic programming. – Kate Paulk Apr 3 '17 at 15:53
  • But all my individual tests are in another folder from the class library. It's even a different project. The test class project has a reference to the class library project, of course. Also, I'm unsure what you mean by "test library"? I've got a test project, with test classes - including the ordered test, and a class library (referenced by the test project) where all Selenium activity is done. (I've updated the question with the project structure.) – Hfrav Apr 3 '17 at 16:14
  • @Hfrav - is the directory structure the same as the project folder structure? I suggest you start by logging the current working directory System.Console.WriteLine(System.IO.Directory.GetCurrentDirectory()); before initializing Chrome. See what the value is when running the ordered tests vs running the test in isolation. – Kate Paulk Apr 3 '17 at 17:25
  • I've updated the question with the results. They're different, for some reason, and I don't know how/why? – Hfrav Apr 3 '17 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.