How does the DataSource attribute on a test run the test multiple times?

Question

This is going to be a bit long/wordy/verbose, but I've been looking at it for a while and I think that I'm bad at explaining things because people seem to get confused about what I'm asking.

You know how, when you add the [DataSource(...)] attribute to a test, the test will perform multiple runs, and then show as multiple executions in the test runner (MTM or VS)? I need to reproduce that kind of behaviour, while still being able to use a datasource.

Maybe showing what I'm trying to do might make more sense.

Let's say I am testing a webservice that has an Add(int,int) method and I want to test a number of different adds, so I have a TestMethod that looks like this:

[TestMethod]
public void CanAdd()
{
    var clientProxy = this.getClientProxy();
    Assert.AreEqual(2, clientProxy.Add(1, 1));
}

now, I add the datafile, a csv that looks like this:

arg1,arg2,expected
1,1,2
0,1,1
-1,1,0

I import that into my test,

[TestMethod]
[DataSource(...)]
[DeploymentItem(...)]
public void CanAdd()
{
    var clientProxy = this.getClientProxy();
    Assert.AreEqual(
        (int)TestContext.DataRow["expected"], 
        clientProxy.Add(
            (int)TestContext.DataRow["arg1"], 
            (int)TestContext.DataRow["arg2"]));
}

Now, I'll get 3 runs of my test, one for each data row.

This all works fine... but the problem is that I must test multiple (configurable) proxy locations. And it has to work from a test run kicked off from MTM.

What it would ideally look like would be:

[TestMethod]
[DataSource(...)]
[DeploymentItem(...)]
public void CanAdd()
{
    /*  in this scenario, getClientProxy would update the client proxy 
        with the address of the deployed location of the service
    */
    var clientProxy = this.getClientProxy(
        TestContext.DataRow["deployedLocation"].ToString()); 

    Assert.AreEqual(
        (int)TestContext.DataRow["expected"], 
        clientProxy.Add(
            (int)TestContext.DataRow["arg1"], 
            (int)TestContext.DataRow["arg2"]));
}

And, if I had 2 deployed services (which are identical outside of their physical location), then I'd have 6 test runs.

This doesn't work, and here's why:

• Can't put the location of the service into the datasource because:
  • It'll be different depending on who's running the test(s)
  • If each person running the test(s) manually changed the datasource, they'd have to do a checkin and a rebuild.
• Can't have another datasource attribute

Basically, I want to have a data driven test, where the data would be like this:

location,arg1,arg2,expected
http://www.location1.com/service.svc,1,1,2
http://www.location1.com/service.svc,0,1,1
http://www.location1.com/service.svc,-1,1,0
http://www.location2.com/service.svc,1,1,2
http://www.location2.com/service.svc,0,1,1
http://www.location2.com/service.svc,-1,1,0

except that I can't actually put the location into the datasource.

Has anyone come across this kind of behaviour? Any ideas on a workaround? I'm currently looking at creating an aspect that will call the testmethod for n locations passed into the attribute (similar to this) but I don't know if it'll work as I expect.

Answer 1

I was able to accomplish this by writing a custom TestClassExtensionAttribute and then calling a custom ITestMethodInvoker from the TestExtensionExecution.

I answered a similar question on SO about this: https://stackoverflow.com/a/14673329/573218

I have posted the code on GitHub and am working on cleaning it up a bit: https://github.com/johnkoerner/MSTestLooper

Answer 2

If I understand you situation correctly you have a data-driven test, and you want to be able to run that data-driven test using the arguments in the 'datafile' against one or more services with varying locations.

One approach may be to read in a file that lists the desired services (location) at runtime into an array, then loop through each element (service) calling your CanAdd() method.