I am working on a testautomation project using .NET Core 3.1 and Specflow 3.5 using the Specflow+ Runner. I am currently running the testautomation from Azure Devops using the VSTest task (see YAML code below for VSTest step)

- task: VSTest@2
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
    searchFolder: '$(System.DefaultWorkingDirectory)'
    vsTestVersion: '16.0'
    runSettingsFile: '.\testAutomation.runsettings'
    rerunFailedTests: false

I recently migrated from the NUnit test adapter to the Specflow Runner since this is the recommended test adapter when testing using an Azure DevOps server, as is mentioned here

Some of our tests are susceptible to timing errors which is why we have setup a retry count in the srprofile like so.

<Execution stopAfterFailures="15" retryCount="2" testThreadCount="2" testSchedulingMode="Sequential" />

However when looking at the results in the test window of the pipeline in Azure DevOps the retries show up as unique tests instead of retries of the same scenario, as was the case when using the Nunit testadapter.

Furthermore the integration of the VSTest task with the Specflow+ Runner works very poorly in general. For instance, setting the retry count in the VSTest task will cause the entire test suite to be rerun if one test does not pass.

My question here is if there is some sort of configuration file of sorts that I am missing that will allow for smoother integration between the Specflow+ Runner and the Azure DevOps VSTest task or whether I perhaps should be using a different task? If so I would dearly like to know. Preferably I would do all my configuration and test analyses in the DevOps environment so a good integration is key.

I would love to hear any recommendations.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.