0

I have an automated test suite of NUnit Selenium tests written in C#. I am trying to test for my test suite's robustness by running my entire suite say 50 times and viewing an amalgamation of the results to see which tests are flaky or brittle. Is there a way to do this automatically instead of either clicking run 50 times and keeping track of the results or setting Jenkins or TeamCity to run every 30 minutes? I'm thinking something like NUnit's [Repeat] attribute, but for the whole test suite.

Using Git, every time I push a new test or tests to my develop branch, I would love for this "robustness test" to run to effectively stress test my automated tests. If this robustness test passes, I would then be comfortable pushing my newest tests to the master branch.

6
  • What stops u from using the solutions you mentioned ? What is the actual question ?
    – PDHide
    Jan 7, 2020 at 7:56
  • In Jenkins test report could be stored with build number so that your previous reports won't be over written . And you could debug to see if the tests are false positive. Also, remember that the focus of test automation is not to pass all the tests but to find defects . So do proper investigation on why the tests fails , is it the issue with product or the script itself ?
    – PDHide
    Jan 7, 2020 at 8:00
  • How does repeating the same operation a number of times indicate "robustness"? Isn't doing the same thing expecting different results the definition of madness? Jan 7, 2020 at 13:18
  • @PDHide My actual question is how to run the test suite x times so I can see which tests are flaky or brittle. I have a suite of tests that sometimes passes 100%, so the test failures when they occur are not due to the product, but the flakiness of the test itself. @ JoãoFarias I am repeating the operation a number of times to figure out how robust my tests are, not the system under test.
    – deasa
    Jan 7, 2020 at 17:31
  • @deasa what was the drawback you found by using the solutions you already mentioned in the question itself?
    – PDHide
    Jan 7, 2020 at 17:34

4 Answers 4

1

You can start NUnit from the command-line; https://github.com/nunit/docs/wiki/Console-Command-Line

Write a small console application or powershell script that loops it 50 times and copies the output after each run. Instead of copying the output you could also log fails into a single logfile, if its empty all tests passed 50 times.

2
1

Modify the test itself to take an extra param number_of_runs

If set, within the test itself, run the steps of that test x times, e.,g. 50 times.

Try to avoid a solution that is outside the framework and involves changing the way tests are run or called.

6
  • Yes, every flaky method needs it. Still, I think its a compromise that is better than two sets of tests or modifying the run. There may well be better approaches though, especially a global approach Jan 7, 2020 at 17:00
  • It gets tricky. One of the problems with re-runs globally is that when either a) you are writing a new test or b) a test really fails, that retries and retests will actually then slow you down at that point. At a minimum u need a way to turn off retries during development of the test. Jan 7, 2020 at 17:02
  • I've addressed this with flags for environment and then test level flag for which tests to retry (including how many times). I have a global default (for the flakies) of 3 and I override it with less (i.e. 2) or more (e.g. 5) as needed. Jan 7, 2020 at 17:03
  • I am trying to prove that my tests have less than a 2% failure rate due to brittleness by forcing all tests on the develop branch to run 50 times. If they pass all 50 times, they are sufficiently stable to push to the master branch. Like you noticed, if I'm writing a new test, I don't want it to run 50 times, so I'd need a way to turn off retries. I'm thinking maybe a better way is not having a local-to-code way of re-running the tests, but managing the re-runs externally, something like Niels suggested or somethink like managing it in Jenkins or TeamCity.
    – deasa
    Jan 7, 2020 at 17:51
  • Is the failure rate important, or is it the fact that a second (or third) run of the same test passes? Or maybe both? Not sure. Whether it's 1% or 10% that fail, it's the amount that fail twice perhaps that I am interested. Not sure if that is the same as your 2% of initial (?) failures. Jan 7, 2020 at 18:24
1

Think of what are you trying to achieve and where and what are your tests and environment.

Locally looping the test suite using a script will test the entire suite but might miss resource allocation on the CI server, repeating test suite using a parameter might miss setup or teardown issues. Personally I would try to run on an environment as close as possible to your real test environment even on the expense of some cumbersomeness, if I understand your setup correctly a loop on your CI would be best.

Then think of how are you going to evaluate robustness, do you want to test your test code (C# Selenium) or the entire environment ?

For example do you expect 100% pass rate or can you allow failure due to external resources like an adware box failing to load or failing to connect to a third party telemetry collection service, assuming those failure can be identified and classified as "external failure"

2
  • Hmmm these are all excellent questions that I hadn't considered. I think you're right that I should definitely take into account failures that are due to the environment, not just failures that are inherent in my test code. Like you suggested, I'm thinking I will put more effort into figuring out how to accomplish this on my CI. Do you have any suggestions on how to accomplish this using TeamCity or Jenkins?
    – deasa
    Jan 7, 2020 at 17:57
  • Some ideas here
    – Rsf
    Jan 8, 2020 at 7:42
0

I found a hack-y workaround way to run my entire suite over and over again:

In TeamCity for the project in question, I added a build trigger that will trigger a new build of the project (i.e. will re-run the tests) once the project has completed building.

This results in an infinite loop, so I'll have to manually stop a build mid-build (again, this is hack-y), but this way I'll be able to run my tests on the environment like @rsf suggested and TeamCity will preserve all the history of the test runs.

I'll use this hack-y way for now, but am open to suggestions on how to make this better.

Thank you everyone!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.