0

I need to manage executing all unit tests on build server. Is there any idea about when is proper to run all of the unit tests on the build server. There are too many of them, and it will take hours to run completely. I have a few ideas about it, but I am not sure of them:

  • Nightly
  • When release for QA team
  • When release for Customer
Improve this question

2 Answers 2

Reset to default
2

Unit tests which are run for hours are bad unit-tests. Since they have to test the pieces of code and mock all the real interfaces. So, basically they should be restricted only with your CPU, RAM capacity.

You probably mean that you use unit-test framework to run your integration tests or so.

If this is really the case I would recommend to break the whole set into the bunches aggregating the tests with the identical priority and run

  • the top priority on each build
  • others - overnight

Above options seem to be enough since your builds might be handed over to QA intraday so spending hours to test them with the whole set is not effective. Customer release is also not that unexpected thing so you can run your set several nights before the relase.

Improve this answer
1
  • Yes, the tests are actually integration tests implemented by unit-test framework. Each run in 1 second in average. But there are too many of them. 20k is the total count. It is not always needed to run all of them at once. It depends on the project has committed changes. We filter the tests. But when there is a commit for framework project, we should execute all tests though.
    – Polymorphic
    Commented Dec 13, 2017 at 15:23
1

Best practice is to run all of your tests for the Nightly scheduled runs and before you signoff/ship to customers.

You will be running your own manual tests, or automation tests for a specific feature/code change when it is delivered to QA. Ideally, you have some kind of test plan, formal or in your head, that you can follow for testing changes related to a feature/bug fix. Running a full test pass with every build will just take too much time. You have the nightly run that will catch anything else.

If your automation is being kicked off with every build by a CI/CD system, then you want to run a kind of bare minimum of tests to ensure viability. I would say these are BVT tests, but different people have different gauges of what a BVT test is. You really just want enough to be able to tell that the software under test is minimally functional. For a website, a minimum test would be something like making sure you can deploy the site and load the homepage. A web service would be deploying it and making sure you can perform the most basic query. Optionally, if you have a robust branching system then you could have feature branches run the "BVT" test and trunk runs the full gamut. It's all really up to you, your team, and what you agree to tolerate.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.