I'm currently working in a software house and we need to assure quality in different configurations. Some example of these configurations are:
- Browser - B1, B2, B3 or B4
- Operative System - OS1, OS2 or OS3
- Database - DB1, DB2 or DB3
That is a simplification of the current number of combinations we have to test, but in this simplification we have 4 * 3 * 3 = 36 possible combinations, for each we need one different test environment. The current approach is to run every test in every combination, so each test will run in all different test environments.
We have thousands of tests that need to run continuously and running each test in so many environments takes too much time and most of the tests don't add value to run in all configurations. If I'm running a browser test, I don't care what database the server has installed, it’s enough to run the test in all different browsers (4 in this example instead of 36).
I'm trying to distribute the tests by all the different test environments, so that a tests is only selected to run in more than one environment if strictly necessary to ensure quality. Can you please tell me how any companies are handling with this sort of problems?
EDIT: To complement the information to my question, most of my tests aren't write in Selenium, I also want to distribute unit and integration tests to reduce the feedback loop. I already can assign the tests that I want to run on one machine (independently of the language), I'm trying to find a way to distribute the tests in the best way possible and I have cases that I need to run the tests in more than one test environment. There are other companies trying to solve the "fast feedback" problem with this approach? Or academic research?