2

I'm currently working in a software house and we need to assure quality in different configurations. Some example of these configurations are:

  • Browser - B1, B2, B3 or B4
  • Operative System - OS1, OS2 or OS3
  • Database - DB1, DB2 or DB3

That is a simplification of the current number of combinations we have to test, but in this simplification we have 4 * 3 * 3 = 36 possible combinations, for each we need one different test environment. The current approach is to run every test in every combination, so each test will run in all different test environments.

We have thousands of tests that need to run continuously and running each test in so many environments takes too much time and most of the tests don't add value to run in all configurations. If I'm running a browser test, I don't care what database the server has installed, it’s enough to run the test in all different browsers (4 in this example instead of 36).

I'm trying to distribute the tests by all the different test environments, so that a tests is only selected to run in more than one environment if strictly necessary to ensure quality. Can you please tell me how any companies are handling with this sort of problems?

EDIT: To complement the information to my question, most of my tests aren't write in Selenium, I also want to distribute unit and integration tests to reduce the feedback loop. I already can assign the tests that I want to run on one machine (independently of the language), I'm trying to find a way to distribute the tests in the best way possible and I have cases that I need to run the tests in more than one test environment. There are other companies trying to solve the "fast feedback" problem with this approach? Or academic research?

Improve this question

3 Answers 3

Reset to default
1

If I am correctly understanding your problem- then Selenium Grid would be a good solution. It allows you to

  • Distribute your tests across multiple machines for execution (parallel processing).
  • You can perform multi-browser and multi-os testing of your web application.

This is what a simple pictorial representation would look like. enter image description here

You can also take help of third party services like BrowserStack or SauceLabs. You can go through the comparison between both services here.

Read about Selenium Grid, it's set up and an easy test script here.

Improve this answer
1
  • 1
    Selenium Grid only runs the test in one node, in my case I have tests that I want to run in more than one node (or even in all) and others that it's good enough to run in only one. I'm not expecting a tool as answer, but instead one approach or a similar problem that already have been solved by another company.
    – Miguel Afonso
    Nov 12, 2015 at 18:00
0

One school of thought is to make sure you test every pair of configurations at least once. This isn't provably better than testing every combination, but it does reduce the number of combinations. See https://sqa.stackexchange.com/search?q=pairwise-testing for more questions and answers about that technique.

When I'm faced with this problem, I make some judgement calls on which combinations are most likely to have problems. You will figure out some of that from experience.

You might also try reducing the dimensionality of your configuration space by performing more specific tests on smaller configuration subspaces. For example, instead of running every test on every combination of browser, DB, OS, and fifty other variables, do this:

  • run some more specific tests that only focus on DB and OS, and
  • run your other tests on the browsers and the fifty other variables plus just one DB and OS
Improve this answer
3
  • Thanks for the answer, that's the problem that I'm trying to solve! I'm trying to find a way to distribute automatically the tests according with some heuristics, heuristics than can be automatically calculated and with the minimum human effort involved. Pairwise testing seems a good heuristic, I'm already looking to it to see what can I use to distribute the tests, do you know other options?
    – Miguel Afonso
    Nov 13, 2015 at 17:38
  • To be clear, Pairwise is a way to reduce how many combinations you test. It does not help or hinder distributing the tests.
    – user246
    Nov 14, 2015 at 19:11
  • Sorry for my last answer, I wasn't very clear. I understood what is Pairwise Testing and how it's used to reduce the number of combinations needed to test one system. I will try to use it to reduce the number of possible environments where each test can run (I have different requirements for each test) and then distribute the tests to run in the environments returned by the "Pairwise Testing". I'm still trying to find one approach more greedy than pairwise , or at least one that I can assign different weights between combinations more probable than others. Do you know any?
    – Miguel Afonso
    Nov 16, 2015 at 10:33
0

Is it a requirement having everything (browser, db and os) installed in the same machine? If not, let me propose what we do in my company because in the real world there are really much more than 36 combinations (service packs for os, hotfixes for db, etc.). Based on our internal acceptance criteria, we would install

  • all of the browsers in one machine (in theory even in the same machine of the os),
  • then we would have one machine for each os (in your case three), and
  • only one machine with the databases (again three databases in your case).

In the end you have 1+3+1 = 5 machines (i suppose you have a cluster with VMWare or HyperV) that can stay turned on at the same time, and you can perform the automatic tests avoiding the combinations between browsers and db that are not useful for the aim of the test.

Improve this answer
2
  • Thanks for the answer! My main goal is not to reduce the number of machines, but instead distribute the tests by all machines in the best possible way. My main concern is the feedback time that we need to wait for receive test results. I don't want to change the infrastructure (we have our motives to have it like that), I just want to use it "full power" to run the tests faster. But in your case you have all the tests running in the same machine? Or all the browsers tests running in only one machine?
    – Miguel Afonso
    Nov 16, 2015 at 9:57
  • Thank you for the question. Our application is composed by several independent and highly distributed services talking each other on needed basis. As a consequence it does not matter where are the testing tools, of course unit tests run with build. For browser testing we have one machine per operating system.
    – Luca Giuffrida
    Nov 17, 2015 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.