I am well aware that there are great services which allows us to use hundreds of available browsers, devices, AI supported false positive prevention and many other things.

But when it comes to build a Continuous Integration workflow I see a lot of developers are using in-house solutions or open source tools.

Particularly I have had hard time to use other SaaS companies available on the market for huge sets of regression and functional tests.

What are your experiences on the subject ? Especially for regression and functional testing.

  • "What are your experiences on the subject?" is archetypal opinion-based answer, and I expect your question be closed as such. Give us some context to your problem so answer will be better than "it depends". BTW Sass is style-sheet language, do you mean SaS (Software as Service)? Commented Dec 18, 2017 at 17:56
  • @PeterMasiar almost: Software as a Service (SaaS) ;)
    – beatngu13
    Commented Dec 18, 2017 at 18:11
  • @beatngu13 - Yeah, SaaS would ring a bell, SASS did not. "SASS Services" uses "Services" 3 times. Over-serviced, IMHO :-) Commented Dec 18, 2017 at 18:41
  • Sorry for typo, I was in a little bit hurry and tried to post question without a double check. Thanks for pointing this out and I'm glad that you guys had fun. @PeterMasiar
    – mirza
    Commented Dec 19, 2017 at 7:53

1 Answer 1


One of the disadvantages of using SaaS for regression testing is that you need to allow outside organization access to your pre-production test environment. In most organizations, it has significant part of real production information, which can be security risk.

Even if you de-identify user data (scramble addresses, phones, names etc) in pre-production testing (which might be significant and repeated effort), industrious industrial spy agent in SaaS company can ferret out significant knowledge about your business. Hence the dislike for it.

Some SaaS companies provide also "on-premises" testing cloud, which avoids such concerns (for added price of course). Or, companies just build the in-house test cloud themselves, if they have the expertise.

In-house solution using open-source might seem less expensive, but only if you do not factor in the cost of your own in-house experts learning the tools and managing the solution. Usually, buying something mass-produced by experts is cheaper than building it in-house from scratch (if mass-produced version can reasonably cover your needs).

Best of both worlds seems to be to buy expert "on-premises" solution build on open-source software, to avoid vendor lock-in, and learn from experts how to run it.

Cost (and feasibility of buy/hire vs sourcing in-house) also depends on the job market and cost of living: it might be more expensive (per hour) to hire admin in USA than in a developing country. This way, open source/cloud computing can shift opportunities to countries with lover cost of living. So it is kind of funny that many of us spend our own free time to teach others (for free) to do what we do for living in countries with high living costs...

  • 1
    Good answer! Plus: SaaS may be more expensive (compared to an on-premise or custom solution) for bigger companies respectively dev/QA teams, whereas smaller ones can save money—and quite often they simply don't have the resources for alternatives.
    – beatngu13
    Commented Dec 18, 2017 at 19:30
  • 1
    I added a blurb about "saving" money by using in-house solution. Commented Dec 18, 2017 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.