3

So im in a bit of a conundrum. I have a Rails project which i've written Automated tests for in Capybara using ChromeDriver/Selenium. It works great, but is quite long.

I also have headless chrome driver "set up" for this project....but honestly it just doesn't work great. Headless chrome driver is a bit early and has gone through a lot of changes but will probably inevitably be the "go-to" headless driver of choice probably taking over PhantomJS.

This being said, i'd typically use poltergeist to run headless because it works well...but at the same time phantomjs will probably eventually lose out to headless chrome, and not to mention a lot of the issues we've seen with this particular app have been browser related.

So that being said: Is it crazy to run a browser based UI Automation on a CI Server? Obviously it will be slower but def. more accurate. Is this common practice? I've only been doing automation for a little over a year but not a whole lot via CI.

Thanks!

Improve this question

1 Answer 1

Reset to default
4

The short answer: No, it is not crazy, as long as you take the proper precautions.

The longer answer:

There's nothing inherently wrong with running UI automation from a CI server. That said, there are a number of factors you need to take into account.

  • Don't block CI - You should be limiting the continuous run suites to a basic smoke test run on the CI server unless you are farming your UI test suites out to other machines. This is completely doable, as long as you configure your UI automation correctly.
  • Configure the run systems carefully - The machines running UI automation must be able to be configured to automatically log on and have the screensaver disabled. Depending on how your UI systems are configured, you may need to ensure that the user running the tests is an administrator on the systems (alternatively, have a set of containers that are configured to meet your needs and spin them up as needed).
  • Maintain a UI automation subnet - In order to limit the security risks that go with needing to keep a set of virtual systems that auto log on and never lock, a dedicated subnet that is not accessible to anyone outside a very limited set of users is also helpful. If you're using containers, you may still need to do this and set the containers to use the subnet IP range.
  • Choose your app version carefully - For an extensive UI automation suite, I prefer to have the general regression tied to a non-production branch in order to catch regressions earlier, and keep smoke tests pointed to production or staging branch environments. Dedicated mirrors of production also help to avoid impacting live application data and performance.

As long as you keep the UI automation performed directly on the CI server as light as possible and distribute the longer running tests in a way that allows you to catch regression issues quickly without compromising the overall application quality, there's nothing wrong with configuring your CI system to run UI automation.

Improve this answer
5
  • Ideally due to the amount of time these tests take to run, I wouldn't want to block merges (IE automation tests every merge). Since we tend to work pretty fast here and merges get done all day for small fixes. We have Unit/System Level integration tests that run with those (that take <2-3 minutes) The UI Suite takes a LOT longer, I feel like putting it on a cron job timing using gitlab's CI (Which is what were using). Everything right now is being tested against a staging DB (Which I have issues there, but thats a separate problem).
    – Mercfh
    Commented Oct 4, 2017 at 15:48
  • 1
    That definitely sounds like a chron job to run at some set time on separate servers is the way to go. It's not hard to set up, but it is fiddly.
    – Kate Paulk
    Commented Oct 4, 2017 at 17:12
  • I'd love to use headless testing, but I see 2 issues: A: It doesn't capture the UI related bugs and... B: Headless chrome is probably going to be the most used in a little while, I see phantomjs going out. However headless chrome just does not work with a lot of things. And would be a major rework (which may be worth it at some point)
    – Mercfh
    Commented Oct 4, 2017 at 17:40
  • @Mercfh - Agreed. Headless would be better for a lot of things but it won't catch rendering problems and with phantomjs no longer being updated and headless chrome still very immature as a technology, it won't be an option for some time.
    – Kate Paulk
    Commented Oct 4, 2017 at 18:56
  • Ok whew glad someone else agrees with me ha!
    – Mercfh
    Commented Oct 4, 2017 at 19:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.