How should I set up a test environment before automated tests are run?

Question

Most of our apps work against a set of files (files that have to be analyzed, backed up, converted etc.), so before running an automated test, the system running the application needs to be in a state where certain files are in the right place.

I would call this the "test environment", but I'm not sure if that is the correct term.

I'm trialling some test automation tools, and neither of them have a features to easily copy files around before running a test, or suite of tests.

I could write a script to run just before the tests are run, but it seems like this should be dealt with by the automation tool.

What is the normal way of setting up a "test environment"?

Answer 1

Generally you should think of tests as doing

1. Setup
2. Execution
   Given X, When Y happens, Then expect Z
3. Teardown

This actually occurs at two (or more) levels:

• Entire test suite
  Certain files, constants, database commands, etc. need to be run before the entire test suite. Often there is no 'tear-down' in this area.

• Each test
  Each test should also have the three stages of setup, execute, teardown. One of the primary items here is frequently the database. Initially it should be empty. Then the test suite starts and it is seeded with common data, then an individual test runs and it is populated with specific set for that test. After the test runs it is important that the db transaction is rolled back or that the db is empty again (if no whole suite seed data).

• Other. There may also be groupings of tests that require specific setup, for example a specific service to be running.

So that's the big picture. For the gritty details of what procedures to run in what cases, it is really going to depend on too many factors to effectively answer here. These include your: business, development environment, datastores, multi-service architecture, existing test infrastructure, test approach, languages of choice, the list is very long. If you have more specifics you could use them for more specific questions.

As for your script, some tools will alleviate the need for custom scripts, others will still require it. There's nothing wrong per-se with scripts however its generally best to try and reduce the number of components and the integration, knowledge, maintenance, etc. they require.

Continuous integration tools that will run your tests in the cloud, such as as Jenkins or CircleCI can be expected to have the capability to run scripts and setup tasks.

Answer 2

I prefer to use separate tools for deployment/configuration and running tests, because a tool that focuses on one thing will do better than a tool that tries to do everything.

There are innumerable tools for automating deployments, where a deployment includes installing binaries and configuration files. Some popular ones include Puppet and Ansible.

For some problems, it can help to run the test within a prepackaged, isolated environment, either with virtual machines (e.g. using VMWare or VirtualBox) or containers (e.g. using Docker).

Answer 3

There are several tools available over the web but my favourite one is Octopus Deploy, because after a relative simple configuration it can deploy your build, setting up automatically your testing environment.

The configuration can be done through a user friendly interface and, it's important at least for me, it's free for a limited number of "tentacles" (an agent to be installed on every machine that you plan to deploy software to).

I really appreciate also the fact that the always annoying setup related to configuration files can be totally automated. Please note that it is not a testing framework (it does not allow to run tests), it allows only to perform an automated deployment.

Hope it helps!

Answer 4

Maybe setting up a VM of your environment is a solution you can look into. So as part of running your tests, you spin up a VM (which has everything where it should be) then run your suite of tests within it.

Answer 5

I'm struggling with the same problem at the moment, but the solution is somewhat dependent on your specific context. For "slow running" tests you can use puppet/chef and friends, they are classic tools to handle environment configuration but they are not meant to be used in test runners. Other solutions I have heard of, especially for python environments, are using a preconfigured virtualenv, installing from a local pypi server using 'pip install requirements.txt'.

Answer 6

Automated tools can do what their designers anticipated.

Your own custom scripts can do what you need.

So I prefer a tool which can simply call my own scripts, like Jenkins.

Answer 7

I've spent a lot of time working with virtualization, Docker and automation recently. One of the biggest challenges has been to get a good testing environment internally, without having to pay exorbitant amounts of money to use services like Heroku. I knew there had to be a better, more affordable alternative, and I was determined to find it.

I've used just about all of the CI services available, including Jenkins, Circle CI, Codeship and Travis. I learned that each of these services has their own quirks. For example: hard to install dependencies, Selenium tests, required infrastructure services, build limits, etc. This is why I have grown to love Drone. I can run my test suite in a clean docker image every time, cache dependencies (just like Heroku does), and run a deploy action if the test build succeeds.

I am using Drone for the continuous integration, and Dokku for the PaaS. Drone is built using Go and utilizes Docker. It can be run inside a container itself with very little configuration.

Drone is a Continuous Integration platform built on container technology. Every build is executed inside an ephemeral Docker container, giving me complete control over my build environment with guaranteed isolation.

Drone's integration with Docker means it can support a huge number of languages including PHP, Node, Ruby, Go, and Python, to name a few. Each test will spawn a new container based off of specified images from the Docker Public Registry. You can even make your own to fit your specific application if needed.

I've coupled Drone with Dokku, a simple Heroku like PaaS built on top of Docker. Using the Github flow with this setup allows automatic staging of all feature branches that pass their tests.

With one simple command:  git push dokku master, as long as my build passes in Drone, the code is deployed in a fresh Docker container. Dokku builds the app on a subdomain of the Dokku host, and it's automatically deployed.