Best practice when writing an automated test suite - all tests in one file?

Question

I have recently been tasked with incorporating automated testing into my company's development life cycle for one of our products (the product is a web-based software application).

After doing some research into the various automated testing suites available, I decided to use Protractor, as it appears to provide all of the functionality that we will need, and also seems to be well documented & maintained.

I have successfully got Protractor set up and running with our application, and am now starting to design & implement the tests that we will require in order to ensure the functionality & stability of the product.

As things currently stand, all of the tests I have written are in the spec.js file in a 'testing' folder within the project structure, and I have been using comments to document that file, and keep the tests grouped with other similar tests/ tests for the same 'part' of the software.

Since I am relatively new to automated testing, my question is: what is the best practice for developing an automated test suite that will be run on every release of your software? Is it best to define all of the tests within a single file (i.e. the spec.js file I'm currently using), or is it better to have separate files for each part of the software you're testing, in order to keep the tests more structured? What are the reasons for/ against either approach? Or is it simply a case of preference?

Answer 1

Generally as code bases grow you should divide up code into small individual chunks.

For example with application code you could have all the classes and methods in one file - say 12,500 lines long. However that becomes impractical and unweildy to edit. It's better to break code up into individual classes and put each in a separate file.

The same principle applies to tests. You could have all the test code in one place but it is better to break it up. This will also give the opportunity to organize it. For example if there are 7 classes that you are testing against you can create tests for each class in a separate file which is named accordingly. This now provides structure and organization for the test code to make changes and maintenance easier.

Answer 2

Definitely, not in a single file.

It does not scale. Putting tests into a single growing file hurts readability, becomes inconvenient to navigate through and prevents you you from seeing a bigger picture. Remember that the Code is much more often read than written.

We are using Protractor as well and have a large test codebase and I think, we've learned a lot of lessons along the way. What is working for us are the following principles:

• use Page Object pattern to abstract away the elements and blocks of a page building a convenient API for your tests to use

• put page objects to under po directory, specs to under specs organized using inner directories per screen or per feature. Here is our current directory structure:

  /config
  /helpers  (a collection of helper methods and internal testing utilities)
  /po
      /dashboard.po
          index.js
          filters.po.js
      profile.po.js
  /specs
      /dashboard
          /validation
          defaultView.spec.js
      /exportToExcel
          ...   
  

  Note how we are defining some page objects as "node packages" in directories with index.js and "child" page objects. We are also using requirePO and requireHelper functions to ease importing page objects and helpers inside tests, suggested here.

• follow Protractor's Style Guide

• one test file should cover one particular block of functionality - it should be easy to set the describe description in one sentence without using "and" ("and" is usually a sign that you are trying to test two or more different things)
• aim to one assertion per test - this is not always achievable and does not always make sense, but most of the times this helps to avoid testing too many things in a single test
• if you are nesting describe blocks into each other - this is a strong sign of over-complicating this particular test. See if you are better off creating separate test files. "Flat is better than nested." (from the "Zen of Python")
• we also had success in using jasmine-data-provider that allows to "multiply" and parameterize tests
• enforce Style Guide and other rules with static code analysis (we've been using ESLint, eslint-plugin-jasmine and eslint-plugin-protractor plugins)
• group tests into logical groups with suites

Answer 3

Generally speaking, it depends on the application

For a very simple application where you're only automating a single type of tests, dividing things up into multiple files adds very little value.

For a complex application with multiple segments, where you're automating several different types of tests, it's clearly a good idea to separate things into a sensible file structure.

For applications that are neither very simple nor very complex, separation into multiple files will be less necessary but it helps the understandability and maintainability of the automated tests to do so.

It's also depends on what tools you're using to automate your tests. When you're writing automated tests in a language like Python, it makes sense to create one or more libraries to hold functions that you will use across multiple test cases and/or test types. Testing frameworks like Cucumber, Fitnesse or Robotframework often also have a set of 'best practices' that stem from how the framework was designed.

While there is no 'one size fits all' sort of solution, as a rule of thumb some separation into multiple files is practically always a good idea. What exact separation you should make is up to you and your team to decide since it's highly dependant on your approach.

Answer 4

The way that I've organized automated test solutions is... 1 Project per application, 1 Folder per major area of the application, 1 TestClass file per screen.

Depending on the granularity of your application and amount of tests you could write for it, you might need nested levels of folders that properly reflect the product's granularity. I.e. if an application has a tabstrip with 10 tabs, and each of these tabs contains 10 page navigations, then your project maybe should have 2 levels of folders, rather than a single folder with 100 test files.

If you're going to have 2 or more people writing 100+ automated tests for a single area (i.e. a single page), you might want to look for a way to divide that area up into multiple test files for each logical unit. That would allow each SDET / Automation Intern to work in a separate file, and you could avoid them having to resolve merge conflicts on a daily basis.

Answer 5

Split the tests by feature

It will definitely help to split the tests up into "suites" which test features or functionality that fit together in human usage flow (eg 'login' might test the login page display, authentication, OAuth2 etc. but is really just all under login).

Use a test runner

If your testing framework doesn't allow you to do this (it does) then you can make your own test runner script. Once you have more tests and more complex cases, you'll want a special set of test data known as "fixtures", such as email addresses that don't contain "@" for testing form validation, and login details for test users. A test runner can help add this data and remove it afterwards.

Advanced Testing

It seems like you're doing automated functional testing. I recommend examining other testing tools like Cucumber (a testing language anyone can read & write) and Selenium (which automates real browsers like Chrome, Firefox etc.) which can be used on any web application - Behat uses both of these.

Answer 6

Here is an example of how I have structured the latest python project I have been working on:

Unfortunately, python doesn't have inbuilt method grouping like ruby Rspec. So I just group unit tests on which particular method I am unit testing:

hope this helps

Answer 7

Splitting into sources files has great benefits, but is most often something you can postpone until you really need it. It adds extra complexity which might not really add value at this moment.

Good reasons to split into multiple files:

• Multiple engineers: Working with multiple test automation engineers on the same project. This limits merge conflicts.
• Long classes: Classes longer than 200-400 lines seem good candidates to split.
• Lots of tests: Tests covering different applications areas seems good candidates to split.
• Running tests in parallel: It is way easier to run tests spread around classes/files in parallel.

Good reasons to have a single file:

• Limited number of tests: In an open-source project of mine I have just 14 end-to-end tests, seems to fit fine in one file. Probably will split it when it becomes to big.
• Central: Simple central location can be handy for smaller projects

Good practise:

I like to keep it as simple as possible from the beginning. I apply KISS, YAGNI and DRY principles. Therefor I always start my testing in a single file. The next step is to move the test steps into simple pageObjects, as tests start repeating the same code. Then I might start splitting the tests into multiple files.

Bad practise:

What I certainly not do is copy over a complicated file structure from a previous project into a new fresh project. It sounds easy, but possibly adds a lot of overhead and learning curve for other team members. Starting simple helps everyone in the team to enjoying writing tests.