Are there specific coding guidelines for test automation?

Question

Details

Currently we are trying to find more ways to write better code. Of course, this does not exclude our automated test cases. We are already testing in the dev environment with SonarCube and WhiteSource to find possible errors in branches or in the master source.

Task:

We also want to implement code quality guidelines for the test.

Question:

• What are the code quality guidelines for testing?
• Do they differ from the general code quality rules for Java/Python?

Answer 1

Summary of Code Quality Guidelines for Automation Code

• Prioritize english readability for descriptions and code
• Each test has an assertion at the end of the test
• Consider mocking and stubbing dependencies
• Each test should test one and only one thing
• Use linting and include test specific rules
• Avoid optimizations for performance
• Devise a test data strategy
• Tests don't share state
• Test the tests

Yes, the requirements for code quality in application and automation code are different.
While some factors are common to both areas, emphasis and priority can be quite different.

I was an application developer for 20 years and have now been an automation engineer for 10 and have learned the following differences:

Application Code.
Generally the key attributes here are performance,say for thousands of users at a time so many optimizations are made in the name of performance. Also modularity, extensibility and being DRY are critical approaches. Multi-threading and memory usage are also critical factors.

Automation Code.
Performance tends not to be an issue. Frequently the code is being run as being used by one user. Readability, while important in Application code becomes super critical in automation code. Automation code can be thought of as the technical specifications - the documentation - for your system. Making sure the documentation is readable, written with good english, potentially to less technical people, is much more important. Another example of a difference is line and file length which can have different characteristics for test code vs automation code.
Also you should figure out your test data strategy. Don't go overboard with test data though. Sometimes hard-coding static data is ok, even if it repeats. Be careful not to sacrifice readability by over using extract method and DRY approaches for data. Be careful about artifacts you create and consider removing them as part of the automation code cleanup.
Tests should not share state meaning they should not depend on each other and should not affect or be affected by other tests or the order that tests run in.
Finally consider testing the tests, for example make sure page objects have no duplicates or orphans

One thing that is common to both areas for quality code is the use of strong linting rules, code grading, etc.

Answer 2

Code is code, regardless if it's for feature code or test code. To me, the only thing that's truly different between feature code and test code is purpose, perspective, and motivations.

That means, yes, you can and should be using SonarCube and WhiteSource on your test automation code. You can and should be using linters here as well. If you have multiple people writing test code, ensure they are all using the same code guidelines rules that are set up in the linter.

By doing this, you are ensuring the test code is keeping a high sense of quality; you're "testing the test code" so to speak.

Also, when you create test code/test automation frameworks, don't you also include dependencies? How do you ensure those dependencies stay up-to-date? By using tools like SonarCube and WhiteSource.

Personally, I always set up, at minimum, a linter on test automation frameworks I create.

Answer 3

One classic resource is Meszaros Gerard's book xUnit Test Patterns: Refactoring Test Code. There is also a website in wiki format for most of the concepts there.

Kent Beck wrote a blog Test Desiterata, where he shows 12 characteristics of great tests. He also made a series of short videos about each characteristic.

Robert Martin has a chapter in the book Clean Craftsmanship: Disciplines, Standards, and Ethics on Test Design, where he discusses the Transformation Priority Premise. If you prefer video format, he has two episodes of the Clean Coders series on this: Test Design and Clean Tests.

Answer 4

Echoing some of the previous answers. Diverse half measures and understanding the context of the test and the code/application/system under test are key. Some non-exhaustive code/implementation specific suggestions below.

• Use clear and intention revealing names, descriptions and assertions
• Use the minimal number of assertions possible per test
• Use robust assertions
• Use established patterns where appropriate
• Be careful when using equivalence matchers (e.g. eql, equal and == in RSpec)
• Use realistic test data (where appropriate)
• Highlight the use of any mocks and stubs
• Be careful when shortcutting or circumventing behaviour
• Ensure that tests cleanup after themselves, particularly when failing

It could be argued that yes the guidelines or heuristics for test automation should differ to some extent as the intent and purpose of the test is different to that of the actual application code. Ideally they should complement the existing general guidelines you are already applying to your application code.

In addition to the above, Richard Bradshaw’s TRIMS heuristic is an excellent starting point for more broader guidance on automation design and quality. ​​

If you're using well established test automation patterns, there might be value in considering pattern specific quality guidelines. For example Angie Jone’s guidance on the Page Object Model.

As an alternative to explicit guidelines, consider applying tools such as Ashley Hunsberger’s Test Suite Canvas or Katrina Clokie’s Test Automation Canvas. Similar to the TRIMS heuristic, these tools can help shift the focus of thinking to high level attributes such as risks and dependencies, rather than purely focusing on implementation specific concerns such as code conventions or individual test framework best practices.

The Association for Software Testing has some guidance on common misconceptions and fallacies related to automation (full disclosure I’ve contributed to some of this content).

For further reading and sources on wider automation topics, I would highly recommend the Ministry of Testing community.

Answer 5

These tips refer to testing javascript using Mocha but other languages and testing frameworks are similar.

The key point is to make it as easy as possible to add tests, both initially and when a bug is found and you want to make sure it never regresses. When it's trivially easy to do, there's no excuse not to test.

1. Create a general way to add test cases rather than ad hoc tests. I do this by making my tests get their input from an array of objects (which contain all needed arguments and the expected result).

2. Name the test using the object being tested to create consistent and informative names for each test.

3. Adding a new test is simple - just add an element to the array with the specified input(s) and expected result.

Example: testing a function cleanStr(s) which a) latinizes characters with diacriticals, e.g. á to a b) lowercases all characters c) replaces unacceptable chars with a space, and d) consolidates all groups of spaces into one space

suite('Testing cleanStr', function() {
var cleanStrArray = [
  {str:'', result: ''},
  {str:'a', result: 'a'},
  {str:'a3', result: 'a '},
  {str:'3a', result: ' a'},
  {str:'3a2', result: ' a '},
  {str:'32a', result: ' a'},
  {str:'3a2 ', result: ' a '},      
  {str:'a    b', result: 'a b'},
  // non-Latin chars
  {str:'Á', result: 'a'},
  {str:'á', result: 'a'},
  {str:'à', result: 'a'},
  {str:'ñ', result: 'n'},
  {str:'Él está en el baño', result: 'el esta en el bano'},
];

// create a name for each test in the array
cleanStrArray.forEach(function(aTest) {
  if (!aTest.testName) aTest.testName = '"' + aTest.str + '" -> "' + aTest.result + '"';
});

// run the tests
cleanStrArray.forEach(function(aTest) {
  test(aTest.testName, function() {
    const newstr = travesty.cleanStr(aTest.str);
    expect(newstr).to.equal(aTest.result);
  });
});

});

Answer 6

Tests and production code serve different purposes. Production code is the thing which must be doing the correct thing in order to be useful. A test suite is the thing which must demonstrate the correctness and document the functionality of the production code in order to be useful. Based on this, there is one thing which is nice to have in production code but which is a must for test code: clarity. Unclear production code can still be useful, but unclear test code is often worse than no test code. Examples:

• A test which asserts the wrong thing looks correct, keeps passing, never does any useful work, misleads the developers, and wastes time. Detecting misleading tests is really important, but can be very difficult. One way to do it would be to check for any tests which never fail during mutation testing. Never failing indicates that no mutation to the production code affects it, so it's probably not testing the production code.
• A test which doesn't say what it's meant to demonstrate (test_foo_is_correct is a favourite) leaves no room to verify whether it's doing the right thing. It also can't be refactored reasonably when the underlying code changes, because the original purpose is often impossible to discover. A useful trick here is to start test names with “should”, such as should replace children when updating instance or should return API response body.

Answer 7

Although there is no silver bullet / one size fits all rule for this, here is a short list to get you started,

• Make sure the code written is properly indented and readable.
• Ensure proper comments to understand the code.
• Use approved and managed code rather than writing new unmanaged code for common tasks.
• Use proper naming conventions for your classes, functions/method and variables. Ensure this is a uniform practice throughout the team.
• Utilize task specific built-in APIs to conduct operating system tasks.
• Do not allow the framework to issue commands directly to the Operating System.
• Use checksums or hashes to verify the integrity of interpreted code, libraries, executables, and configuration files.
• Protect shared variables and resources from inappropriate concurrent access.
• Explicitly initialize all your variables and other data stores, either during declaration or just before the first usage.
• Do not pass user supplied data to any dynamic execution function.
• Restrict users from generating new code or altering existing code. Have role based permission for deploying changes in the code.

You can add more to the list.

Also, the thing to note is all standards and rules that are applicable to development code will also apply to testing/application code.

Answer 8

Like in software, the pareto principles apply to automated tests with 80% of the effort passed on their maintenance.

I think "Maintainability" is the single most important thing to consider first while writing automation code is 'Maintainability'. I think a whole book can be written just considering this one aspect in test automation . It gives further great ideas to consider in guidelines like readability , flexibility and adaptability even on long term upgrade,migration and even portability.

Any code is prone to changes not just automation however UI automation not only can be impacted by functional changes but as well with UI changes.

There are no set rules/guidelines for anything including automation, everything depends on the context - the tech stack, product, team skill set so if one start thinking in their given context , what impacts them now and in near future as well as in distant future in terms of maintainability, they will go a long way.

Also make sure to "review with the team" the automation guidelines once you come up with something to make sure they make sense in your team context , not just long list of borrowed rules.

At the end of the day, they need to be your "own".