What are anti-patterns in test automation?

Question

anti-pattern:

There must be at least two key elements present to formally distinguish an actual anti-pattern from a simple bad habit, bad practice, or bad idea:

• A commonly used process, structure or pattern of action that despite initially appearing to be an appropriate and effective response to a problem, typically has more bad consequences than beneficial results, and
• A good alternative solution exists that is documented, repeatable and proven to be effective.

I have recently encountered different anti-patterns in test automation, that make reading, reusing and maintaining tests hard.

A similar catalog has been created for anti-patterns in unit testing, but automation of end-to-end tests is different in nature from unit-testing. First, some patterns that are anti-patterns in unit testing, might be acceptable in end-to-end test automation. For instance, adding a new assertion to existing test instead of creating a new test case (The Free Ride / Piggyback design pattern) might be acceptable because setup in end-to-end environment takes usually more time. Second, there are anti-patterns specific for end-to-end test automation, e.g.:

• Test data too much coupled with SUT DB
• Environment configuration hardcoded in tests

Can you suggest others?

Answer 1

Unit and system test automation is different, but at least a few of the unit test anti-patterns apply, such as concentration on happy-path scenarios. Thanks for including that link!

In the automation I have implemented, I was forced to implement GUI automation due to the structure of the legacy Java client-server systems I was testing. However, these are also applicable to web automation tools like Selenium that act at the user interface. Note that this applies to keyboard/mouse interfaces, touch interfaces are a different animal entirely and would require their own evaluation.

I will start with an obvious anti-pattern: Dependence on Record and Playback The alternative that I have implemented on all of my automation is to use recorded actions to obtain the structure of the GUI elements, then parameterize them into a function that searches the application interface structure to locate the object. This makes the GUI automation resilient to structural changes to the user interface (which happens often).

Making Intermittent Bugs Low Priority This will show up as "glitches" that a user or a manual tester would overcome quickly and often ignore. However, unstable or unpredictable operation is the bane of automated tests. Even if they are EBM ("every blue moon") intermittent defects, they would still prevent most automation from being implemented without a considerable number of restarts.

Here is my favorite: Ignoring Accessibility Development Standards These standards were developed to allow persons with disabilities access to software applications. Not only is it the right thing to do, but implementation of these standards improves the accessibility for all users. A little-known benefit of adopting these standards is the dramatic improvement in the testability of the application, both for manual and automated approaches. One aspect of these standards is especially important: making the application interface readable by external screen reader applications (such as Jaws). Not implementing accessibility standards forces you to implement "blind" automation, which is essentially the Record and Playback approach discussed above.

Answer 2

Not testing in Isolation Anti-Pattern

In order to run a test in isolation you often have to bootstrap one or multiple systems, configure these systems, setup test data, archive test artefacts and destroy the setup after the test run.

Easier is to just run the end-to-end tests against a some central system test database and shared server resources, with multiple accounts per test situation.

• Dependencies on (production) infrastructure to run test suite
• Parallel test runs could interfere each other
• Test data is corrupted and needs manual fixing every now and then
• Easier for test to become depended on each other

Better is to use a CI to use something like Vagrant to bootstrap a clean Test environment foreach individual test run.

Answer 3

PageObject littered with Assertion anti-pattern.

Problem: Assertions are part of PageObject class

def varify_edit_link_presence
  //BAD: Assertion in page Object. 
  has_link?("edit").should be true
End

Solution: PageObject just provides status of the element to the caller. The caller itself will verify the status:

def has_edit_link?
   has_link?("edit")
End

• Duplication makes life easy.
• Long live dead code!!!!!
• Rescue strategies.

This is an excerpt from the slide-deck on Agile Testing Anti-Patterns and Rescue Strategies. For more on this, you should look into that presentation.

Answer 4

I'll take a stab at a few:

1. Including business logic at the test case level instead of building a business layer. Existing pattern: Page Object Pattern
2. Declaring page elements inline in test cases with locator information instead of in the business layer. Existing pattern: Page Object Pattern
3. Sleeping for arbitrary amounts of time. Existing pattern: polling, explicit/implicit waits.

Answer 5

Anti-patterns in Test Automation

1. Using implicit waits
2. Using Recorded playback
3. Using brittle, layout based selectors
4. Not isolating test data between tests
5. Not consistently using Page Objects
6. Tests that only have negative assertions
7. Using assertions mid-way through a test
8. Repeating parts of tests due to the same setup
9. Not considering the Sad path of user entering invalid data
10. Forcing all tests to go through all pages in complex workflows
11. Assuming that test automation cover accessibility and usability
12. Using cryptic names and acronyms in test and suite descriptions
13. Not considering optional workflows for both happy and sad flows
14. Only testing 1 browser and assuming that other browsers work fine
15. Not using Page Helpers to DRY up and name common code sections
16. Using general assertions like have_css over declarative ones like have_button
17. Using authenticate / authorization in multiple tests leading to massive test failures
18. Bullet Points that gradually get longer as you scan your eyes down the full page of them

I would say that I've made every mistake in this list when starting out, before I knew better so I think they all meet your criteria of 'seemed ok initially'

Answer 6

Different Stack Anti-Pattern

The automated tests (also test frameworks) are implemented using a different software stack then the system under test (SUT) uses. This will make it harder for developers to maintain the tests. I have seen web development teams needing to learn Java as an extra language, because the test team decided to us a Java testing framework without having the app developers involved. The testers where familiar with Java and not the PHP/JavaScript stack use by the other teams.

• Demotivates developers to write, fix and or extend tests
• Harder to maintain for a team
• Extra setup to run the test local (Tools, Runtime, etc)

Answer 7

White Box Testing In The Absence of Functional Requirements

Sometimes, lacking requirements, we read the system code to attempt to discern what it was supposed to do, and then write tests based on that knowledge. This can arise when someone is asked to automate tests for a legacy system.

There is a place for white box testing, but it is not a substitute for actually knowing what the system is supposed to do. White box testing without a knowledge of actual requirements also risks enshrining system logic errors in your tests.

An alternative (aside from getting some requirements) is to use comparator tests. You run two versions of a system using the same inputs and then compare the outputs. This kind of test is less precise than a functional test; if the outputs do not match, you know something has changed, but you do not know whether the change represents a new bug, a correct bug fix or intentional feature change.