Does codeless automation tools work for large test suites?

Question

AI-based codeless automation tools are on the rise. There are plenty of them that even claim to have flexibility and reusability.

Are they good enough for large complicated test suites with many thousands of test cases?
Is strong programming skills not required for automation when using these tools?

Three examples:

Leapwork: https://www.leapwork.com/blog/codeless-test-automation
"codeless selenium with AI maintenance"

Perfecto: https://www.perfecto.io/codeless-automation
"AI-driven test automation creation"

Applitools: https://applitools.com/
"AI powered functional and visual testing"

Answer 1

In many cases, no

The first automation case is easy to write - no page objects, no need to use DRY, etc.

By the 100th test, you need a lot of programming - or codeless automation suites.

The problems really start when you then continue to use the

add or change functionality or bug discovered / fixed
  \|/
write new UI test
  \|/
repeat

The test suite starts to grow and before long takes first hours and then days to run. The number of test cases keeps growing (I've seen tens of thousands), as do the intermittent errors that destroy the confidence of the tests adding value. Multiply test cases by browsers and devices and versions and orientation and the Universe might end. Ultimately the original intent of rapid feedback disappears and QA becomes a separate function that does checking and verification but doesn't directly influence quality in the context of application code creation by developers. Testing gets done for the sake of testing and quality does not improve long term.

It also becomes apparent that you should be following the principles behind the Agile Testing Pyramid with the Unit, Integration and UI layers and just using a tool to write ever more UI cases does not promote this conversation and action within the organization.

Currently approaches such as 'keep the test suite under 15 mins by deciding which end to end UI tests are needed and which existing ones could be moved to integration or unit tests' are not approaches that AI test automation can handle as far as I'm aware. Maybe google have it internally.

Answer 2

Here's my two cents.

I tried Applitools (the free account) and I think it offers some excellent features, like the speed of test creation and modification, less code written and ease of catching visual bugs. Another advantage compared to standard automation tests is the possibility of catching unexpected (visual) bugs. For example, you would usually write asserts for known web elements (that they are visible, contain the right text and so on) but you wouldn't usually assert that another element would become visible or that page layout would change.

Some of the downsides would be the price (obviously) and test speed execution since those types of tools need to upload screenshots to the server.

An approach that could benefit from visual testing would be to keep the usual test suites but maybe add a visual test case (one per page) that would check the page layout.