Tests in CI: Are API + Integration Tests the most suitable?

Question

In my experience, automated GUI tests may take too long for a build of Jenkins (even parallelized) and may demand a lot of maintenance. On the other hand API + Integration tests are way faster, do not demand so much maintenance, can be very functional and test e2e (backend, APIs, database...).

So thinking of the Test Pyramid, keeping the majority of the tests Unit Tests and in order to have a more functional test in Continuous Integration... Is API + Integration testing the best option for tests in CI? What's your take and experience on that? Is there a 'consensus'?

This article on Techbeacon talks about the issues of GUI tests in CI. Functional Tests in the eShopOnContainers are API tests (it's specifically about micro services, but it's still a good reference, right?).

Answer 1

As already mentioned in your TechBeacon article, teams indeed often spend too much at the top of the test automation pyramid. In general, the pyramid is a good rule of thumb, but (as always) it depends on the project. Many systems are well designed so that the GUI is just some glue between the user and the underlying APIs, which is why such systems often don't need GUI-based E2E tests at all.

However, the need of GUI-based E2E tests is something you can't decide per se—if you need them, you need them. The thing is that nowadays you can quite easily scale out these tests. For instance, "UI Testing at Scale with AWS Lambda" shows how to leverage Function as a service (FaaS) to run every test in parallel. This way, your entire GUI tests may run in less than a minute.

Therefore, I would say you can have fast GUI-based E2E tests within your CI pipeline, but it is not always feasible. Analyize the given project, calculate the risks, determine the trade-offs, decide.

Answer 2

Use the test pyramid as mentioned by Peter Masiar.

My contribution here is help answer the question of exactly how to decide the 20% and 80% figures that Peter mentions.

My approach to this is to make sure that the UI testing focuses on the functionality of the UI itself. In other words, from a user perspective the text and links are correctly displayed and - in particular - HTML forms and input fields work as expected.

The way to avoid having a large number of UI cases is to avoid what I have come to term 'data combinatorial testing'. By this I mean any test that says 'given this extra piece of data "x", I should see "y". "x" can be data input by the user, data retrieved from a data store, data from an external service, etc. This should be tested on the back end.

An example will probably help:

A business operates in 50 states. Each State has a different welcome message targeted to the local culture. The business might ask 'can we have 50 tests please to make sure we have all the messages correct?'. It would be possible to write 50 browser UI tests, one for each state that verify the information. However, the correct way to do this would be to recognize that it isn't "the browser" that determine the text. It is something on the back end that says "when I am passed the text "Wyoming" (for example) I will return the text "Home of Yellowstone", etc. So we should write back end unit/integration tests that test that each state passed to the routine that determines the message results in the correct test. Then we test one state in the browser to make sure that the lookup works.

Given that browser tests might take 30 seconds each (including some navigation to get the page in question) that would mean 50 x 30 would be 1500 seconds or 30 minutes test run time. Whereas a unit test might run in 1 second (or much less), so (30 * 1) + 30(for 1 browser test) = 1 minute test time - 1/30 of the time when compared to 30 browser tests.

To the question of "Is API + Integration testing the best option for tests in CI", I don't think specifically so. CI is a good option for both unit/api tests as well as UI tests. In both cases you get the advantage of:

• run on a different machine which is always good
• parallelization and elastic clouds
• run tests on a known configuration
• ability to create an automated devops pipeline

The fact that UI tests are soooo slow and benefit so greatly from the parallelization benefit of CI, those are the ones I often want to run in CI. Also as Unit and Development tests are intended for instant feedback to the developer it feels more appropriate to make sure that these, at least initially run locally to be as 'close' as possible in the simplest way, e.g. no CI, local code. UI tests are often at a later phase or during UAT testing when there has been separation of time from the coding so the need to run them locally can be lessened.

Answer 3

Consider test pyramid. Using Pareto principle (also known as 80/20 rule), you can get 80% of benefits from testing by focusing on unit test with 20% of the effort. And 80% benefit of the remainder if you focus on API/service test.

So of course you need "some" UI-level test, 96% of effort should be lower: 80% in unit and 16% (80% of the remaining 20%) in API/service test.

I doubt than you can find "consensus" but if you do it right, your UI test would be exactly right size for your overall testing: small enough to be maintainable and agile, and large enough to make managers and customers confident that system works as expected.

Answer 4

I mostly agree with other great answers however I want to add one thing from my side- UI tests are no longer so slow with new generation of UI automation libraries like puppeteer/Playwright.

I am talking about the difference on the magnitude of at least 10 times faster compared to traditional test libraries/frameworks like selenium.

I recently converted a suite of 70+ tests from selenium to playwright and execution time went down from 1 hour to 5 minutes. And on top of that it has ZERO flakiness in execution.

If I could write a single data driven UI test with different 70 data combinations which will be executed in 5 mins, I would prefer that one over API test with 70 data combinations which is taking 1 minute.

The reason is , I am covering more layers in a single test where lot of logic/validation is also client side(being a modern web app) which will be covered.