10

In my experience, automated GUI tests may take too long for a build of Jenkins (even parallelized) and may demand a lot of maintenance. On the other hand API + Integration tests are way faster, do not demand so much maintenance, can be very functional and test e2e (backend, APIs, database...).

So thinking of the Test Pyramid, keeping the majority of the tests Unit Tests and in order to have a more functional test in Continuous Integration... Is API + Integration testing the best option for tests in CI? What's your take and experience on that? Is there a 'consensus'?

This article on Techbeacon talks about the issues of GUI tests in CI. Functional Tests in the eShopOnContainers are API tests (it's specifically about micro services, but it's still a good reference, right?).

  • 1
    What is the Integration test in your example? – Alexey R. Apr 4 '18 at 15:01
  • @AlexeyR. Call different APIs and query database, for example. – Roberto de Toledo Apr 4 '18 at 15:18
  • 2
    Re linked article with a title "Is GUI test automation a recipe for continuous delivery failure?" - Betteridge's law of headlines - "Any headline that ends in a question mark can be answered by the word no." But author has a point, as I explain in my answer – Peter M. Apr 4 '18 at 19:29
  • BTW good and very valid question, and good linked article. – Peter M. Apr 4 '18 at 19:40
  • 1
    I think , there is no black or white answer, it all depends on specifics of an project. – Vishal Aggarwal Apr 4 '18 at 21:39
9

As already mentioned in your TechBeacon article, teams indeed often spend too much at the top of the test automation pyramid. In general, the pyramid is a good rule of thumb, but (as always) it depends on the project. Many systems are well designed so that the GUI is just some glue between the user and the underlying APIs, which is why such systems often don't need GUI-based E2E tests at all.

However, the need of GUI-based E2E tests is something you can't decide per se—if you need them, you need them. The thing is that nowadays you can quite easily scale out these tests. For instance, "UI Testing at Scale with AWS Lambda" shows how to leverage Function as a service (FaaS) to run every test in parallel. This way, your entire GUI tests may run in less than a minute.

Therefore, I would say you can have fast GUI-based E2E tests within your CI pipeline, but it is not always feasible. Analyize the given project, calculate the risks, determine the trade-offs, decide.

  • 4
    +1 for test pyramid. But your system (and your tests) are not complex enough if you can run each of them under a minute. We have tests which take up to five minutes to just prepare all the test data using UI (and those data are are time-sensitive so they cannot be reused, and complex enough so they cannot be just injected to DB) – Peter M. Apr 4 '18 at 17:06
  • @PeterMasiar thanks for your input. Great example that—as is said—it may not always be feasible to "just do it". – beatngu13 Apr 4 '18 at 18:28
5

Consider test pyramid. Using Pareto principle (also known as 80/20 rule), you can get 80% of benefits from testing by focusing on unit test with 20% of the effort. And 80% benefit of the remainder if you focus on API/service test.

So of course you need "some" UI-level test, 96% of effort should be lower: 80% in unit and 16% (80% of the remaining 20%) in API/service test.

I doubt than you can find "consensus" but if you do it right, your UI test would be exactly right size for your overall testing: small enough to be maintainable and agile, and large enough to make managers and customers confident that system works as expected.

5

Use the test pyramid as mentioned by Peter Masiar.

enter image description here

My contribution here is help answer the question of exactly how to decide the 20% and 80% figures that Peter mentions.

My approach to this is to make sure that the UI testing focuses on the functionality of the UI itself. In other words, from a user perspective the text and links are correctly displayed and - in particular - HTML forms and input fields work as expected.

The way to avoid having a large number of UI cases is to avoid what I have come to term 'data combinatorial testing'. By this I mean any test that says 'given this extra piece of data "x", I should see "y". "x" can be data input by the user, data retrieved from a data store, data from an external service, etc. This should be tested on the back end.

An example will probably help:

A business operates in 50 states. Each State has a different welcome message targeted to the local culture. The business might ask 'can we have 50 tests please to make sure we have all the messages correct?'. It would be possible to write 50 browser UI tests, one for each state that verify the information. However, the correct way to do this would be to recognize that it isn't "the browser" that determine the text. It is something on the back end that says "when I am passed the text "Wyoming" (for example) I will return the text "Home of Yellowstone", etc. So we should write back end unit/integration tests that test that each state passed to the routine that determines the message results in the correct test. Then we test one state in the browser to make sure that the lookup works.

Given that browser tests might take 30 seconds each (including some navigation to get the page in question) that would mean 50 x 30 would be 1500 seconds or 30 minutes test run time. Whereas a unit test might run in 1 second (or much less), so (30 * 1) + 30(for 1 browser test) = 1 minute test time - 1/30 of the time when compared to 30 browser tests.

To the question of "Is API + Integration testing the best option for tests in CI", I don't think specifically so. CI is a good option for both unit/api tests as well as UI tests. In both cases you get the advantage of:

  • run on a different machine which is always good
  • parallelization and elastic clouds
  • run tests on a known configuration
  • ability to create an automated devops pipeline

The fact that UI tests are soooo slow and benefit so greatly from the parallelization benefit of CI, those are the ones I often want to run in CI. Also as Unit and Development tests are intended for instant feedback to the developer it feels more appropriate to make sure that these, at least initially run locally to be as 'close' as possible in the simplest way, e.g. no CI, local code. UI tests are often at a later phase or during UAT testing when there has been separation of time from the coding so the need to run them locally can be lessened.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.