5

I joined a full stack automation team recently. There are some selenium tests for the frontend and APIs aren't automated yet. My question or thought process is should I pick the test cases carefully to avoid any overlap between frontend selenium and backend Rest assured based tests? Or it's common to have overlapping test cases in this scenario.

The backend APIs are consumed only by web, there isn't any mobile or other teams that consume them.

  • 1
    Could you explain some overlapping tests use cases that you could come up . – PDHide Sep 1 at 8:46
  • And unit tests are created by devs, left out completely, or taken care of by someone else? – pavelsaman Sep 2 at 18:35
12

You should always pick tests carefully when it comes to test automation. :)

One of the reasons, as you said, is overlap (and with that, execution time and robustness). An example to clarify:

  • Your API has 10 endpoints which might return several different error messages each.
  • Don't test each error as a UI test: this will take up lots of runtime and will also cause the highest maintenance. And yes, you'll have functional overlap with API and unit tests.
  • Don't test each error as an API test if their logic is fully covered in unit tests.
  • Do write a UI test for one or two errors to make sure they are properly displayed by the front-end. But this is probably a generic system, so if the system works it'll work for any error message. UI tests should be considered as probing the application's flows and see a user can get the job done, not in-depth testing of logic.
  • Do write API tests for one or two errors to make sure the back-end integration is playing nicely from request to response (beyond the scope of unit tests). Or write more tests for specific cases (e.g. where database access comes into play, which will be mocked in a unit test).

Another reason to think about which cases to automate is simply that not every automated tests is as useful or cost efficient in the long run. I advise you to look on YouTube for Angie Jones' presentation on "Which tests should we automate" - also see https://slides.com/angiejones/which-tests-should-we-automate#/20

| improve this answer | |
  • nice answer :) this makes it more clear for both – PDHide Sep 1 at 9:45
  • 1
    But in most cases Backend error handling is different from UI. You won't just display backend messages straight on to UI . If the app is using front end validation then you cannot consider it as overlapping – PDHide Sep 1 at 9:48
  • 1
    @PDHide in those cases, it does make sense to test at the UI level (although that doesn't necessarily mean end-to-end - UI unit tests are a thing). However, it's not that uncommon for the UI to treat all backend errors the same - either because it wasn't seen as worth doing, or because security mandated it. – James_pic Sep 1 at 12:41
6

There is no concept of overlapping testcases in different test levels,

Both are completely isolated

Just because API works fine you cannot guarantee UI works fine.

Imagine all your API tests passing but user not able to use the UI , Imagine all your UI working due to cached information but actual backend is failing.

Ensure more low level coverage like unit test and API test , this ensures that you will have faster test execution and build feedback. This will also ensures faster debugging as your tests will be more focused on component or feature.

In UI test actual business flow and error handling tests

In each test level we have different test scopes.

Unit test;

We don't test the business flow but the component and functionality

Integration Test

Integration with other components , how stable is the integrated subsystem to be able to be used to extend with higher level components . Like API with UI

System Test

Here you test Usability , user interactions , visual regression , business logic and flow.

So there are no concept of overlapping tests in different testlevels

| improve this answer | |
  • 3
    Please add some sensible comments thats useful for the community when you down vote. Be professional and learn to discuss and suggest things. – PDHide Sep 1 at 8:55
  • 4
    I didn't downvote, but saying there is no overlap between test levels is a bold statement. :) – FDM Sep 1 at 9:35
  • 1
    @FDM thanks for the comment :) – PDHide Sep 1 at 9:50
  • 1
    +1 for answer anyway. happy to have side-effect of countering the downvote also (but not why I upvoted of course). – Michael Durrant Sep 1 at 18:52
  • 1
    I agree with both this answer and the ones mentioning avoiding overlaps. Strictly speaking, when working for a properly engineered project or with a large development team you can never make the assumption that overlapping test cases test each other. However, for a small team of developers working closely together where everyone understand all the code you can look at the architecture and be satisfied with the understanding that some logic is already tested in some other test. – slebetman Sep 2 at 6:58
3

TL;DR: you will have overlap between E2E and API integration test cases, in terms of the same endpoints being exercised in both and that's OK - it helps you figure out where the problem is if (...when) something goes wrong.

When working with a codebase that doesn't currently have comprehensive automated testing, start with the E2E(/functional/UI) tests. Why?

  1. Automating the app through the UI workflows helps build empathy for the users - what are they using this for, and how do they do it?

  2. These tests let you check the software actually delivers the value it's supposed to; your users don't care about API calls or functions! Note that this would be different if your API was a product in itself, not just consumed by the web client.

  3. From a more technical testing perspective, lower-level tests will likely require some changes to implement (e.g. to introduce appropriate boundaries to test at); code written without thinking about testing is often difficult to test. You need the higher-level tests to give you confidence those changes were made correctly.

This will likely lead to a place where you have too many E2E tests, characterised by overly long test run times, but you can now start pushing tests down the stack to integration and unit tests. Focus on maintaining a set of key workflows (this can be a good conversation with the product people in your team - does everyone know what the key workflows are?) at the E2E level, then push the less important paths and repetition to lower-level tests.

In terms of the API tests specifically, there will be a lot of overlap; your E2E test cases should exercise every endpoint at least once (if not, think about whether the unused endpoints can be removed). This overlap is fine, because now if an E2E test fails but the relevant API tests pass you've localised the issue to the UI. But there will be things that are hard to test through the UI. Commonly these are the unhappy paths, for example:

  • you probably have validation of inputs at the UI level that prevents the request getting made if they're invalid, but you should still be testing server-side validation; and

  • you probably don't have links to missing resources in the UI, but still want to test 404s.

Similarly there are things that are hard to test through the API, and require a lot of setup and teardown; in this case, push down further to unit test the service/business logic layer (I would not recommend unit testing the controller/transport or repository/persistence layers; these tend to be largely boilerplate, if they have a lot of logic it's probably in the wrong place).

| improve this answer | |
1

There is no need to test the same thing with API and UI tests.

Start with API (keeping in mind the test pyramid), provided code is sufficiently covered with unit tests, and automate certain e2e scenarios that would cover cases not covered by individual API.

| improve this answer | |
0

My gut feeling is to concentrate on automating the backend API first.

  • Unit tests are good and necessary, but they don't give me high confidence that the system as a whole functions. Some of the most insidious bugs happen when different parts of the specification interact in ways the writer did not think about, and unit tests tend to capture only a very "local" view on the specification.
    Say in one class null values are rejected as invalid, in another class null values are interpreted as an empty list. Odds are that the unit tests for each class faithfully test exactly that behaviour.

  • GUI tests are good and necessary, but they are also hard if they are supposed to replace manual testing. There are so many different devices, so many different browsers. An automated test which tells you that the system is "good to use" on many devices is a lot of work. (That might be a slight bias on my part because of my backend background, and it assumes that the business logic is in the backend ...).

  • API tests represent the "contract" of one subsystem against another. It might be difficult to generate test data which is both realistic and exhaustive, but once you have that you can have some confidence that the backend does what it is supposed to do.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.