End to End Test cases for code merge approval

Question

We have lots of end to end test cases to reduce the manual automation work, and the list keeps growing as we add new features.

Now we added this test cases in our development workflow, to automatically run and approve the code merge to master.

But this creates lots of other problems,

1. It eats up lots of time in the development release cycle
2. The probability of build failures ( so merge failures) are high as there is a high chance of a mismatch between the e2e cases and the new features.

So what is the best practice to do the 'code approval' process,

1. To include all the automated test cases or
2. To include only the happy path, so the build sanity is guaranteed, and any other issues will be part of QA cycle and raised against the developer

Answer 1

Code merge permission is all about getting the code to JBGE (Just barely good enough) quality quickly and that component Integration is not borked. Stick to things that tell you the "factory" production line is on fire. Don't add tests that tell you if the Canteen is on fire, that's not blocking production.

• If your set of automated tests that block merge actually start running end-to-end tests, you will find that running times always grow from there. Avoid functional and feature tests that may block a dev team.
• Change your mindset, any minor bugs a suite finds but that block code merge are ultimately holding up the production line. Don't hold up the line for a bug that can be fixed and detected later in a triggered follow-on job. This means you need to divide your testing into 2 halves, tests that tell you about the build stability and testability, and tests that tell you about the ship-ability of the product.
• Do not test any deeper than that all components can interact, stay on happy path. By limiting yourself to happy path, you can still allow code that's not complete to get merged so long as that code does not block the primary function.
• Remove tests. Remove any tests that have not failed for a long time, they are not telling you anything.
• any tests you remove, push them into a regression suite, trigger the regression suite when the smoke-test or merge-blocking suite completes green.

In summary you want 2 suites:

1. One suite of tests (it might actually be just one big test) that tells you the build is further testable, use these to block code merges that could get you into merge hell. Example only test that the system can deploy/install and that a user can login, add one "object", remove the object, and logout again without any errors. Some people call this a BVT (Build Validation Test.) All produt features need to be enabled/active, but don't actually verify the feature in a BVT, if you do, you are chasing a moving target, and will constantly be making edits to your BVT suite, edit's that can break the suite itself. This suite should run in a minimal environment that matches the MVP (Minimal viable product) in it's configuration.
2. Feature checks should be done in the second suite. Any more testing that falls into the category of helping you decide if a build is ready for handing to manual testers. This will include a mix of new feature and regression suites that run quickly. And you will want to use a better environment for this suite.

Stress suites and load tests as well as performance and any tests are another class or suite you might have. You might also have a suite of test cases that are hard for humans to run competently, using large tables of data. But this is all under the covers stuff that most non testers might not care much about.