Resources on implementing a well-balanced test pyramid

Question

The test pyramid is a concept developed by Mike Cohn, described in his book "Succeeding with Agile". Basically, end-to-end tests (often called UI tests) go across all the layers of the application, while unit tests cover single components.

While the idea is sound and logical, I found implementing a well-balanced test pyramid hard, particularly when different teams are responsible for different levels of tests. For instance, in our teams devs focus on unit tests, while testers focus on end-to-end tests. Service-level tests are done either by testers, devs or by both, depending on the knowledge of architecture, testing skills, available time, etc.

Therefore, I'm looking for resources (books, blogs) that could help me answer:

1. How do you make sure you do not duplicate test cases at different levels? Sure, when you verify how the system handles invalid input from a user, you may discover different problems at unit level, service level and UI level. Does it mean you want to replicate all unit tests at higher level as well? I would like to avoid that, as end-to-end tests takes longer and thus their number should be limited.

2. Following the above question, how do you recognize redundant test cases?

3. When a tester discover a case on end-to-end level (UI) that could be easily tested at unit level (and is missing in unit test suite), should she add it to the unit test suite?

I understand the area is wide, hence I'm more interested in hands-on experiences and lessons learned than silver bullet solutions.

Answer 1

I've been wondering the same thing for the last couple of years. In my organization we've begun using BDD style testing to help with our pyramid problems. Right now we have about 800 automated acceptance tests, 3000 manual tests, 1500 integration/service tests, and 1000 unit tests. Obviously not ideal.

I've realized that a lot of those manual tests, automated or not, can be tested more efficiently through integration or even unit tests, but our QA analysts have no idea what those lower level tests are doing. Using BDD to describe the behavior that is being tested is helping QA analysts, SDETs, developers, etc all get on the same page as to what is being tested. In most cases it doesn't really matter to anybody what the implementation of a specific test is, just that some behavior is functioning as intended.

As for your specific questions, I think this helps us avoid #1 and #2 before it's an issue. Regarding test duplication, certain things like input validation are requirements at multiple levels (service, web) and we prefer to validate those at both levels with unit tests if possible.

For #3 when a tester discovers a defect, he can write a BDD test for it which will be implemented at the correct level as determined by a developer or SDET (see the Depth of Test link below).

Here are a couple of things I've read about this topic:

Depth of test

Automated Testing and the Test Pyramid

Answer 2

How do you make sure you do not duplicate test cases at different levels?

First, set expectations about what's tested on each level. E.g. validation may be tested on Unit level while the fact the validation is applied can be tested on Component Level (Services, Controllers) and System Level (JavaScript, HTML). See an example of how to build Test Pyramid for Single Page apps: Building Test Pyramid to optimize automated testing

Additionally all the test levels need to be written/maintained/reviewed by the same people. There are multiple ways of doing this:

• Get rid of dedicated AQA engineers. Devs can write all the tests on their own. Manual QA or BA would need to be available for the consult on what cases to cover.
• Move away from QA as Quality Assurance to QA as Quality Assistance by introducing people who're very good at testing but instead of testing they teach Devs how to test.
• Entrust AQA with privileges to review/update/reject tests at all the levels - including Unit Tests written by Devs.

How do you recognize redundant test cases

This can be effectively detected by reviewing the code that was written previously. Once in a while we can retrospect the tests and see whether they duplicate each other and whether the coverage is sufficient. You may include Manual QA in these reviews.

You need to ensure that the tests are very readable and the test cases are easily implementable - this will help in finding the holes and duplicates and won't impose high maintenance costs if these are found.

When a case is discovered on UI level that could be tested at unit level, do we add it to the unit test suite?

Absolutely. Though different people may do this - either AQA if those people exist, or Devs.

I understand the area is wide, hence I'm more interested in hands-on experiences and lessons learned than silver bullet solutions.

In spite of all the given attention, Test Pyramids are still not widely spread. And therefore you should take all the information with grain of salt even from authoritative sources. As for now most of the hubbub about the Pyramid is concentrated in QA circles. But those need to widen to include Devs since without people who can write maintainable code it won't be possible to implement solid pyramids.

Answer 3

I've found this to be a touch and touchy sticking point in several organizations. I frequently see QA testers get excited about doing UI testing only to be brought down by the "we already have a unit test for that" conversation.

I now take this approach:

UI testing will duplicate units tests and that's ok.

Unit testing, once an experiment is now mainstream. It's getting normal to expect 80-100% of code to have unit tests. So when I write higher level UI automation tests I expect that most of the things I am exercising will have lower level unit tests. That's ok. I'm making sure all these parts work together and in order to do that I have to create, manipulate and use the same objects and code. But I am now more interested in how they work when all put together as frequently the interfaces are where unexpected things break.

I've found in practice that to get a well-balanced pyramid you should do a little bit of planning. Right now in my current organization I've created 20 top level UI tests (in selenium), we have 100 integrated feature tests using rspec-capybara and about 1000 unit tests in rspec. To maintain this going forward we review tickets during sprint grooming and decide if new/changed automated UI testing should be done with the ticket.