Measuring code coverage in end-to-end tests?

Question

I've been hearing about ideas for measuring code coverage in end-to-end tests (e.g., Selenium tests) and I really don't see a reason for that, even if this is technically feasible.

There's a concept of test pyramid that says that on the unit level you should have the highest number of tests, on integration level a medium number, and on end-to-end (Selenium) the smaller number of tests, because they are the hardest to maintain, the longest to execute and the least stable (because SUT is not isolated). So there's little chance you will cover all the code in end-to-end tests. Also, there might be areas in the lower layers of the code, which has not been exposed through, say, API so far (e.g., because the feature will be accessible in the next release), and can be tested only with unit tests.

I think the idea of Selenium or end-to-end test is to verify different business scenarios, so it is more about function/feature coverage.

Does measuring code coverage in end-to-end tests gives any value in addition to code coverage for unit tests?

Answer 1

I always like to get code coverage for my functional tests, but not because I want to hit a certain percentage of code coverage. I like it because:

1. It points me to areas of the code that are not covered.
2. There are areas of the code that are very difficult to unit/integration test without having the entire system in place and doing end to end tests, so I like to compare the coverage from unit/integration tests to the coverage from my functional tests and see if there are things that should or could be covered in the end to end tests that may be more difficult in the earlier stages.
3. I want to know which of my tests are equivalent, so I can look at what is covered by each test and see if there are tests I can eliminate or consolidate to be more efficient.

Answer 2

What you really want to measure with functional tests is functional coverage: how much of the functionality of the program was tested? Unfortunately, that's hard to measure in an automated fashion; the best measurement we have is by hand, correlating tests to requirements and counting up what didn't get covered.

Code coverage can be used as an indicator of functional coverage: if a huge section didn't get touched, you can deduce what functionality that code was designed to implement and check why it didn't get tested. Code coverage is easy to implement and, if you're also doing unit testing, probably already in place; it costs almost nothing extra to measure, so the fact that it's only useful as a fallback still makes it worth doing.

Answer 3

Ideally, it would be great to get code coverage metrics for the entire test suite (unit, functional, and end-to-end), but I've found that its difficult to actually get this measure. Also, I'm not sure what I'd do specifically with the measure - other than gain confidence that it's going up (not down).

In practice, its difficult to get this number because we have a variety of technologies in the system (Java, JavaScript, HTML), which usually means different tools used to measure coverage.

Knowing code coverage for unit tests is very valuable, because we can see what code is not covered by tests & design tests to exercise that code. For end-to-end tests, you have a doubt if the uncovered code is reachable at system level (without fault injection, for example), or is uncovered because of poor tests.

Frankly, if we have high levels of unit test coverage, good functional coverage, I would find a different problem to solve rather than trying to get a blended coverage metric.