How can I measure the coverage of my E2E tests?

Question

I'm looking for a way to measure the coverage of my end-to-end tests. I have some existing E2E tests already and I'm adding more test scenarios to it. But I would like to have a way to quantify the increase in coverage after the new tests have been added.

As E2E tests are more functional and feature based than unit tests, I'm not sure if I could use code coverage as a measurement.

If it matters, I'm using Cypress in my E2E tests.

How can I measure the coverage of my E2E tests? What are some common methods and tools that can help me in this?

Answer 1

Michael Bolton proposes testing coverage as:

“X coverage is how thoroughly we have examined the product with respect to some model of X”.

And he completes:

"Test coverage, like quality, is not something that yields very well to quantitative measurements, except when we’re talking of very narrow and specific conditions."

Being that, you can think about the different types of coverage in relation to each element of your system. The Heuristic Testing Strategy Model shows a list of possible elements you may want to take a look:

(You can download the mindmap above here)

E.g., you can think of coverage in terms of types of users you have, you can think of timings and interruptions, in terms of concurrency, etc.

Since "quality is value to some person (that matters)"(Jerry Weinberg), you can choose the types of coverage that better will yield information for the people interested in your testing. E.g., programmers may be interested in error handling and sizing (big/small inputs); managers may be interested in compatibility and environments.

Answer 2

I am afraid there is no good enough answer for that.

There are tools and methods but they all have major disadvantages that the people you report to should be aware of.

• Test coverage, how many tests/checks out of the entire test set were run. The problem here is that you measure the percentage out of the tests you have and not out of what should be tested. It is usually a huge difference.

• Requirements coverage, counts how many requirements have related tests. This method has a major problem because it doesn't check how good the tests are, for example is it enough to have one "sunny day" test to mark a requirement covered ? There are other problems related to the implementation for example what to do with requirements that depends on each other or tests that relate to multiple requirements.

• Code coverage can sometimes be done even with E2E tests, but it would be cumbersome to run and the results are again have limited meaning for example a branch is counted as tested for every value but as testers we want to also test edge conditions.

So what should you do ? first answer yourself about why do you need to know your test coverage- if it is to plan your work then choose a simple method and remember its limitations, if its for management try the requirements or test coverage and add a BIG disclaimer under the graph.

Finally, remember that test coverage will measure against a list of pre defined tests, you will usually want to add to that sessions of exploratory tests that are almost impossible to meaure.

Answer 3

I might be late to answer.

If you are looking specifically for coverage while using Cypress for E2E, you can check this doc: https://docs.cypress.io/guides/tooling/code-coverage.html#Introduction

Here it is shown that we can measure code coverage using additional plugins though this coverage is just an indicator of what all lines of code are not at all covered in E2E.

Answer 4

The methods and tools of measuring the coverage depend on two things:

1. Coverage of what your going to measure
2. If you want to measure the code coverage what is the PL your app is written in

In the case you'd like to measure the coverage of requirements the things are straightforward. Just calculate what part of your requirements are covered by the tests included into E2E set

In the case of code coverage you can use JaCoCo for Java. If you have a .Net app you can check this big post on SO where the different tools are discussed.

The main point is that you compile your code with debug information, then instrument your code with a chosen tool. Then you deploy your app and run your tests. And finally you execute a tool in reporting mode so that it builds a coverage report for you.

P.S. - As far as you're talking about integration tests you will probably need to instrument all the apps which participate in integrated environment (depends on what your goals are).

Answer 5

You didn't mention what language your application was written in. However, I have a good quick solution for anyone coming across this question for a Java application.

JaCoCo Java agent

I'll be using the JaCoCo test coverage tool. Specifically it's java agent.

First go to the JaCoCo github releases page to download the zip of the latest release. Once downloaded, extract the zip and find the jacoco.jar file. A quick example script:

wget https://github.com/jacoco/jacoco/releases/download/v0.8.6/jacoco-0.8.6.zip
unzip jacoco-0.8.6.zip
find -name 'jacocoagent.jar'

Note the location of the output file (for me it was in ./lib/jacocoagent.jar)

In the JaCoCo java agent it specifies that you need to run the java agent like so:

-javaagent:[yourpath/]jacocoagent.jar=[option1]=[value1],[option2]=[value2]

In our case you'll be running your application with something like this:

java -javaagent:lib/jacocoagent.jar=destfile=test.exec [...]

Then the rest of your java commands (-jar, -classpath, etc).

You can also optionally specify the append=false java agent option if you want to overwrite your test coverage results.

Run your application with the java command specified above and then run your tests against it. Once the tests are finished it's important to shut down the application before accessing your test.exec file. The `test.exec file is written to when the application exits. See the java agent docs for more information.

JaCoCo reporting with Gradle

I'm not familiar of a more generic way of running JaCoCo (though I do believe it has a CLI tool) so I'm going to wrap this up with a gradle example.

First, make sure that you change the above agent option to destdir=build/jacoco/test.exec as this is the default location where the 'jacoco' plugin looks for your coverage data file.

In order to generate your test coverage reports, add the 'jacoco' plugin to your to your build.gradle file then run the jacocoTestReport gradle task (e.g. ./gradlew jacocoTestReport if you're using gradlew). You can view your test coverage in your browser by navigating to build/reports/jacoco/test/html/index.html relative to your project root.

I've used this before for E2E testing but I can see it being useful in other situations like seeing what code is being run during exploratory testing.

Answer 6

Today you can use many Test Management tools to contain the results of your automation. These tools also provide a variety of dashboards and reports that can represent your coverage in any way you like. Since your tests are E2E the best tool for you is probably PractiTest - this tool has all End-to-End modules, from Requirements to test executions and bugs. They have open API, so you can just apply it to your code - I know for sure that it can be done with Cypress. Good example for coverage you can extract from there is Execution Progress dashboard graph - this one will show the progress of your testing per execution statuses (% of passed, % of failed etc) per certain time period. I use this one to present data to our PMs.