I'm looking for a way to measure the coverage of my end-to-end tests. I have some existing E2E tests already and I'm adding more test scenarios to it. But I would like to have a way to quantify the increase in coverage after the new tests have been added.

As E2E tests are more functional and feature based than unit tests, I'm not sure if I could use code coverage as a measurement.

If it matters, I'm using Cypress in my E2E tests.

How can I measure the coverage of my E2E tests? What are some common methods and tools that can help me in this?

  • 1
    What type of coverage are you looking for? There are many different types of coverages. – João Farias Jun 19 at 9:40
  • 1
    @JoãoFarias I actually want to know what kind of coverage and metric I can use for E2E tests. It’s not about coverage for unit test but for E2E tests which are usually more functional and feature based. – xenon Jun 19 at 9:42
  • Nice, so I think it's better to rephrase the question a bit to ask about different types of coverage. The Cypress / E2E details are close to irrelevant to this question. – João Farias Jun 19 at 10:08

The methods and tools of measuring the coverage depend on two things:

  1. Coverage of what your going to measure
  2. If you want to measure the code coverage what is the PL your app is written in

In the case you'd like to measure the coverage of requirements the things are straightforward. Just calculate what part of your requirements are covered by the tests included into E2E set

In the case of code coverage you can use JaCoCo for Java. If you have a .Net app you can check this big post on SO where the different tools are discussed.

The main point is that you compile your code with debug information, then instrument your code with a chosen tool. Then you deploy your app and run your tests. And finally you execute a tool in reporting mode so that it builds a coverage report for you.

P.S. - As far as you're talking about integration tests you will probably need to instrument all the apps which participate in integrated environment (depends on what your goals are).

| improve this answer | |
  • Note that the aforementioned requirement coverage can only encompass the explicit requirements. Things dependent on implicit and tacit knowledge would be ignored by this approach. – João Farias Jun 19 at 9:42
  • @JoãoFarias is right, also note that the term "covered requirement" is also misleading- it usually means that a test was linked to a requirement and nothing about how good the coverage is. – Rsf Jun 25 at 7:29

I am afraid there is no good enough answer for that.

There are tools and methods but they all have major disadvantages that the people you report to should be aware of.

  • Test coverage, how many tests/checks out of the entire test set were run. The problem here is that you measure the percentage out of the tests you have and not out of what should be tested. It is usually a huge difference.

  • Requirements coverage, counts how many requirements have related tests. This method has a major problem because it doesn't check how good the tests are, for example is it enough to have one "sunny day" test to mark a requirement covered ? There are other problems related to the implementation for example what to do with requirements that depends on each other or tests that relate to multiple requirements.

  • Code coverage can sometimes be done even with E2E tests, but it would be cumbersome to run and the results are again have limited meaning for example a branch is counted as tested for every value but as testers we want to also test edge conditions.

So what should you do ? first answer yourself about why do you need to know your test coverage- if it is to plan your work then choose a simple method and remember its limitations, if its for management try the requirements or test coverage and add a BIG disclaimer under the graph.

Finally, remember that test coverage will measure against a list of pre defined tests, you will usually want to add to that sessions of exploratory tests that are almost impossible to meaure.

| improve this answer | |

Michael Bolton proposes testing coverage as:

“X coverage is how thoroughly we have examined the product with respect to some model of X”.

And he completes:

"Test coverage, like quality, is not something that yields very well to quantitative measurements, except when we’re talking of very narrow and specific conditions."

Being that, you can think about the different types of coverage in relation to each element of your system. The Heuristic Testing Strategy Model shows a list of possible elements you may want to take a look:

enter image description here enter image description here

(You can download the mindmap above here)

E.g., you can think of coverage in terms of types of users you have, you can think of timings and interruptions, in terms of concurrency, etc.

Since "quality is value to some person (that matters)"(Jerry Weinberg), you can choose the types of coverage that better will yield information for the people interested in your testing. E.g., programmers may be interested in error handling and sizing (big/small inputs); managers may be interested in compatibility and environments.

| improve this answer | |
  • Bolton likes to make big statements and doesn't really care about implementation and real life. This mind map is really nice as presentation material or as a base for developing list of tests and checks (yes I know..), but useless when you try to measure things using it. If you count the percentage of leaves covered you miss an entire world of sub leaves for example "creation of data" hides under it a huge number of things. – Rsf Jun 25 at 7:25
  • And that's the point of breaking down the elements of each product: You are trying to find the information that the people involved in the product value for making decisions. When a shared understanding is reached and we feel we can be honest and clear with each other, we probably have it enough. Software development (and specially testing) is more of a social activity than a mathematical endevour. E.g., simply "Data coverage" hides a lot of things under it; as does "data creation coverage" - however the latter hides less things, so we made progress towards something more clear and honest. – João Farias Jun 25 at 7:40
  • 2
    Again you are using slogans and not concrete actions and results. To measure coverage you need a list and then count how much of this list is covered, even code coverage is turned into a list somewhere in the process. Your and Bolton's suggestion will lead to something close to requirement coverage which is highly inaccurate since you miss the implementation of each test, unless you write a document with very low level detailing – Rsf Jun 25 at 7:45

Today you can use many Test Management tools to contain the results of your automation. These tools also provide a variety of dashboards and reports that can represent your coverage in any way you like. Since your tests are E2E the best tool for you is probably PractiTest - this tool has all End-to-End modules, from Requirements to test executions and bugs. They have open API, so you can just apply it to your code - I know for sure that it can be done with Cypress. Good example for coverage you can extract from there is Execution Progress dashboard graph - this one will show the progress of your testing per execution statuses (% of passed, % of failed etc) per certain time period. I use this one to present data to our PMs.

| improve this answer | |
  • While technically tools shows coverage the number itself is meaningless, it's the percentage of tests ran. What one is looking for is the percentage from a theoretical "all tests" or "all the product" – Rsf Jun 25 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.