2

I'm looking for a way to measure the coverage of my end-to-end tests. I have some existing E2E tests already and I'm adding more test scenarios to it. But I would like to have a way to quantify the increase in coverage after the new tests have been added.

As E2E tests are more functional and feature based than unit tests, I'm not sure if I could use code coverage as a measurement.

If it matters, I'm using Cypress in my E2E tests.

How can I measure the coverage of my E2E tests? What are some common methods and tools that can help me in this?

3
  • 1
    What type of coverage are you looking for? There are many different types of coverages. Jun 19 '20 at 9:40
  • 1
    @JoãoFarias I actually want to know what kind of coverage and metric I can use for E2E tests. It’s not about coverage for unit test but for E2E tests which are usually more functional and feature based.
    – xenon
    Jun 19 '20 at 9:42
  • Nice, so I think it's better to rephrase the question a bit to ask about different types of coverage. The Cypress / E2E details are close to irrelevant to this question. Jun 19 '20 at 10:08
3

Michael Bolton proposes testing coverage as:

“X coverage is how thoroughly we have examined the product with respect to some model of X”.

And he completes:

"Test coverage, like quality, is not something that yields very well to quantitative measurements, except when we’re talking of very narrow and specific conditions."

Being that, you can think about the different types of coverage in relation to each element of your system. The Heuristic Testing Strategy Model shows a list of possible elements you may want to take a look:

enter image description here enter image description here

(You can download the mindmap above here)

E.g., you can think of coverage in terms of types of users you have, you can think of timings and interruptions, in terms of concurrency, etc.

Since "quality is value to some person (that matters)"(Jerry Weinberg), you can choose the types of coverage that better will yield information for the people interested in your testing. E.g., programmers may be interested in error handling and sizing (big/small inputs); managers may be interested in compatibility and environments.

3
  • Bolton likes to make big statements and doesn't really care about implementation and real life. This mind map is really nice as presentation material or as a base for developing list of tests and checks (yes I know..), but useless when you try to measure things using it. If you count the percentage of leaves covered you miss an entire world of sub leaves for example "creation of data" hides under it a huge number of things.
    – Rsf
    Jun 25 '20 at 7:25
  • And that's the point of breaking down the elements of each product: You are trying to find the information that the people involved in the product value for making decisions. When a shared understanding is reached and we feel we can be honest and clear with each other, we probably have it enough. Software development (and specially testing) is more of a social activity than a mathematical endevour. E.g., simply "Data coverage" hides a lot of things under it; as does "data creation coverage" - however the latter hides less things, so we made progress towards something more clear and honest. Jun 25 '20 at 7:40
  • 2
    Again you are using slogans and not concrete actions and results. To measure coverage you need a list and then count how much of this list is covered, even code coverage is turned into a list somewhere in the process. Your and Bolton's suggestion will lead to something close to requirement coverage which is highly inaccurate since you miss the implementation of each test, unless you write a document with very low level detailing
    – Rsf
    Jun 25 '20 at 7:45
1

I am afraid there is no good enough answer for that.

There are tools and methods but they all have major disadvantages that the people you report to should be aware of.

  • Test coverage, how many tests/checks out of the entire test set were run. The problem here is that you measure the percentage out of the tests you have and not out of what should be tested. It is usually a huge difference.

  • Requirements coverage, counts how many requirements have related tests. This method has a major problem because it doesn't check how good the tests are, for example is it enough to have one "sunny day" test to mark a requirement covered ? There are other problems related to the implementation for example what to do with requirements that depends on each other or tests that relate to multiple requirements.

  • Code coverage can sometimes be done even with E2E tests, but it would be cumbersome to run and the results are again have limited meaning for example a branch is counted as tested for every value but as testers we want to also test edge conditions.

So what should you do ? first answer yourself about why do you need to know your test coverage- if it is to plan your work then choose a simple method and remember its limitations, if its for management try the requirements or test coverage and add a BIG disclaimer under the graph.

Finally, remember that test coverage will measure against a list of pre defined tests, you will usually want to add to that sessions of exploratory tests that are almost impossible to meaure.

1

I might be late to answer.

If you are looking specifically for coverage while using Cypress for E2E, you can check this doc: https://docs.cypress.io/guides/tooling/code-coverage.html#Introduction

Here it is shown that we can measure code coverage using additional plugins though this coverage is just an indicator of what all lines of code are not at all covered in E2E.

0

The methods and tools of measuring the coverage depend on two things:

  1. Coverage of what your going to measure
  2. If you want to measure the code coverage what is the PL your app is written in

In the case you'd like to measure the coverage of requirements the things are straightforward. Just calculate what part of your requirements are covered by the tests included into E2E set

In the case of code coverage you can use JaCoCo for Java. If you have a .Net app you can check this big post on SO where the different tools are discussed.

The main point is that you compile your code with debug information, then instrument your code with a chosen tool. Then you deploy your app and run your tests. And finally you execute a tool in reporting mode so that it builds a coverage report for you.

P.S. - As far as you're talking about integration tests you will probably need to instrument all the apps which participate in integrated environment (depends on what your goals are).

2
  • Note that the aforementioned requirement coverage can only encompass the explicit requirements. Things dependent on implicit and tacit knowledge would be ignored by this approach. Jun 19 '20 at 9:42
  • @JoãoFarias is right, also note that the term "covered requirement" is also misleading- it usually means that a test was linked to a requirement and nothing about how good the coverage is.
    – Rsf
    Jun 25 '20 at 7:29
0

You didn't mention what language your application was written in. However, I have a good quick solution for anyone coming across this question for a Java application.

JaCoCo Java agent

I'll be using the JaCoCo test coverage tool. Specifically it's java agent.

First go to the JaCoCo github releases page to download the zip of the latest release. Once downloaded, extract the zip and find the jacoco.jar file. A quick example script:

wget https://github.com/jacoco/jacoco/releases/download/v0.8.6/jacoco-0.8.6.zip
unzip jacoco-0.8.6.zip
find -name 'jacocoagent.jar'

Note the location of the output file (for me it was in ./lib/jacocoagent.jar)

In the JaCoCo java agent it specifies that you need to run the java agent like so:

-javaagent:[yourpath/]jacocoagent.jar=[option1]=[value1],[option2]=[value2]

In our case you'll be running your application with something like this:

java -javaagent:lib/jacocoagent.jar=destfile=test.exec [...]

Then the rest of your java commands (-jar, -classpath, etc).

You can also optionally specify the append=false java agent option if you want to overwrite your test coverage results.

Run your application with the java command specified above and then run your tests against it. Once the tests are finished it's important to shut down the application before accessing your test.exec file. The `test.exec file is written to when the application exits. See the java agent docs for more information.

JaCoCo reporting with Gradle

I'm not familiar of a more generic way of running JaCoCo (though I do believe it has a CLI tool) so I'm going to wrap this up with a gradle example.

First, make sure that you change the above agent option to destdir=build/jacoco/test.exec as this is the default location where the 'jacoco' plugin looks for your coverage data file.

In order to generate your test coverage reports, add the 'jacoco' plugin to your to your build.gradle file then run the jacocoTestReport gradle task (e.g. ./gradlew jacocoTestReport if you're using gradlew). You can view your test coverage in your browser by navigating to build/reports/jacoco/test/html/index.html relative to your project root.

I've used this before for E2E testing but I can see it being useful in other situations like seeing what code is being run during exploratory testing.

-2

Today you can use many Test Management tools to contain the results of your automation. These tools also provide a variety of dashboards and reports that can represent your coverage in any way you like. Since your tests are E2E the best tool for you is probably PractiTest - this tool has all End-to-End modules, from Requirements to test executions and bugs. They have open API, so you can just apply it to your code - I know for sure that it can be done with Cypress. Good example for coverage you can extract from there is Execution Progress dashboard graph - this one will show the progress of your testing per execution statuses (% of passed, % of failed etc) per certain time period. I use this one to present data to our PMs.

1
  • While technically tools shows coverage the number itself is meaningless, it's the percentage of tests ran. What one is looking for is the percentage from a theoretical "all tests" or "all the product"
    – Rsf
    Jun 25 '20 at 7:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.