What's the difference between code coverage and test coverage?

Question

At first I though it was the same but with eclemma it seems different. But I can't find anything about it.

Answer 1

Test coverage and code coverage as generic terms can mean a broad variety of different measures of what has been covered: lines of code, subconditions of boolean expressions, accesses to data, calls to API functions, ...

In the common vernacular, and for almost all the tools you can get, "test coverage" and "code coverage" tell you what code executed by whatever testing process you use (whether that is unit test, functional test, informal test, interactive test, ...).

The only widely used variation is MC/DC (test/code) coverage, which tells you which of the boolean subconditions in a logical expression have been exercised in a way that controls the outcome of the boolean expression. Only a few tools offer this.

I'm sure eclemma provides test/code coverage of the standard type. Whether it provides anything else... I don't know.

Answer 2

Here is how I use the terms. Code coverage is a measure of how much code is executed in response to a stimulus (e.g. running a test). Test coverage is a measure of how much of the feature set was executed as a result of a test.

Others will have other definitions. When someone uses those terms, if you are not sure what they mean, ask them.

Answer 3

Here is how I use these terms (although I almost never actually uses these terms).

Code coverage is a measure of how much code is executed during testing.

Test coverage is a measure of how many test cases have been executed during testing.

Your mileage may vary...

Answer 4

Code Coverage: As a human, we can make mistakes or miss some important issues which lead to errors and defects in a product. Testing thorough makes a product high quality. Performing Unit testing is a good start, but it is sometimes is difficult to measure or improve.

Code Coverage

To begin with, let’s learn about code coverage. The main purpose of this metric is to calculate the number of lines covered by the test cases. It reports total number of lines in the code and number of lines executed by tests. Think of it as the degree to which the source code of a program is executed when a test suite runs. The intent is, the higher the code coverage, the lower the chance of having undetected software bugs.

Subtypes

Coverage is split into several subtypes – condition coverage, branch coverage, loop coverage, function coverage, statement coverage, and parameter value coverage.

Let me define the subtypes for the sake of clarity.

· If testing cover each function calls in code, function coverage is called to be 100%.

· Similarly, if all branches in the code, i.e. all if-else conditions have been tested with every possible input, then branch coverage is said to be 100%.

· If all loop statements in the code are executed then loop coverage and statement coverage would be 100% respectively.

· In an ideal case, for condition coverage to be 100% every single condition with all feasible combination of condition parameters are perfectly tested. One can check the Parameter value coverage by passing all common values to the criteria in the function call.

Coverage testing tools

These testing tools are obtainable today for each major languages. Note, that mostly are relevant only for unit test and perhaps not for the complete tests performed ( system, integration, Unit, manual and more).

Now have a look at few of them to understand the code coverage tools scenario.

Cobertura is a Java code coverage reporting tool. It is based on coverage and works on any platform with Java 5 or higher. You can use Cobertura with ant, maven or command-line access. While instrumenting classes, Cobertura generates Cobertura.ser file containing basic information about each class. This file is utilized when instrumenting, running, and creating reports. It also uses ASM when instrumenting. On running testing, it creates a report comprising each package, all class within every package & corresponding branch coverage, line coverage, and complexity.

For Python, a well-admired tool for code coverage measurement is Coverage.py. It functions in 3 different phases –

· Execution, where it runs the code and monitors it to see what lines were executed.

· Analysis, where it examines the code to determine what lines could have run.

· Reporting is where it combines the results of execution and analysis to produce a coverage number and an indication of missing execution.

This measurement tool extensively used for Go is gocov. This kind of tool provides 4 key commands – test, convert, report & annotate. Making use of these commands, you can generate textual reports with annotations giving coverage information of each function in each go file. A GUI wrapper is also accessible for gocov called as GoCovGUI. It displays the entire report generated by the tool in GUI which is easy to understand. It also has wrappers to generate reports in HTML and XML format.

Test Coverage

Test Coverage aims to measure the effectiveness of testing in a qualitative manner. It evaluates either the test cases are covering whole functional needs. You can believe it as a type of black-box testing when test cases are not written contrary to code but based on user requirements or expected functionality.

Subtypes

The common structure and means used for test coverage measurement are unit testing, functional testing, performance testing, integration or system testing and acceptance testing.

Unit tests are written at a granular level to check if a function/method performs as expected. In functional testing, each functionality mentioned in the requirement document is tested. Performance testing is normally a method to stress test the code for its stability, scalability, and responsiveness at various workloads. Integration testing or system testing is done to test if the completely integrated product works in an expected manner. Acceptance testing is generally done at the end of the development cycle. For acceptance testing, the product is handed over to the stakeholders and tested by them to determine whether the product is in an acceptable state.

Tools

Such test coverage tools aim to measure the test effort in a qualitative way. The idea here is to see if all requirements and functionalities are logically covered via testing. Yet, it is not easy and straightforward to calculate as compared to code coverage. Methodologies such as Test-Driven Development (TDD) are also useful for analyzing test coverage and bring in the discipline for adding testing from the get-go. To measure the impact of such tests, you will need to manually list out requirements and then analyze test case to understand which of those are covered.

Still, you need frameworks to write tests.

Let’s see some testing frameworks available for various programming languages.

· JUnit, the popular Java testing framework serves as a foundation for launching tests on the Java Virtual Machine. With support for assumptions, assertions, default test techniques, and other mechanisms for testing functionality JUnit is a better option to test Java-based software.

· For Python, a framework called PyUnit is available. As the name indicates it is the JUnit equivalent for Python. However, there are various other test frameworks accessible to assist you as well.

Go also offers a built-in test package for programs allowing you effortlessly write, plan, organize and run testing. This also offers several profiling tools to understand behavior of the software under different loads.

Answer 5

As the above responses the definition varies. I sometimes say code coverage does not matter to me but test coverage does.

By test coverage I mean in this context that any possible behaviour of a class should be documented by tests (ie. if you look at the tests you should be able to make a safe assumption of the outputs and behaviour of any method of that class, provided any given set of inputs. Which also includes assumptions of invalid inputs.

Lines of code that do not impact behaviour are obviously undesirable, but if they exist that is usually less relevant. For me typical examples of such lines are small best practice approaches that are actually rarely used, so I might not test if I catch an exception if a file is written and I check if it exists before trying to write it. Having that sort of check often is better than not writing or testing this kind of code.

Answer 6

My 2 cents:

Code Coverage means the ratio of non-test code run by automated tests. Every line of code or instruction that was somehow evaluated and run counts towards this.

Test Coverage means the ratio of logically tested behaviour. "Behaviour" in this case means everything a method is supposed to do according to its description: validate and react to wrong inputs, return the expected result for all correct inputs.

Consider a method add(a, b) which is supposed to add up two numbers and return the sum. Now consider a test that calls the method with two positive numbers, but only looks for errors and does not check the logical outcome. This would probably mean 100% code coverage (whole method was run) but much less test coverage, since the method was tested only against one misbehaviour: erroring out on correct input. Neither the result was tested with different inputs, nor were wrong/unexpected inputs tested.

Answer 7

Test Coverage is Code Coverage but measured from the perspective of running your tests. You should aim for 100% test coverage. ;)

There is a 10 minute video here demonstrating the differences using python. I recommend watching it in double speed so it's only 5 minutes.

https://youtu.be/7BJ_BKeeJyM

Answer 8

Test coverage and Code coverage metrics are both measurements that can be helpful for assessing the quality of your application code.

Difference between code coverage and test coverage

CODE COVERAGE

• Code coverage is a measure of how much code is executed during tests &

• Subtypes- Coverage is split into several subtypes – loop coverage, branch coverage, condition coverage, statement coverage, function coverage, and parameter value coverage.

• Tools- Cobertura is a Java code coverage reporting tool. It is based on jcoverage and works on any platform with Java 5 or higher. You can make use of Cobertura with maven, ant, or command line access. While instrumenting classes, Cobertura generates cobertura.ser file comrising essential information about each class. This file is used when instrumenting, running, and creating reports.

For Python, the major and most popular tool for code coverage measurement is Coverage.py. It works in three distinct phases –

• Execution, where it runs the code and checks it to see what lines were executed.

• Analysis, where it examines the code to determine what lines could have run.

• Reporting is where it combines the results of execution and analysis to produce a coverage number and an indication of missing execution.

TEST COVERAGE

• Test coverage is a measure of how many test cases have been executed during tests

• Subtypes- The common mechanisms used for test coverage measurement are performance testing, functional testing, unit testing, integration or acceptance testing and system testing.

• Tools- JUnit, the well called as Java testing framework serves as a foundation to launch tests on the Java Virtual Machine. With support for assertions, assumptions, default test methods, and other mechanisms to test functionality JUnit is a very good choice for testing Java-based software.

For Python, a framework called PyUnit is available.

As the name indicates it is the JUnit equivalent for Python. But there are several other testing frameworks available to help you as well.