11
I always like to get code coverage for my functional tests, but not because I want to hit a certain percentage of code coverage. I like it because:
It points me to areas of the code that are not covered.
There are areas of the code that are very difficult to unit/integration test without having the entire system in place and doing end to end tests, so I ...
8
Thinking of 100% test coverage as a holy grail of testing is a common misconception which leads to over-confidence in the testing strategy and a false sense of security:
code coverage metric only tells you what percentage of your code was executed. That's it - this is all that the coverage metric determines. The actual value of this depends on how your ...
7
I'll assume the question is this:
Why measure how much of the SUT's code is covered by integration tests?
and not this:
Why measure how much of the integration test code is covered?
I think you should avoid measuring code coverage for integration tests, for two reasons:
Unit tests are a much more effective way to exercise code than integration tests....
7
In addition to the other answers, a few extra thoughts:
100% coverage does not mean it meets user expectations - Your tests could cover 100% of the code, but that does not mean that the application satisfies potential users. The application could:
have performance issues under load
have workflows that users find too awkward or too difficult to use
not ...
6
We're over-testing when we are not adding value
It's easy to test to feel good but it's important to do testing that truly adds value as the business defines it.
This is a difficult thing to determine however as there are a number of factors to consider in order to determine both cost and value added by the testing. It is further complicated that some ...
5
Achieving branch coverage is possible but it doesn't mean complete testing.
Indeed, no amount of fact-checking can achieve complete testing, because fact-checking is only one of the activities of testing, which also includes modeling, learning, experimentation, etc.
(Reference)
What kind of defect you can find with this technique?
Code and mutation ...
4
Coverage is always coverage related to some model. This often gets skipped over, which leads to much confusion "you said you had 100% coverage so how come there's a bug?!"
When you're looking at unit tests, then it's possible to use code coverage as an indicator (there are tools that can measure what percentage of the lines in your code are exercised when ...
4
What you really want to measure with functional tests is functional coverage: how much of the functionality of the program was tested? Unfortunately, that's hard to measure in an automated fashion; the best measurement we have is by hand, correlating tests to requirements and counting up what didn't get covered.
Code coverage can be used as an indicator of ...
4
Black box approach in and of itself is specifically designed to come from the user perspective backwards. The difference between black box and white box testing is knowledge of the underlying code and components.
Therefore when you are prepping for black box testing you should be coming from the user perspective who utilize the application. This is any ...
4
I've not, yet, listened to the podcast so the what follows here are my initial, unfiltered, tuppence worth thoughts on the simple question regarding over-testing implied by coverage.py output so far as I understand it.
Having scanned the doc set for coverage.py, I cannot find any mechanism for determining data variance over the executed code. The outputs ...
3
There are the following types of test coverage criteria:
structural criteria
data criteria
fault-model criteria
requirements-based criteria
criteria based on explicit test case specifications
criteria based on statistical methods for random test data generation
criteria based on mutation-analysis
All criteria except the first one are non-structural.
...
3
It's three paths. If I express it via a more c# syntax you get this:
if (condition1)
{
statement = statement1;
}
else
{
statement = statement2;
}
That gives us 2 paths:
condition1 == true -> statement1
condition1 == false -> statement2
Next, statement 2 is checked for true/false:
if (statement2)
{
statement = statement3
}
So your ...
3
First of all, coverage isn't coverage. There're several coverage criteria, but most used are:
Function coverage – Has each function (or subroutine) in the program been called?
Statement coverage – Has each statement in the program been executed?
Branch coverage – Has each branch (also called DD-path) of each control structure (such as in if and ...
3
Over-testing also hides defects in layers.
I discovered in test code reviews & in executions analysis that if we repeat a basic initial assertion in multiple transactional flow tests(maybe due to copy/paste) then on failing it fails all those tests and hides all the other subsequent defects which might be far critical than this one and more
importantly ...
3
Normally the app is not just the code, but infrastructure, resources, protocols, databases, etc. Usability, performance, stability aspects are not usually a matter of code coverage as well.
3
Domain testing is an umbrella term for Equivalent partition and boundary value analysis. Here, we try to cover all the available behaviours of a system by using the least number of inputs. Here domain means each partition that is created.
Equivalent partition means we divide the inputs into different partitions, here each partition means input value range ...
3
Michael Bolton proposes testing coverage as:
“X coverage is how thoroughly we have examined the product with respect to some model of X”.
And he completes:
"Test coverage, like quality, is not something that yields very well to quantitative measurements, except when we’re talking of very narrow and specific conditions."
Being that, you can think ...
2
You could implement your own version of minset that would be useful to anyone that uses gcov/lcov or, with a small modification, also to users of other code coverage measuring tools.
Export gcov/lcov report to some processable format, e.g. XML. The report should
describe coverage per method/class/package (granularity is up to you) for each test you ...
2
For backend coverage (mostly java), we use Cobertura: http://cobertura.github.io/cobertura/
The developers tend to use Emma, in Eclipse: http://emma.sourceforge.net/
For frontend (JavaScript), I've heard good things about ScriptCover: http://googletesting.blogspot.com/2011/10/scriptcover-makes-javascript-coverage.html
Good luck!
2
I would give them a percentage number of test cases automated of all test cases that can be automated. The key here is to develop the full list of test cases that "can" be automated. What test case management tool do you use?
2
You typically exclude test code from code coverage reports. The entire goal is to learn which parts of production code are touched by your test code.
If you run integration tests against a deployed version of your application you should look into remote code coverage, typically exposed via JMX. Jacoco has support for this, see the agent documentation.
2
Like Kate mentioned, these are easy to miss.
Many systems have tasks that kick off at certain times. Its good to ask for, or check, the cron tables to see those jobs. Then you can run a "mid-night" simulation test where you test those jobs.
Another example was a subscription billing system, where new tasks were fired at monthly intervals (charge the ...
2
This is the kind of potential problem that tends to be picked up by testers with a lot of in-depth and broad knowledge of the application when reviewing user stories/use cases/requirements - and it's very easy to miss.
Some of the methods I've used to try to catch this kind of problem include:
Asking which rules are supposed to apply, and then asking for ...
2
Test coverage and implementation of automation framework are 2 different things. According to me they are not linked directly.
Automation frameworks won't write the test scripts on their own. People will be writing the test scripts, which they will later execute using the automation tool/framework.
The test coverage of test scripts will be as good as the ...
2
So you need real-world examples of LOC metrics in a software sources and a test suite LOC volume, right? I can share some rough metrics for one real-world project. Never heard about representative figures. It highly depends on technologies and tools you're using.
The applications are written in C++ (Windows only though). Metrics were done with simple Python ...
2
I am not aware of a specific type/name for this documentation, but you can use draw.io to create a flow/path diagram as you mentioned. You can also explore the different types of diagrams they have and see if any of them is a good fit.
2
Coverage should be done by developers, especially coverage by unit tests. You can track the numbers and suggest that if code is added to a module, test is added so new code is covered.
But there is nothing you can do (as QA) to increase code coverage by unit tests. So if you responsible for that, it is by definition an exercise in frustration (because you ...
2
You could try Diff-Cover: https://github.com/Bachmann1234/diff-cover
Diff coverage is the percentage of new or modified lines that are
covered by tests. This provides a clear and achievable standard for
code review: If you touch a line of code, that line should be covered.
Seems Diff-Cover uses Cobertura XML coverage reports, those might or might not ...
answered Feb 10 '15 at 16:25
Niels van Reijmersdal
31.6k4 gold badges52 silver badges118 bronze badges
2
PhantomJS notoriously has a lot of memory issues and will crash after a certain amount of memory use. I have dealt with this issue by splitting up the tests into separate phantom processes rather than all in one.
2
There is exactly one executable statement in the example, so if the question is not asking for branch coverage, one test case will give 100% coverage.
An if statement is a flow, or branching statement. It evaluates conditions and redirects the application flow according to whether or not the conditions are met.
An executable statement acts on the ...
Only top voted, non community-wiki answers of a minimum length are eligible