10
I always like to get code coverage for my functional tests, but not because I want to hit a certain percentage of code coverage. I like it because:
It points me to areas of the code that are not covered.
There are areas of the code that are very difficult to unit/integration test without having the entire system in place and doing end to end tests, so I ...
10
I know this problem way too well. There's no "right" answer, unfortunately, but there are some things you can do to help with this problem.
Dependency map - do you have a list of application features that have heavy dependencies and tend to break when changes occur in other areas? If you know changes in feature X tend to break feature Y, you know you always ...
8
Thinking of 100% test coverage as a holy grail of testing is a common misconception which leads to over-confidence in the testing strategy and a false sense of security:
code coverage metric only tells you what percentage of your code was executed. That's it - this is all that the coverage metric determines. The actual value of this depends on how your ...
7
This is a tool-agnostic question. All of them work the same way: they instrument the code of your system. Each time a line or branch of your code is visited at runtime, the tool caches this information. Afterall, the number of visited lines (or branches) is divided by the total number of lines (or branches). This is done in the scope of a single class/file, ...
7
I'll assume the question is this:
Why measure how much of the SUT's code is covered by integration tests?
and not this:
Why measure how much of the integration test code is covered?
I think you should avoid measuring code coverage for integration tests, for two reasons:
Unit tests are a much more effective way to exercise code than integration tests....
7
In addition to the other answers, a few extra thoughts:
100% coverage does not mean it meets user expectations - Your tests could cover 100% of the code, but that does not mean that the application satisfies potential users. The application could:
have performance issues under load
have workflows that users find too awkward or too difficult to use
not ...
6
We're over-testing when we are not adding value
It's easy to test to feel good but it's important to do testing that truly adds value as the business defines it.
This is a difficult thing to determine however as there are a number of factors to consider in order to determine both cost and value added by the testing. It is further complicated that some ...
5
Firstly, I've been exactly where you are and I know the pain you're going through. It's also very 'cool' at the moment for everyone to talk about 'Automation' and how it's a 'golden ticket' to Continuous Integration. Everybody wants their stuff out fast.
There are ways to tackle this, but personally, I think a lot of comes with having an understanding of ...
5
Achieving branch coverage is possible but it doesn't mean complete testing.
Indeed, no amount of fact-checking can achieve complete testing, because fact-checking is only one of the activities of testing, which also includes modeling, learning, experimentation, etc.
(Reference)
What kind of defect you can find with this technique?
Code and mutation ...
4
Not exactly.
Unit testing only tests units in isolation. In these tests all dependencies to other units are mocked or stubbed out. So how do you know those units together do what they are supposed to do? Code tends to grow hierarchically in complexity, and with that growth comes more and more units working together, more and more groups of units (modules) ...
4
Coverage is always coverage related to some model. This often gets skipped over, which leads to much confusion "you said you had 100% coverage so how come there's a bug?!"
When you're looking at unit tests, then it's possible to use code coverage as an indicator (there are tools that can measure what percentage of the lines in your code are exercised when ...
4
What you really want to measure with functional tests is functional coverage: how much of the functionality of the program was tested? Unfortunately, that's hard to measure in an automated fashion; the best measurement we have is by hand, correlating tests to requirements and counting up what didn't get covered.
Code coverage can be used as an indicator of ...
4
Black box approach in and of itself is specifically designed to come from the user perspective backwards. The difference between black box and white box testing is knowledge of the underlying code and components.
Therefore when you are prepping for black box testing you should be coming from the user perspective who utilize the application. This is any ...
4
I've not, yet, listened to the podcast so the what follows here are my initial, unfiltered, tuppence worth thoughts on the simple question regarding over-testing implied by coverage.py output so far as I understand it.
Having scanned the doc set for coverage.py, I cannot find any mechanism for determining data variance over the executed code. The outputs ...
3
There are the following types of test coverage criteria:
structural criteria
data criteria
fault-model criteria
requirements-based criteria
criteria based on explicit test case specifications
criteria based on statistical methods for random test data generation
criteria based on mutation-analysis
All criteria except the first one are non-structural.
...
3
You have many excellent answers already. Adding my 2 cents to say that your story hits close to home. Management and I have come to an agreeable compromise which is that I created "Mini regression" test scenarios for only the high traffic areas and modules of the program. The Mini Regression is performed as needed, but usually during installation testing. ...
3
There are many cases where unit testing as you have described would not be "sufficient".
(And you haven't really defined what you mean by "sufficient" in this case. Good enough to move the code to Production? Good enough to pass it on to QA? Good enough to please your boss? Good enough to feel like you did a good job? Something else?)
In most practical ...
3
I have encountered exactly the same problem and there is no "right" answer, but I can share with you what I've learned:
1) Code changes in one place can cause problems in another. Unless your project is so incredibly well documented that you know exactly what a change can impact, testing the full application on releases is important.
2) Find a balance ...
3
It's three paths. If I express it via a more c# syntax you get this:
if (condition1)
{
statement = statement1;
}
else
{
statement = statement2;
}
That gives us 2 paths:
condition1 == true -> statement1
condition1 == false -> statement2
Next, statement 2 is checked for true/false:
if (statement2)
{
statement = statement3
}
So your ...
3
First of all, coverage isn't coverage. There're several coverage criteria, but most used are:
Function coverage – Has each function (or subroutine) in the program been called?
Statement coverage – Has each statement in the program been executed?
Branch coverage – Has each branch (also called DD-path) of each control structure (such as in if and ...
3
Over-testing also hides defects in layers.
I discovered in test code reviews & in executions analysis that if we repeat a basic initial assertion in multiple transactional flow tests(maybe due to copy/paste) then on failing it fails all those tests and hides all the other subsequent defects which might be far critical than this one and more
importantly ...
3
Normally the app is not just the code, but infrastructure, resources, protocols, databases, etc. Usability, performance, stability aspects are not usually a matter of code coverage as well.
3
Domain testing is an umbrella term for Equivalent partition and boundary value analysis. Here, we try to cover all the available behaviours of a system by using the least number of inputs. Here domain means each partition that is created.
Equivalent partition means we divide the inputs into different partitions, here each partition means input value range ...
3
Michael Bolton proposes testing coverage as:
“X coverage is how thoroughly we have examined the product with respect to some model of X”.
And he completes:
"Test coverage, like quality, is not something that yields very well to quantitative measurements, except when we’re talking of very narrow and specific conditions."
Being that, you can think ...
2
Can't help with .net as I mainly have experience at a java shop. We use a combo of sonar and Jacoco.
Check out Jacoco!
I just installed it on the machines where we execute our service tests and the reports it generates are nice and can be integrated into CI easily (using ant). It's actively worked on, and has a Maven plug-in. The coolest feature for me ...
2
Feb 10 2016 Update
For the .net world mstest continues to provide code coverage. We also worked on integrating the Microsoft build technologies (msbuild, TeamBuild and the new TFS Build) with SonarQube.
https://blogs.msdn.microsoft.com/visualstudioalm/2015/08/24/build-tasks-for-sonarqube-analysis/
Btw, TFS build also supports Java builds (via Maven, Ant ...
2
You could implement your own version of minset that would be useful to anyone that uses gcov/lcov or, with a small modification, also to users of other code coverage measuring tools.
Export gcov/lcov report to some processable format, e.g. XML. The report should
describe coverage per method/class/package (granularity is up to you) for each test you ...
2
For backend coverage (mostly java), we use Cobertura: http://cobertura.github.io/cobertura/
The developers tend to use Emma, in Eclipse: http://emma.sourceforge.net/
For frontend (JavaScript), I've heard good things about ScriptCover: http://googletesting.blogspot.com/2011/10/scriptcover-makes-javascript-coverage.html
Good luck!
2
Basically your problem is you are trying to squeeze two months of work into one, make yourself super efficient while also dealing with an inordinate amount of code, and a large group of people who are claiming that you are slowing them down. All true, and while all of the answers given will help, you will also need to use some diplomacy in being able to ...
2
Coverage should be done by developers, especially coverage by unit tests. You can track the numbers and suggest that if code is added to a module, test is added so new code is covered.
But there is nothing you can do (as QA) to increase code coverage by unit tests. So if you responsible for that, it is by definition an exercise in frustration (because you ...
Only top voted, non community-wiki answers of a minimum length are eligible
