11
votes
Accepted
Measuring code coverage in end-to-end tests?
I always like to get code coverage for my functional tests, but not because I want to hit a certain percentage of code coverage. I like it because:
It points me to areas of the code that are not ...
- 8,409
8
votes
Why do code coverage of integration test?
I'll assume the question is this:
Why measure how much of the SUT's code is covered by integration tests?
and not this:
Why measure how much of the integration test code is covered?
I think ...
- 22.8k
8
votes
How can 100% test coverage be "not enough to prove working code"?
Thinking of 100% test coverage as a holy grail of testing is a common misconception which leads to over-confidence in the testing strategy and a false sense of security:
code coverage metric only ...
- 11.4k
7
votes
Accepted
How can 100% test coverage be "not enough to prove working code"?
In addition to the other answers, a few extra thoughts:
100% coverage does not mean it meets user expectations - Your tests could cover 100% of the code, but that does not mean that the application ...
- 30.6k
6
votes
Accepted
How would we know when we are over-testing?
We're over-testing when we are not adding value
It's easy to test to feel good but it's important to do testing that truly adds value as the business defines it.
This is a difficult thing to ...
- 24.2k
5
votes
Accepted
What to call a matrix with all possible combinations?
I think it is called a "Combinatorial test matrix" .
With more combinations the matrix could become infinitely large, have a look at Pairwise Testing to simplify this, but still have a high coverage ...
5
votes
Accepted
Working just on branch or line coverage could prevent us from finding which defect?
Achieving branch coverage is possible but it doesn't mean complete testing.
Indeed, no amount of fact-checking can achieve complete testing, because fact-checking is only one of the activities of ...
- 9,786
4
votes
How should a Black-box Tester measure Test Coverage?
Black box approach in and of itself is specifically designed to come from the user perspective backwards. The difference between black box and white box testing is knowledge of the underlying code ...
- 1,653
4
votes
Accepted
What are basic metrics of coverage with functional UI automation testing
Coverage is always coverage related to some model. This often gets skipped over, which leads to much confusion "you said you had 100% coverage so how come there's a bug?!"
When you're looking at unit ...
- 5,085
4
votes
Measuring code coverage in end-to-end tests?
What you really want to measure with functional tests is functional coverage: how much of the functionality of the program was tested? Unfortunately, that's hard to measure in an automated fashion; ...
- 3,344
4
votes
What to call a matrix with all possible combinations?
It could be called anything as long as the stakeholders of the project understand what it is and how to use it. You can call it combinatorial input matrix, all pairs matrix, exhaustive input matrix ...
- 6,603
4
votes
How would we know when we are over-testing?
I've not, yet, listened to the podcast so the what follows here are my initial, unfiltered, tuppence worth thoughts on the simple question regarding over-testing implied by coverage.py output so far ...
3
votes
What non-structural coverage criteria are there?
There are the following types of test coverage criteria:
structural criteria
data criteria
fault-model criteria
requirements-based criteria
criteria based on explicit test case specifications
...
- 1,635
3
votes
ISTQB Foundation Level Paper 3, no. 37
It's three paths. If I express it via a more c# syntax you get this:
if (condition1)
{
statement = statement1;
}
else
{
statement = statement2;
}
That gives us 2 paths:
condition1 == ...
- 30.6k
3
votes
Is it a good practice to run functional tests with coverage?
First of all, coverage isn't coverage. There're several coverage criteria, but most used are:
Function coverage – Has each function (or subroutine) in the program been called?
Statement ...
- 2,102
3
votes
How would we know when we are over-testing?
Over-testing also hides defects in layers.
I discovered in test code reviews & in executions analysis that if we repeat a basic initial assertion in multiple transactional flow tests(maybe due to ...
- 5,370
3
votes
How can 100% test coverage be "not enough to prove working code"?
Normally the app is not just the code, but infrastructure, resources, protocols, databases, etc. Usability, performance, stability aspects are not usually a matter of code coverage as well.
- 11.4k
3
votes
Accepted
How many test cases can I run if I'm using domain testing?
Domain testing is an umbrella term for Equivalent partition and boundary value analysis. Here, we try to cover all the available behaviours of a system by using the least number of inputs. Here domain ...
- 10.8k
3
votes
Accepted
How can I measure the coverage of my E2E tests?
Michael Bolton proposes testing coverage as:
“X coverage is how thoroughly we have examined the product with respect to some model of X”.
And he completes:
"Test coverage, like quality, is not ...
- 9,786
2
votes
What are basic metrics of coverage with functional UI automation testing
I would give them a percentage number of test cases automated of all test cases that can be automated. The key here is to develop the full list of test cases that "can" be automated. What test case ...
- 166
2
votes
Accepted
How to interpret / analyse code coverage results and provide suggestions for improvements?
Coverage should be done by developers, especially coverage by unit tests. You can track the numbers and suggest that if code is added to a module, test is added so new code is covered.
But there is ...
2
votes
Test coverage tool that shows call graph of untested new code from a merge
You could try Diff-Cover: https://github.com/Bachmann1234/diff-cover
Diff coverage is the percentage of new or modified lines that are
covered by tests. This provides a clear and achievable ...
2
votes
Why do code coverage of integration test?
You typically exclude test code from code coverage reports. The entire goal is to learn which parts of production code are touched by your test code.
If you run integration tests against a deployed ...
- 487
2
votes
Testing approaches with temporal delays
Like Kate mentioned, these are easy to miss.
Many systems have tasks that kick off at certain times. Its good to ask for, or check, the cron tables to see those jobs. Then you can run a "mid-night" ...
- 1,274
2
votes
Testing approaches with temporal delays
This is the kind of potential problem that tends to be picked up by testers with a lot of in-depth and broad knowledge of the application when reviewing user stories/use cases/requirements - and it's ...
- 30.6k
2
votes
Accepted
Comparing test coverage metrics to identify an increase/decrease after implementing automation
Test coverage and implementation of automation framework are 2 different things. According to me they are not linked directly.
Automation frameworks won't write the test scripts on their own. People ...
- 6,603
2
votes
Accepted
Examples for LOC figures, of test suites, and software projects?
So you need real-world examples of LOC metrics in a software sources and a test suite LOC volume, right? I can share some rough metrics for one real-world project. Never heard about representative ...
- 268
2
votes
How to document the paths of my windows service in an easy to read diagram?
I am not aware of a specific type/name for this documentation, but you can use draw.io to create a flow/path diagram as you mentioned. You can also explore the different types of diagrams they have ...
- 84
2
votes
Accepted
Why Phantomjs got crashed while running more test cases?
PhantomJS notoriously has a lot of memory issues and will crash after a certain amount of memory use. I have dealt with this issue by splitting up the tests into separate phantom processes rather than ...
- 136
2
votes
How many test cases required to achieve 100 % statement coverage
There is exactly one executable statement in the example, so if the question is not asking for branch coverage, one test case will give 100% coverage.
An if statement is a flow, or branching ...
- 30.6k
Only top scored, non community-wiki answers of a minimum length are eligible