11 votes
Accepted

Measuring code coverage in end-to-end tests?

I always like to get code coverage for my functional tests, but not because I want to hit a certain percentage of code coverage. I like it because: It points me to areas of the code that are not ...
user avatar
  • 8,409
8 votes

Why do code coverage of integration test?

I'll assume the question is this: Why measure how much of the SUT's code is covered by integration tests? and not this: Why measure how much of the integration test code is covered? I think ...
user avatar
  • 22.8k
8 votes

How can 100% test coverage be "not enough to prove working code"?

Thinking of 100% test coverage as a holy grail of testing is a common misconception which leads to over-confidence in the testing strategy and a false sense of security: code coverage metric only ...
user avatar
  • 11.4k
7 votes
Accepted

How can 100% test coverage be "not enough to prove working code"?

In addition to the other answers, a few extra thoughts: 100% coverage does not mean it meets user expectations - Your tests could cover 100% of the code, but that does not mean that the application ...
user avatar
  • 30.6k
6 votes
Accepted

How would we know when we are over-testing?

We're over-testing when we are not adding value It's easy to test to feel good but it's important to do testing that truly adds value as the business defines it. This is a difficult thing to ...
user avatar
5 votes
Accepted

What to call a matrix with all possible combinations?

I think it is called a "Combinatorial test matrix" . With more combinations the matrix could become infinitely large, have a look at Pairwise Testing to simplify this, but still have a high coverage ...
user avatar
5 votes
Accepted

Working just on branch or line coverage could prevent us from finding which defect?

Achieving branch coverage is possible but it doesn't mean complete testing. Indeed, no amount of fact-checking can achieve complete testing, because fact-checking is only one of the activities of ...
user avatar
  • 9,786
4 votes

How should a Black-box Tester measure Test Coverage?

Black box approach in and of itself is specifically designed to come from the user perspective backwards. The difference between black box and white box testing is knowledge of the underlying code ...
user avatar
  • 1,653
4 votes
Accepted

What are basic metrics of coverage with functional UI automation testing

Coverage is always coverage related to some model. This often gets skipped over, which leads to much confusion "you said you had 100% coverage so how come there's a bug?!" When you're looking at unit ...
user avatar
  • 5,085
4 votes

Measuring code coverage in end-to-end tests?

What you really want to measure with functional tests is functional coverage: how much of the functionality of the program was tested? Unfortunately, that's hard to measure in an automated fashion; ...
user avatar
  • 3,344
4 votes

What to call a matrix with all possible combinations?

It could be called anything as long as the stakeholders of the project understand what it is and how to use it. You can call it combinatorial input matrix, all pairs matrix, exhaustive input matrix ...
user avatar
4 votes

How would we know when we are over-testing?

I've not, yet, listened to the podcast so the what follows here are my initial, unfiltered, tuppence worth thoughts on the simple question regarding over-testing implied by coverage.py output so far ...
user avatar
3 votes

What non-structural coverage criteria are there?

There are the following types of test coverage criteria: structural criteria data criteria fault-model criteria requirements-based criteria criteria based on explicit test case specifications ...
user avatar
3 votes

ISTQB Foundation Level Paper 3, no. 37

It's three paths. If I express it via a more c# syntax you get this: if (condition1) { statement = statement1; } else { statement = statement2; } That gives us 2 paths: condition1 == ...
user avatar
  • 30.6k
3 votes

Is it a good practice to run functional tests with coverage?

First of all, coverage isn't coverage. There're several coverage criteria, but most used are: Function coverage – Has each function (or subroutine) in the program been called? Statement ...
user avatar
  • 2,102
3 votes

How would we know when we are over-testing?

Over-testing also hides defects in layers. I discovered in test code reviews & in executions analysis that if we repeat a basic initial assertion in multiple transactional flow tests(maybe due to ...
user avatar
3 votes

How can 100% test coverage be "not enough to prove working code"?

Normally the app is not just the code, but infrastructure, resources, protocols, databases, etc. Usability, performance, stability aspects are not usually a matter of code coverage as well.
user avatar
  • 11.4k
3 votes
Accepted

How many test cases can I run if I'm using domain testing?

Domain testing is an umbrella term for Equivalent partition and boundary value analysis. Here, we try to cover all the available behaviours of a system by using the least number of inputs. Here domain ...
user avatar
  • 10.8k
3 votes
Accepted

How can I measure the coverage of my E2E tests?

Michael Bolton proposes testing coverage as: “X coverage is how thoroughly we have examined the product with respect to some model of X”. And he completes: "Test coverage, like quality, is not ...
user avatar
  • 9,786
2 votes

What are basic metrics of coverage with functional UI automation testing

I would give them a percentage number of test cases automated of all test cases that can be automated. The key here is to develop the full list of test cases that "can" be automated. What test case ...
user avatar
2 votes
Accepted

How to interpret / analyse code coverage results and provide suggestions for improvements?

Coverage should be done by developers, especially coverage by unit tests. You can track the numbers and suggest that if code is added to a module, test is added so new code is covered. But there is ...
user avatar
2 votes

Test coverage tool that shows call graph of untested new code from a merge

You could try Diff-Cover: https://github.com/Bachmann1234/diff-cover Diff coverage is the percentage of new or modified lines that are covered by tests. This provides a clear and achievable ...
user avatar
2 votes

Why do code coverage of integration test?

You typically exclude test code from code coverage reports. The entire goal is to learn which parts of production code are touched by your test code. If you run integration tests against a deployed ...
user avatar
  • 487
2 votes

Testing approaches with temporal delays

Like Kate mentioned, these are easy to miss. Many systems have tasks that kick off at certain times. Its good to ask for, or check, the cron tables to see those jobs. Then you can run a "mid-night" ...
user avatar
  • 1,274
2 votes

Testing approaches with temporal delays

This is the kind of potential problem that tends to be picked up by testers with a lot of in-depth and broad knowledge of the application when reviewing user stories/use cases/requirements - and it's ...
user avatar
  • 30.6k
2 votes
Accepted

Comparing test coverage metrics to identify an increase/decrease after implementing automation

Test coverage and implementation of automation framework are 2 different things. According to me they are not linked directly. Automation frameworks won't write the test scripts on their own. People ...
user avatar
2 votes
Accepted

Examples for LOC figures, of test suites, and software projects?

So you need real-world examples of LOC metrics in a software sources and a test suite LOC volume, right? I can share some rough metrics for one real-world project. Never heard about representative ...
user avatar
2 votes

How to document the paths of my windows service in an easy to read diagram?

I am not aware of a specific type/name for this documentation, but you can use draw.io to create a flow/path diagram as you mentioned. You can also explore the different types of diagrams they have ...
user avatar
2 votes
Accepted

Why Phantomjs got crashed while running more test cases?

PhantomJS notoriously has a lot of memory issues and will crash after a certain amount of memory use. I have dealt with this issue by splitting up the tests into separate phantom processes rather than ...
user avatar
  • 136
2 votes

How many test cases required to achieve 100 % statement coverage

There is exactly one executable statement in the example, so if the question is not asking for branch coverage, one test case will give 100% coverage. An if statement is a flow, or branching ...
user avatar
  • 30.6k

Only top scored, non community-wiki answers of a minimum length are eligible