12
Solution Overview
To solve this problem, you will want to intelligently select a manageable set of combinations based on a pairwise coverage approach (explained below) or a more thorough variation of combinatorial test design.
Glowcoder and user246 have good points. I particularly like testerab's comment for reasons that will become clear in a minute.
I'...
10
I know this problem way too well. There's no "right" answer, unfortunately, but there are some things you can do to help with this problem.
Dependency map - do you have a list of application features that have heavy dependencies and tend to break when changes occur in other areas? If you know changes in feature X tend to break feature Y, you know you always ...
10
I always like to get code coverage for my functional tests, but not because I want to hit a certain percentage of code coverage. I like it because:
It points me to areas of the code that are not covered.
There are areas of the code that are very difficult to unit/integration test without having the entire system in place and doing end to end tests, so I ...
8
Thinking of 100% test coverage as a holy grail of testing is a common misconception which leads to over-confidence in the testing strategy and a false sense of security:
code coverage metric only tells you what percentage of your code was executed. That's it - this is all that the coverage metric determines. The actual value of this depends on how your ...
7
Actually, the underlying heuristic for combinatorial testing of multiple input parameters is that they are interdependent and the specific values assigned affect on a common output condition or state.
Based on the concept of testing various combinations of input variables that affect a common output condition or state, combinatorial testing may not be ...
7
The simplest option is to just mark the line as ignored by your coverage tests. You know more than coverage.py does, you can just excuse the line from the measurements:
if __name__ == '__main__': # pragma: no cover
return main(sys.argv)
You can also use some tricks with coverage.py to get it to measure code in launched subprocesses. This sounds ...
7
This is a tool-agnostic question. All of them work the same way: they instrument the code of your system. Each time a line or branch of your code is visited at runtime, the tool caches this information. Afterall, the number of visited lines (or branches) is divided by the total number of lines (or branches). This is done in the scope of a single class/file, ...
7
In addition to the other answers, a few extra thoughts:
100% coverage does not mean it meets user expectations - Your tests could cover 100% of the code, but that does not mean that the application satisfies potential users. The application could:
have performance issues under load
have workflows that users find too awkward or too difficult to use
not ...
6
We're over-testing when we are not adding value
It's easy to test to feel good but it's important to do testing that truly adds value as the business defines it.
This is a difficult thing to determine however as there are a number of factors to consider in order to determine both cost and value added by the testing. It is further complicated that some ...
5
Firstly, I've been exactly where you are and I know the pain you're going through. It's also very 'cool' at the moment for everyone to talk about 'Automation' and how it's a 'golden ticket' to Continuous Integration. Everybody wants their stuff out fast.
There are ways to tackle this, but personally, I think a lot of comes with having an understanding of ...
5
I wouldn't necessarily put it high on the ToDo list, but I think it's beneficial to measure test code coverage to find dead tests. You probably won't get to 100%**, but you can find dead functions and binaries - which makes a big difference when you have a 20-hour automation run you're trying to whittle down to an overnight run.
** note - test code often ...
4
Do all combinations need to be run? This sounds like a classic place to apply all-pairs testing. I haven't played with this yet myself, but I hear good things about Hexawise which might help with display and analysis of the results.
4
How to do it
Let's let your dimensions' sizes be C1, C2 ... Cn where n is the number of dimensions. So, C1 might be 3 if your values are Windows, Mac, Linux (I'm sure you'd have different versions of Windows and what not, but for the example, it works.) Your total number of tests will be C1 * C2 * ... * Cn.
I'm sure you already have a 2d matrix defined ...
4
Try these articles about RCov and simple_cov
http://highgroove.com/articles/2011/03/01/code-coverage-and-ruby-1-9.html
http://www.storm-consultancy.com/blog/development/tools-plugins/generating-code-coverage-metrics-for-a-ruby-on-rails-project-with-simplecov/
and rails_code_qa uses both of the above so also worth a look
https://github.com/nathanhumbert/...
4
Not exactly.
Unit testing only tests units in isolation. In these tests all dependencies to other units are mocked or stubbed out. So how do you know those units together do what they are supposed to do? Code tends to grow hierarchically in complexity, and with that growth comes more and more units working together, more and more groups of units (modules) ...
4
Coverage is always coverage related to some model. This often gets skipped over, which leads to much confusion "you said you had 100% coverage so how come there's a bug?!"
When you're looking at unit tests, then it's possible to use code coverage as an indicator (there are tools that can measure what percentage of the lines in your code are exercised when ...
4
What you really want to measure with functional tests is functional coverage: how much of the functionality of the program was tested? Unfortunately, that's hard to measure in an automated fashion; the best measurement we have is by hand, correlating tests to requirements and counting up what didn't get covered.
Code coverage can be used as an indicator of ...
4
I've not, yet, listened to the podcast so the what follows here are my initial, unfiltered, tuppence worth thoughts on the simple question regarding over-testing implied by coverage.py output so far as I understand it.
Having scanned the doc set for coverage.py, I cannot find any mechanism for determining data variance over the executed code. The outputs ...
3
You have many excellent answers already. Adding my 2 cents to say that your story hits close to home. Management and I have come to an agreeable compromise which is that I created "Mini regression" test scenarios for only the high traffic areas and modules of the program. The Mini Regression is performed as needed, but usually during installation testing. ...
3
There are many cases where unit testing as you have described would not be "sufficient".
(And you haven't really defined what you mean by "sufficient" in this case. Good enough to move the code to Production? Good enough to pass it on to QA? Good enough to please your boss? Good enough to feel like you did a good job? Something else?)
In most practical ...
3
I have encountered exactly the same problem and there is no "right" answer, but I can share with you what I've learned:
1) Code changes in one place can cause problems in another. Unless your project is so incredibly well documented that you know exactly what a change can impact, testing the full application on releases is important.
2) Find a balance ...
3
Have you tried the book The Art of Unit Testing ?
http://www.manning.com/osherove/
3
The short answer is situations when a bug can occur only when three or more states must be set a certain way.
The long answer ...
Pairwise testing good technique that actually applies some pretty sound logic. That said, it is fairly easy to understand what the potential tradeoffs are .. let me illustrate using an example I have used previously.
Let’s ...
3
The underlying assumption for those methods is that any two parameters are independent from each other, so any two parameters that do have some kind of dependency will show poor results, i.e. your coverage will not be as good as expected.
When I tried to use pairwise the first step was to find independent enough parameters, I couldn't find any that will also ...
3
There are the following types of test coverage criteria:
structural criteria
data criteria
fault-model criteria
requirements-based criteria
criteria based on explicit test case specifications
criteria based on statistical methods for random test data generation
criteria based on mutation-analysis
All criteria except the first one are ...
3
I'll assume the question is this:
Why measure how much of the SUT's code is covered by integration tests?
and not this:
Why measure how much of the integration test code is covered?
I think you should avoid measuring code coverage for integration tests, for two reasons:
Unit tests are a much more effective way to exercise code than integration tests....
3
Black box approach in and of itself is specifically designed to come from the user perspective backwards. The difference between black box and white box testing is knowledge of the underlying code and components.
Therefore when you are prepping for black box testing you should be coming from the user perspective who utilize the application. This is any ...
3
It's three paths. If I express it via a more c# syntax you get this:
if (condition1)
{
statement = statement1;
}
else
{
statement = statement2;
}
That gives us 2 paths:
condition1 == true -> statement1
condition1 == false -> statement2
Next, statement 2 is checked for true/false:
if (statement2)
{
statement = statement3
}
So your ...
3
First of all, coverage isn't coverage. There're several coverage criteria, but most used are:
Function coverage – Has each function (or subroutine) in the program been called?
Statement coverage – Has each statement in the program been executed?
Branch coverage – Has each branch (also called DD-path) of each control structure (such as in if and ...
3
Over-testing also hides defects in layers.
I discovered in test code reviews & in executions analysis that if we repeat a basic initial assertion in multiple transactional flow tests(maybe due to copy/paste) then on failing it fails all those tests and hides all the other subsequent defects which might be far critical than this one and more
importantly ...
Only top voted, non community-wiki answers of a minimum length are eligible
