6

Does it make sense to run once in a while, say, on a weekly basis, integration/functional tests against special build to have a coverage report. If anyone here has some practical experience of using coverage data for non-unit tests it'll be nice to hear out from you.

Is it worth it, would you recommend to make it an integral part of pipeline?

One thing that bothers me is that you can-not just test with coverage all the time, the whole idea behind functional tests is to test the product as it is. So, just like I've mentioned, it would be sort of "special" run which is launched periodically. This is doable but still introduces some additional complexity.

Improve this question
6
  • Related: sqa.stackexchange.com/questions/28010/…
    – FDM
    Jul 7, 2017 at 9:44
  • @FDM thank you for the link, an interesting question, still not sure how this is related - well, other than that both questions are about testing? )
    – shabunc
    Jul 7, 2017 at 9:46
  • 1
    @FDM my question is about special periodical build to fetch coverage report, but now I understand what you meant, yeah, in that sense it's related (definitely not a duplicate though)
    – shabunc
    Jul 7, 2017 at 9:50
  • 1
    Sure thing, didn't flag it as duplicate. ;) Hope you get a good answer!
    – FDM
    Jul 7, 2017 at 9:50
  • 1
    (moved to a comment) May be I am missing something here, but, we, for instance, always run daily and weekly tests with enabled coverage. The difference between running the tests with coverage and without is, when running with enabled coverage, there are "read-only" tracers put in places to monitor the execution of the program under test - they don't change the way the program under test behaves. The most noticeable difference then becomes the test execution speed - tests typically run slower with enabled coverage.
    – alecxe
    Jul 7, 2017 at 14:24

2 Answers 2

Reset to default
3

First of all, coverage isn't coverage. There're several coverage criteria, but most used are:

  • Function coverage – Has each function (or subroutine) in the program been called?
  • Statement coverage – Has each statement in the program been executed?
  • Branch coverage – Has each branch (also called DD-path) of each control structure (such as in if and case statements) been executed? […]
  • Condition coverage (or predicate coverage) – Has each Boolean sub-expression evaluated both to true and false?

However, whether coverage (in general) is a good measure is a quite controversial topic. For instance, Wei, Meyer, and Orial argue that—even on unit testing level—"branch coverage is not a good indicator for the effectiveness of a test suite." Whereas Gopinath and Ahmed state that it's a good predictor in the real world.

From my experience, I can say that it's helpful, although one should never rely solely on it. As I think Measuring code coverage in end-to-end tests? is related, consider the following answer by Sam Woods:

I always like to get code coverage for my functional tests, but not because I want to hit a certain percentage of code coverage. I like it because:

  1. It points me to areas of the code that are not covered.
  2. There are areas of the code that are very difficult to unit/integration test without having the entire system in place and doing end to end tests, so I like to compare the coverage from unit/integration tests to the coverage from my functional tests and see if there are things that should or could be covered in the end to end tests that may be more difficult in the earlier stages.
  3. I want to know which of my tests are equivalent, so I can look at what is covered by each test and see if there are tests I can eliminate or consolidate to be more efficient.

Regarding running these tests "once in a while, say, on a weekly basis": As already pointed out by alecxe, the observer effect is negligible for measuring coverage. Nowadays, one can safely include it in the continuous testing pipeline.

Improve this answer
4
  • well, of course there is continuous testing, but running every build with coverage is just wrong - one always should test the product as close to the state it's deployed in as possible - and the product never deployed with coverage.
    – shabunc
    Jul 9, 2017 at 22:35
  • @shabunc why is it "just wrong"? Coverage data is collected in the same build phase in which the automated tests run. If you think that state isn't a good reference, the same goes for the automated tests. I neither know the technologies you're using, nor how your build is configured, but it's usually no big deal to setup a coverage report—it's just another metric that can be handy.
    – beatngu13
    Jul 10, 2017 at 9:02
  • @beatgnut13 all coverage collection approaches I'm aware of are based on modifying original code that way it's possible to collect coverage. That said, technically it's different code.
    – shabunc
    Jul 10, 2017 at 9:18
  • @shabunc Indeed, coverage tools typically instrument the underlying code to gain visibility. But this is also true for many other tools. Furthermore, coverage tools are usually widely used (and tested) by the community. It's a solved problem and you shouldn't have issues finding a decent and sophisticated implementation.
    – beatngu13
    Jul 10, 2017 at 9:49
1

Yes it is a good practice.

I am familiar with coverage reports running for each and every build, both in branches and for master merges.
Same as full test suite, so we know immediately and who if coverage deteriorates / tests fail.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.