6

I am currently undergoing black-box testing campaigns for applications that require file input in linux environments. For that I need sets of test files of different formats that can be used to be passed as input to the different applications I am interested in.

To monitor the code coverage for a specific set of testing files, I am using lcov/gcov. The problem is that a set of files that has decent code coverage can become rather large, therefore taking a lot(and I mean a lot) of time to complete the tests using such a set. So I would like to be able to minimize such a set, while keeping the same code coverage (in other words eliminate the redundant test files that stress the same part of an application).

I tried using the minset tool from the peach fuzzing framework, which would be exactly what I would need but unfortunately it can be only used on Windows and as far as i have tried cannot be ported on linux-like operating systems.

So, do you guys know any tool that could do what I have described above. Or perhaps have an idea on how I could achieve that. Thanks a lot.

Improve this question
2
  • 1
    I don't know of any tools that use code coverage metrics to help you select a reduced test data set. There are ways to generate a reduced test data set for combinatorial testing problems. See sqa.stackexchange.com/search?q=combinatorial+testing for details.
    – user246
    Commented Oct 19, 2013 at 12:40
  • 2
    biggdman, note that maximizing code coverage is not the same as maximizing test coverage. You may have ideal code coverage while still important test cases not covered.
    – dzieciou
    Commented Oct 20, 2013 at 8:49

2 Answers 2

Reset to default
2

You could implement your own version of minset that would be useful to anyone that uses gcov/lcov or, with a small modification, also to users of other code coverage measuring tools.

  1. Export gcov/lcov report to some processable format, e.g. XML. The report should describe coverage per method/class/package (granularity is up to you) for each test you executed and its execution time, e.g., as a quadruple:

    (test, package, coverage, executiontime).

  2. Describe it as an optimization problem and find corresponding algorithm that will work on your exported data. The optimization problem is that by selecting a subset of tests you want to

    • maximize the test coverage per package (or method, class),
    • while minimize/limit total test execution time to some value.

This sounds a bit like a knapsack optimization problem, where you try to limit the total mass (here test execution time) of items put into a knapsack while maximizing their total value (here test coverage). The only additional requirement is you are trying to maximize local value (coverage per package). But I'm sure there are such variations of the knapsack problem and so proper algorithms.

Stackoverflow users may be of more help here or authors of minset method in the peach fuzzer tool you mentioned.

Improve this answer
1

Here's a tool that allows you to get code coverage data per test case and then compare against other tests or all tests: http://www.semanticdesigns.com/Products/TestCoverage/JavaTestCoverage.html

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.