How to interpret / analyse code coverage results and provide suggestions for improvements?

Question

Knowledge level

I am looking for a how-to for a beginner, not so much for a expert or consultant answer like "it depends".

Feel free to add the "it depends" parts nevertheless.

Question

How would you proceed if you were tasked with:

"Analyse this code base and tell us where to do more testing to find bugs and to prevent introducing bugs in critical components in the future".

Procedure

Step 1: Get data

This part seems clear to me.

• Get the code base
• Run it through a code coverage tool
• Get a heat map of which parts of the code are well covered and which are not (example screenshot)

Step 2: Analyse the results

This is the part I need help with.

• A
  • Take the code coverage heat map
• B
  • Take the parts with the most dependencies?
  • Take the parts the software architects labelled as critical and important?
  • Take the parts that used to have lots of bugs in the past?
• C
  • Overlay the data from A with the information from B and find areas to improve
  • Draw conclusions

Step 3: Provide suggestions for improvement

As a manager I probably would expect a prioritized list of areas to work on and improve.

• Priority 1
  • Method X in class Y is called a million times and has low code coverage so we should increase coverage to 100% (100% coverage doesn't necessarily mean you've done a good job)
• Priority 2
  • Component Z has low code coverage and has had lots of problems and bugs, so we should test more there
• Priority 3
  • ...

Help required

Steps 2 and 3 are where I'd love to get some suggestions for.

Answer 1

Coverage should be done by developers, especially coverage by unit tests. You can track the numbers and suggest that if code is added to a module, test is added so new code is covered.

But there is nothing you can do (as QA) to increase code coverage by unit tests. So if you responsible for that, it is by definition an exercise in frustration (because you cannot control it).

If you have a suite of automated tests, matter of interests might be to check the coverage (separate from coverage by unit tests). And again, you may aim on increasing the coverage as the time passes. But again, number would be just some bogus metrics. It does not tell you anything about what input combination causes what code to be executed, if executed code make any assertions (and can fail if something is wrong) and failure in which modules will cause the most damage for your customers.

In testing, there is always more test cases to be tested than the time available. It is always about the compromise doing as good job as we possibly can in the time available.

"A" is QA does not stand for "Assurance" but for "Assistance". We provide additional input for managers to make business decision about releases of new versions. It is not the only input to make such decision.

Coverage can hint at the problem ("these wast areas of code have no unit test coverage"), but if someone tells you that 78% coverage is better than 75%, s/he is probably selling you some snake oil solution for a problem you don't have.

Answer 2

Code coverage of applications are more the responsibility of developers and not testers. For example it is them who are supposed to the white box tests (Unit tests in the code).

However as testers it is true we may be curious.

For Java code I advice you to have a look on SONAR: http://www.sonarqube.org
This tool will answer to your questions about what is critical or not. It is nice because it analyses the code and then sort the issue following how they are critical.

Answer 3

Fair warning: You're not going to be able to do this unless you're comfortable reading code. This is a technical task, which is why it's usually left to developers.

The main goal is to discover what isn't being covered. Any coverage tool ought to be able to identify what lines aren't being covered. Where your expertise comes in is in identifying what scenarios aren't being covered based on the lines. How would someone end up at the lines that aren't covered? What equivalence partitions are missing in the test cases? What paths are lacking?

The next big concept you can bring to the table is risk analysis. For each area that's not covered, what's the risk to the business if there's a defect in that area? Ultimately, if the risk is super low, it may not be worth the time to get adequate coverage. However, if there are critical pieces of functionality that are never being exercised, that ought to be a major red flag you can wave.

Answer 4

It is interesting that the question title refers to code coverage but the mission statement in the question refers to analyzing the code base.

If you plan to analyze the code base, consider doing some refactoring. One effective way to "prevent introducing bugs in critical components in the future" is to refactor the code to eliminate duplication (a common source of errors) and accommodate changes. You can never anticipate every possible change to a source base, but often you can rework the code so that common changes are easier and less-error prone to make. For example, if your library needs to interface with a variety of file formats, then by the time you have interfaced with 4 file formats, you can probably predict some of the changes you will need for the 5th file format.