How to understand existing test coverage?

Question

I believe this is a common issue for any QA that enters a project with existing test base.

I entered the project which have lots of unit tests and FIT tests for existing functionality. When I test a certain defect or new sub-feature added to existing functionality, I try also to understand if existing tests cover the application domain well. That's hard to trace. There is no test management software in use so far, and many tests contain detailed data without high-level descriptions ("The goal of this test is to..." or "We vary XML input files by variable X..."). The good thing is catalog and file names for test cases are quite descriptive.

I have two complementary ideas to understand test coverage:

• Systematic: For a small feature under tests, I create matrix of possible test cases (e.g., combination of possible input variables) and compare it with space covered by existing tests.

• Ad-hoc: An idea about possible missing test comes to my mind and I retrieve the existing test base for it.

However, both ideas hard to apply and time-consuming for two reasons:

1. The domain of possible variables is complex and many variables are interrelated.
2. It is hard to map my combinations matrix to the way someone else classified the test so far, and from this classification it is not straighfoward to see all dimensions already tested.

Do you have any approach to understand the coverage?

Answer 1

Are the tests at least categorized by product, feature and separated by unit/integration/functional? If not, that would be a good place to start to start before trying to figure out what coverage they provide.

One possibility would be to run code coverage on your product while executing the tests. This could give you a good indication of what the tests do and do not cover. You could then run your new test matrix and see what differences there are.

It won't help you much in terms of finding equivalency tests (multiple tests that are unnecessary because they are only exercising behaviors that were already exercised by other tests). You can however execute a single test and see the code coverage for that individual test and compare it to the overall or other tests. I'm not sure off the top of my head which code coverage tools allow you to do this.

While it may be a big project, I would be willing to bet that on a very large set of tests, taking some time to understand and trim it down would in the end save you time, potentially save a lot of pain (you don't want to rely on tests that don't provide the coverage you actually need) and be worth the effort.