1

I've been tasked to identify gaps in the regression test coverage of a large C# based data management system that comprises a SQL database, server application, multiple web clients and multiple windows forms clients as well as a SOAP based web service API.

There are a large number of manual black-box regression tests that have evolved over the 15+ years this system has been evolving but there are gaps in the coverage. We are in process of automating these test cases with Selenium and White based test frameworks.

I wonder if anyone can suggest a good analysis technique to identify the gaps in coverage.

I'm also keen not to duplicate existing white-box integration test coverage of business logic when we come to automate the ~150 manual regression test scripts.

1

Two basic indicators of coverage are:

  1. Code Coverage: The analysis of which parts of the code were exercised by tests. In C#, there many tools, such as dotCover, OpenCover, and NCover. Check out a description of them here.

  2. Mutation Testing: The analysis possible changes in code (introduction of bugs) that do not trigger a failure in a test. In C#, as far as I know, the only mature tool is VisualMutator.

For the "manual" test cases, you can look at the options for instrumentation for your tech stack, in particular the options for tracing.

Alternatively, you can create a Requirements Traceability Matrix, and use it as a reference for brainstorm risks of uncovered areas.

enter image description here

  • Thanks João Farias, those type of tools will help to analyse the white-box tests but as for figuring out the coverage gaps in the large number of manual black-box tests, I wonder if anyone can suggest a good analysis technique or strategy for that? – Stuart Pawsey Jul 30 at 10:25
  • Updated with some more thoughts. – João Farias Jul 30 at 10:39
1

Extending on João Farias's answer Code Coverage:

What I have done in the past is instrument/start the deployed application with OpenCover. Run all automated and manual test-cases. Combine all coverage files (unit, integration, manual tests) and generate a single report. This makes the manual tests somewhat slower, but now you can see which code is not covered by tests and which branch paths are not taken.

Based on this data I decided which area's needed more manual testing. This because we didnt have a manual test regression plan. So we just explored, looked at the data and explored some more. It was a 10 year old legacy application in C#, VB.Net that was released yearly.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.