Documenting application test coverage

Question

We have both automated and manual tests for our actively developed application. A regular Jira workflow in our case is something like this:

Open -> In Development -> In Testing -> In Automation -> Resolved

From release to release, we are usually behind in automating the new features and fixes. But, we do, of course, cover the new features with manual tests and test scenarios.

This leads me to the actual question:

How do you keep track of what functionality/features/screens/pages/fixes of the application are covered by an automated test and what are not?

The ideal situation would be to be able to see, at any given point in time, what parts of the application are covered with manual tests scenarios and what parts are covered with automated tests. Currently, we are trying to use a simple spreadsheet in a "checklist" style format, but it is not readable, scalable and maintainable at all.

Answer 1

The ideal situation would be to be able to see, at any given point in time, what parts of the application are covered with manual tests scenarios and what parts are covered with automated tests. Currently, we are trying to use a simple spreadsheet in a "checklist" style format, but it is not readable, scalable and maintainable at all.

Trello is a good option for testing management.

• what parts of the application are covered with manual tests scenarios and what parts are covered with automated tests. Using Trello, you can assign manual tests and automated tests under different column; moreover, you can color code them for easier visual identification.
• readable, scalable and maintainable at all. Comparing with a simple Excel sheet, Trello offers a better presentation for easier readability. You can add as many cards/columns to Trello as you need to, this takes care of scalability. Trello offers a range of actions such as batch delete, copy and etc, I have found it relatively easy to be maintained.
• Member function: With each card/task, you can assign none or all members of your team to it. If you are subscribed to a card, you will get an email whenever its status changes.
• Checklist function: You can create a checklist for each card; this adds one additional depth to task management.
• Browser-based: You no longer need to install Excel application to a PC, anyone can access it from anywhere anytime.
• User privilege hierarchy included: A member can be assigned as the team leader, only this team leader can perform actions such as card deletion. You may find this feature helpful from a managerial point of view.
• Plugin compatible: Trello is supported by its community, many plugins are free to use. Some of them are specifically for programmers, such as burndown charts and etc.
• Cost: It is free.

Answer 2

Approach for tracking automated test coverage:

1. Testers first write manual test cases. Each test case template has standard columns.

Add two more columns::

Automated? (with Yes and No options)

A sub field AutomatedScriptID appears, if Yes is selected in above field)

Revisited? (with Yes, No and Not Applicable options). Default value is Not Applicable.

2. Once manual test cases have been created and complete requirement coverage has been ensured, automation testers start updating the manual test case sheet and select the value for Automated? field for which the default value is No.

3. Once Yes is selected in Automated? field, automation tester has to mention the automated script id in AutomatedScriptID field.

4. Now, as the testers test the functionality, sometimes test cases fail because of bugs. Test team mention the BugId in Comments column (which is part of every standard testing template) and fail the test case. And also sets the Revisited? column to No.

5. This Revisited? column will be updated by automation testers only. Even if the manual test case passed.

6. Now even if all the test cases have passed during manual testing. It can clearly be identified that there are test cases that need to be revisited by the automation testers as they might have to update automation scripts because of the code changes made in order to fix the bugs reported for those test case.

7. Once the automation testers revisits a test case that had associated bugs, he checks the corresponding AutomatedScriptId and analyze it if any change in the script is required on not. And then he takes the appropriate action. After that, the Revisited? column is set to Yes; which means that the corresponding automation script is up to date now.

8. Charts are created on the first sheet to show:

A. What all test cases have been covered/not covered/partially covered by automation testing?

B. How many test cases are pending to be revisited by the automation testers?

Note: Using this data, one can create more charts as per your project needs.

Benefits:

1. At any point in time, the team is aware of the status of complete coverage of test cases in manual scripts and complete tracking is taken care of.

Drawbacks:

If using excel, automation tester has to be informed that a particular manual test case have failed because of bugs. But, if the team moves to some tool, this is taken care of as the tools sends a notification to the team as soon as any new bug is created or test case fails. So, all the stakeholders are updated. Some of our testing teams are using Jira and they are fine with it.