Measuring feature coverage with BDDs

Question

I'm looking for approaches to measuring feature coverage (however you want to define that), and how BDDs can help (or be a neutral, or even negative factor) in such measurement.

My question is: do you measure feature coverage, and how do you do it?

In particular, I assume that we in our QA function will have to do a fair amount of work to fill in specifications. We don't have:

• Well defined acceptance tests
• Product owners who define user stories
• Developers working on stories in a single sprint tracked through JIRA tickets (or similar)
• Any catalogue which relates stories to features

We do have:

• QAs who understand our product really well, and are smart. One is writing automation tests; the others should be able to get up to speed with BDD writing.
• Automation testing can catch regressions
• SDETs who write automation tests

I don't have a special definition of feature coverage, although I'm very open to the idea that maybe I should have such a definition.

Answer 1

We need to manage our tests a hierarchical way. This is necessary to keep a proper check on the coverage of any feature. We should have a separate feature for every page in our application. Lets take an example to understand the feature coverage more clearly.

Lets assume that we have an website to manage some employees of a company and there are following modules:

1. Admin Account

1.1. Add Employee

1.2. Edit Employee

1.3. Delete Employee

We need the test coverage for above modules. So the we can manage these modules in our tests by having some directory and sub-directory. Each directory will be reflecting a module or tab of the website.

Test coverage for above modules can be covered by having following setup

1. To cover "Add Employee" under the "Admin Account" module, set the hierarchy as shown below

Admin (directory 1) > Employees (directory 1.1)> Add Employees (directory 1.1.a) > addEmployeeDetails.feature(your test file name)

In the file add the path "Admin : Employees : Add Employees : Add Employee Details" as a feature description on the top of the file.

2. To cover "Edit Employee" under the "Admin Account" module, set the hierarchy as shown below

Admin (directory 1) > Employees (directory 1.1)> Edit Employees (directory 1.1.b) > editEmployeeDetails.java(your test file name)

In the file add the path "Admin : Employees : Edit Employees : Edit Employee Details" as a feature description on the top of the file.

3. To cover "Delete Employee" under the "Admin Account" module, set the hierarchy as shown below

Admin (directory 1) > Employees (directory 1.1)> Delete Employees (directory 1.1.c) > deleteEmployeeDetails.java(your test file name)

In the file add the path "Admin : Employees : Delete Employees : Delete Employee Details" as a feature description on the top of the file.

Here, we have added the path of the file in comments of every file. This will help you to locate your files easily. Splitting your website in above mentioned hierarchy will tell help you keep a track that which module of your website has been covered with BDD tests.

For example, lets cover some scenario for "Add Employee" under the "Admin Account" module.

In addEmployeeDetails.feature file

Feature: Admin : Employees : Add Employees : Add Employee Details
  In order to verify new employees can be added in the system
  As an administrative user
  I want to be able to add new employees in the system

  Notes:
  - Verify that validation messages are shown if invalid characters are used in add new employee form
  - Verify that new employee is added in the system if valid information is given

Scenario: Verify that validation messages is shown if all fields are left blank
Given Admin is on "Add New Employee" page
When Admin clicks "Submit" button leaving all fields blank
Then Admin gets validation message that says "Fields can't be blank"

Scenario: Verify that validation messages are shown if special characters are used in date of birth field
Given Admin is on "Add New Employee" page
When Admin fills "$%^*&^%$#" in "Birth Date" field
And Admin clicks "Submit" button after entering valid data in rest of the fields
Then Admin gets validation message that says "Invalid birth date"

Scenario: Verify that employee is registered if valid information is filled
Given Admin is on "Add New Employee" page
When Admin fills "Test Employee" in "Name" field
And Admin fills "12-05-1985" in "Birth Date" field
And Admin selects "India" from "Country" drop down
And Admin clicks "Submit" button
Then Admin gets message that says "Test Employee is added successfully"

So, In the above feature file, we have a section named Notes, under which we write description of all the test cases for which coverage is required. Thus the feature file should consists of scenarios that fulfills the Notes. This way you can check the coverage for all the tests.

Feature coverage largely depends on what the web page consist of. As you don't have Product owners who define user stories, your QA team needs to decide which scenarios should be covered by BDD tests. Generally, they should add scenarios which covers the feature spec documents.

Answer 2

Let's separate out BDD and feature coverage. BDD is just a way to name your tests. And we can cover features irrespectively of the test names. So the question boils down to "how do we measure feature coverage".

Here is one of the ways:

• Use reporting tool that allows linking your tests to the requirements (e.g. Allure).
• Actually link your tests. In Java + Allure it will look like this: @Features("BLAH-12 My Feature1") and @Stories("BLAH-13 My Story1"). If you have the ability - add them to your issue tracker. If not, just keep them in the constants in your code.
• Write test names stubs (only names without implementation) for 100% coverage of the feature. Of course different people would come up with different set of tests, but to your best knowledge that should be 100% coverage.
• Implement the tests that you have time/skills to implement, others will be left not-implemented.

Now after you generate the report the tests that you implemented are going to be green (hopefully) and the tests that are pending to be implemented are going to be blue. That would:

• List the whole functionality (if that's a BDD) or the whole set of the tests which comprises 100% coverage. Per each piece of the app.
• Highlight the coverage - you'll see which tests are implemented and which should be covered manually

Answer 3

You question is: do you measure feature coverage, and how do you do it?

My answer is: yes, I ask the knowledable users to train me or document them.

Consider functions by user type such as regular user and admin

Consider functions by area such as admin vs regular

Consider functions by web page or web site area

For each area, one approach is to divide up according to happy, sad and optional

• Happy is the minimum workflow a customer will do and assume they enter all fields, use defaults when possible and enter all data correctly the first time through. This tends to be a relatively small set of test(s).
• Sad is when the user messes up and either fails to enter data or enters invalid data. This tends to be long, complicated and involve testing a lot of different conditions and data.
• Optional is when the user had the option to enter optional / additional data. This will actually have both happy and sad paths. The specifics will be particular to the business domain the organization is working in.