Return to Question

I have recently found one of my test is verifying only 3 of 10 fields of entity modified by the system under test.

I've added a couple of more assertions to the same tests. That obviously hasn't increased line coverage or branch coverage. Is there a way to measure oracles quality?

In research papers I found two related metrics:

• assertion coverage/checked coverage: percentage of statements directly covered by the assertions, proposed in "Assessing Oracle Quality with Checked Coverage"
• assertions count proposed in "Assertions Are Strongly Correlated with Test Suite Effectiveness".

but that's research.

Are those metrics used in industry? If so, what are challenges in introducing them?

I have recently found one of my test is verifying only 3 of 10 fields of entity modified by the system under test.

I've added a couple of more assertions to the same tests. That obviously hasn't increased line coverage or branch coverage. Is there a way to measure oracles quality?

In research papers I found two related metrics:

• assertion coverage/checked coverage: percentage of statements directly covered by the assertions, proposed in "Assessing Oracle Quality with Checked Coverage"
• assertions count proposed in "Assertions Are Strongly Correlated with Test Suite Effectiveness".

but that's research and I couldn't find any tooling for measuring/reporting them for Python.

Are those metrics used in industry?

I have recently found one of my test is verifying only 3 of 10 fields of entity modified by the system under test.

I've added a couple of more assertions to the same tests. That obviously hasn't increased line coverage or branch coverage. Is there a way to measure oracles quality?

In research papers I found two related metrics:

• assertion coverage/checked coverage: percentage of statements directly covered by the assertions, proposed in "Assessing Oracle Quality with Checked Coverage"
• assertions count proposed in Assertions Are Strongly Correlated with Test Suite Effectiveness.

but that's research.

Are those metrics used in industry? If so, what are challenges in introducing them?

I have recently found one of my test is verifying only 3 of 10 fields of entity modified by the system under test.

I've added a couple of more assertions to the same tests. That obviously hasn't increased line coverage or branch coverage. Is there a way to measure oracles quality?

In research papers I found two related metrics:

• assertion coverage/checked coverage: percentage of statements directly covered by the assertions, proposed in "Assessing Oracle Quality with Checked Coverage"
• assertions count proposed in "Assertions Are Strongly Correlated with Test Suite Effectiveness".

but that's research.

Are those metrics used in industry? If so, what are challenges in introducing them?

I have recently found one of my test is verifying only 3 of 10 fields of entity modified by the system under test.

I've added a couple of more assertions to the same tests. That obviously hasn't increased line coverage or branch coverage. Is there a way to measure oracles quality?

I've seen some research papers on reporting metric for that:

• assertion coverage aka checked coverage described in Assessing Oracle Quality with Checked Coverage
• assertions count described in Assertions Are Strongly Correlated with Test Suite Effectiveness.

but that's research.

Are those metrics used in industry? If so, what are challenges in introducing them?

I have recently found one of my test is verifying only 3 of 10 fields of entity modified by the system under test.

I've added a couple of more assertions to the same tests. That obviously hasn't increased line coverage or branch coverage. Is there a way to measure oracles quality?

In research papers I found two related metrics:

• assertion coverage/checked coverage: percentage of statements directly covered by the assertions, proposed in "Assessing Oracle Quality with Checked Coverage"
• assertions count proposed in Assertions Are Strongly Correlated with Test Suite Effectiveness.

but that's research.

Are those metrics used in industry? If so, what are challenges in introducing them?