Assertion coverage and assertions count in industry

Question

I have recently found one of my test is verifying only 3 of 10 fields of entity modified by the system under test.

I've added a couple of more assertions to the same tests. That obviously hasn't increased line coverage or branch coverage. Is there a way to measure oracles quality?

In research papers I found two related metrics:

• assertion coverage/checked coverage: percentage of statements directly covered by the assertions, proposed in "Assessing Oracle Quality with Checked Coverage"
• assertions count proposed in "Assertions Are Strongly Correlated with Test Suite Effectiveness".

but that's research and I couldn't find any tooling for measuring/reporting them for Python.

Are those metrics used in industry?

Answer 1

One alternative is mutation testing.

From the Pitest website:

Mutation testing is conceptually quite simple.

Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived.

The quality of your tests can be gauged from the percentage of mutations killed.

Pitest is for JVM, but most main programming languages have mature tools for it.

For more details, I suggest this talk by the Pitest main developer.