I have recently found one of my test is verifying only 3 of 10 fields of entity modified by the system under test.

I've added a couple of more assertions to the same tests. That obviously hasn't increased line coverage or branch coverage. Is there a way to measure oracles quality?

In research papers I found two related metrics:

but that's research and I couldn't find any tooling for measuring/reporting them for Python.

Are those metrics used in industry?


1 Answer 1


One alternative is mutation testing.

From the Pitest website:

Mutation testing is conceptually quite simple.

Faults (or mutations) are automatically seeded into your code, then your tests are run. If your tests fail then the mutation is killed, if your tests pass then the mutation lived.

The quality of your tests can be gauged from the percentage of mutations killed.

Pitest is for JVM, but most main programming languages have mature tools for it.

For more details, I suggest this talk by the Pitest main developer.

  • 1
    Thanks, this seems more practical are there seems to be more tooling for that, e.g. mutmut for Python.
    – dzieciou
    Commented Apr 15, 2022 at 5:40
  • You are welcome. If the answer is satisfactory, could you mark is as accepted? Commented Apr 15, 2022 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.