2

We would like to shift quality left and motivate developers to write integration tests - both narrow and broad. The biggest challenge is to introduce metrics we could use for measuring the effectiveness of the tests. Code coverage is out of scope as it does not tell much in integration test space. We thought of some sort of limited coverage measurement - checking if collaboration points like app to database, app to external service are covered by tests. Another option was to check how the test passing rate changes if we mutate the response from external dependencies. Is there any tool supporting to extract such metrics or is there a better approach to check effectiveness? Our domain is Java.

Update: The metric is used to give feedback to developers on their progress, later we want to use some of the metrics for setting quality gates

Improve this question
4
  • Regarding your update: it still strikes me as odd to "give feedback on the progress" - if you know what constitutes a good test, why not proactively outline it during sprint planning (with a tester) so it's done right from the start? Cfr. my answer.
    – FDM
    Nov 4, 2021 at 13:53
  • We are talking about dozends of projects, many without QAs. We want to have a high level transparency how differen product areas progress on integration testing
    – HamoriZ
    Nov 4, 2021 at 13:58
  • That would have been useful info from the start. ;) Although having many projects without QA sounds more troublesome than a metric. Good luck!
    – FDM
    Nov 4, 2021 at 14:21
  • By having metrics , what problems you are aiming to resolve? Let's begin with the end in mind..
    – Vishal Aggarwal
    Nov 10, 2021 at 11:04

3 Answers 3

Reset to default
6

We thought of some sort of limited coverage measurement - checking if collaboration points like app to database, app to external service are covered by tests.

Most code coverage tools allow filtering by file name or location; you can create multiple reports or checks for certain areas of your codebase. Here is how Jacoco does

Naturally, these would not have a big demand if you do TDD, since you will only be able to write integration code if you have a check to drive you.

Another option was to check how the test passing rate changes if we mutate the response from external dependencies.

You can use tools for Mutation Testing and Property-Based Testing, e.g. PIT or JUnit Quickcheck.

Improve this answer
3

Can you detail why you need those metrics? It sounds like you just want to have metrics - but do you have any plan with them afterwards?

Personally, I'd first focus on getting the integration tests on point on a story-by-story basis. That is, during implementation, have the developer and tester sit down to discuss what is being unit tested, which cases should be handled as integration test, and what shouldn't be automated at all. If the team takes these tests seriously, they will (and should be) relevant and done right from the start.

Then, if after a while you notice that specific issues slip through - use that feedback to fill out any gaps in the test coverage or test implementation.

Improve this answer
1
  • +1, for putting why first before how.
    – Vishal Aggarwal
    Nov 10, 2021 at 11:07
1

Could a code review checklist help?

Something like:

Impacted Integration points:
 
 - external service a [ ] 
 
 - external service b [ ] 

Did you include integration tests for impacted services?

Often integration test classes end with *IT. Java

Could you mandate one is included if integrations impacted? Could you count the test methods?

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.