Monitoring and Notifications on individual and custom integration test metrics

Question

I'm looking for advice on what tools/frameworks are available for monitoring our integration tests in terms of:

1. Monitoring individual test metrics (mostly time spent but also other metrics found in the code) to obtain trends and statistics for individual builds and over time
2. Configuring notifications on specific integration tests metrics that fail certain criteria (allowing me to say that test X should never have a metric Y that is above Z, notifying me if that happens)

The goal is to add another quality gauge to ensure that some internal metrics are not inadvertently sliding (an example could be a match percentage of an algorithm on a known test set which should never decline).

We are currently developing in C#, bootstrapping tests using NUnit as part of our Nant based build in CruiseControl (looking to migrate to TeamCity) and could start developing our own internal stuff for this, but we are lazy developers and would like to build on top of other great tools.

I guess we are looking for a solution that allows us to (preferrably automatically) capture these individual metrics and run the quality checks after each build. We don't necessarily need to fail the build or the tests but we'd like the ability to go back and find the code changes that caused the quality metrics to decline.

Any recommendations, advice, experiences?

Answer 1

Sonar works well for this. We've been using it for about a year on a suite of products written in .NET. Monitoring metrics for individual tests is outside of the normal use case, but I believe it can be done if that's what you want. It's also quite straightforward to add your own plugin to work with custom metrics if you decide you need to do that.