2

I'm looking for advice on what tools/frameworks are available for monitoring our integration tests in terms of:

  1. Monitoring individual test metrics (mostly time spent but also other metrics found in the code) to obtain trends and statistics for individual builds and over time
  2. Configuring notifications on specific integration tests metrics that fail certain criteria (allowing me to say that test X should never have a metric Y that is above Z, notifying me if that happens)

The goal is to add another quality gauge to ensure that some internal metrics are not inadvertently sliding (an example could be a match percentage of an algorithm on a known test set which should never decline).

We are currently developing in C#, bootstrapping tests using NUnit as part of our Nant based build in CruiseControl (looking to migrate to TeamCity) and could start developing our own internal stuff for this, but we are lazy developers and would like to build on top of other great tools.

I guess we are looking for a solution that allows us to (preferrably automatically) capture these individual metrics and run the quality checks after each build. We don't necessarily need to fail the build or the tests but we'd like the ability to go back and find the code changes that caused the quality metrics to decline.

Any recommendations, advice, experiences?

1 Answer 1

2

Sonar works well for this. We've been using it for about a year on a suite of products written in .NET. Monitoring metrics for individual tests is outside of the normal use case, but I believe it can be done if that's what you want. It's also quite straightforward to add your own plugin to work with custom metrics if you decide you need to do that.

2
  • I was hoping for a more lightweight framework than SonarQube for our scenario but I'll mark this as the answered since no other answers have surfaced Jun 30, 2014 at 7:54
  • I've been hoping for a more lightweight tool as well, but sonar is pretty easy to set up and works really well. The nice thing is that you can pass it things like .trx files that come from your build process, so it's just kind of collecting and displaying that data. Jun 30, 2014 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.