I'm looking for advice on what tools/frameworks are available for monitoring our integration tests in terms of:
- Monitoring individual test metrics (mostly time spent but also other metrics found in the code) to obtain trends and statistics for individual builds and over time
- Configuring notifications on specific integration tests metrics that fail certain criteria (allowing me to say that test X should never have a metric Y that is above Z, notifying me if that happens)
The goal is to add another quality gauge to ensure that some internal metrics are not inadvertently sliding (an example could be a match percentage of an algorithm on a known test set which should never decline).
We are currently developing in
C#, bootstrapping tests using
NUnit as part of our
Nant based build in
CruiseControl (looking to migrate to
TeamCity) and could start developing our own internal stuff for this, but we are lazy developers and would like to build on top of other great tools.
I guess we are looking for a solution that allows us to (preferrably automatically) capture these individual metrics and run the quality checks after each build. We don't necessarily need to fail the build or the tests but we'd like the ability to go back and find the code changes that caused the quality metrics to decline.
Any recommendations, advice, experiences?