The team I am currently working with has a very large application to test (several custom programs, running in a distributed environment), and has built up a very large set of automated test cases for regression and feature testing. These tests are large and there are a lot, so full test runs are dispatched across many machines, the results gathered, and then imported into a custom web app.
There are some very nice tools out there for writing the automated tests, but there doesn't seem to be much for managing a "automated testing workplace". We ended up writing custom applications to do all of the following:
- Schedule tests to run on one of several machines
- Gather results, import into database
- Web application for result viewing and querying
- Results per run/software version/test machine/etc
- Integration with external bug tracker for tracing failures to tests
- Nice to have: Smart analysis of results
- "This test started failing/passing"
- "This test is unreliable/intermittent"
Does anyone know of anything off the shelf that handles all of the above? Most of what I have found before is very focused on manual testing, which we don't do much of.