We write our manual test protocols in an in-house XML format, and have a simple tool for executing them electronically (vs. paper and pencil) and saving the results.
This has worked well, but we're struggling with the test management side of things. Currently we have scripts that:
- Show which requirements are traced to which (already written) tests
- Generate a summary of tests that need to be executed
- Summarize the pass/fail results for tests that have been executed for a given test round
However, these are rather ad-hoc, and we don't have a good way of managing things or answering questions such as:
- Test authoring
- Have we written a test for feature X yet?
- How many planned test cases do we have left to write?
- Test results across test rounds
- Does this particular test have a history of failing?
- Are we ready to re-execute the tests that failed during the last round?
- Which tests should we run in the next round?
We've tried using JIRA, but since there's no connection between our actual protocols and their associated JIRA items, there's a lot of manual work to keep JIRA in sync, and it inevitably falls behind.
From what I've seen, a lot of test management tools require you to write and run your manual tests in their tool. We would like to keep our in-house format, or at the very least a text-based format (as in, the source files should be text-based; exporting to a text-based format doesn't count) that we can keep in source control and easily grep, edit, etc. along with our code.
Is anyone aware of any test management tools that support integrating existing manual tests without migrating them to the tool itself?