Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

There must be many questions like this on the site but I have some unique requirements that affect my choice of test management / QA process management system. This is for managing the test cases and recording results, not for performing the actual tests, which will be done manually and with other automation tools.

  1. We're building tests for a website that has at least a few changes in behavior every day. So if there are 1,000 tests, at least 5 will definitely break every day because of a new feature or design change. I mention it because I don't want to use something that assumes once a test case is defined it will never change, or changing it requires a formal review process.

  2. I want to write a test case 1 time, then record many variations of the same result. One for each device type, browser, OS, variations in other. For example, the "User Sign Up" main test case is defined once, then the result recorded every day for each browser / device / OS combo.

  3. I'd like it to be web based. Also with some kind of API to let our automated test tools (which mainly use Selenium) to post test results as they happen).

share|improve this question

1 Answer 1

There are a number of systems that can handle this. I'm familiar with Test Link and TFS, both of which have APIs (although the TFS API is a little challenging - speaking as someone who's worked out how to post test results to TFS through the API) and both of which have ways to handle test cases that need to change.

A lot depends on your budget - Test Link is free (as in open source, and as in beer), TFS is not (Microsoft Team Foundation Server, if you're not familiar with the abbreviation). QA Complete from SmartBear is another commercial tool that offers the functionality you're looking for.

Here's how these tools (and I suspect the other major commercial offerings) work: You define one or more test environments - this covers your device types, browsers, OSs and so forth. Depending on the tool you define the environments required at the test suite, test plan, or test case level. The API to communicate the tool varies, but can be worked with. For TFS it needs a fair amount of code and assorted other dependencies taken care of before you start the tests. Test Link also has dependencies that need to be set up in the tool before you start.

For test case updates, the usual arrangement is that the test case is versioned once it has been run (that is, there's a result posted to the tool). The rationale is that if the test case has to change, the historical results will be invalid unless you retain the history of the test case itself. Updating changed test cases is usually a matter of creating a new version (and incrementing the test case versioning), then running against that until there's another change.

Depending on the tool, you may be able to break down your test cases into reusable steps that minimize the amount of re-versioning you need to do.

I know Team Foundation Server allows this through what it calls "Shared Steps" - which are pretty much what they sound - one or more steps in a test case which can be used by multiple test cases. (Caveat - Team Foundation Server needs to use Microsoft Test Manager for have test suites and test plans defined - currently it's not possible to create these through the TFS interface - and MTM can't be used without TFS).

I'll often set up a single step as a shared step so it can be used by any test case (such as "click the logon button" which doesn't care what the user credentials are). You can also parameterize steps so that instead of a whole lot of steps for each different value your tests will enter into a particular field, you have one step to enter "@value" into the field, with the values that can be used defined by the test case or an attached data file.

Test Link doesn't have that level of reusability, that I know of. I don't know enough about other tools to offer any suggestions.

There is a good listing of test management tools available in the Test Management section of the Ministry of Testing's test tool listing. (It's a long list... you'll need to do a fair amount of research in this)

On the plus side, most of the commercial tools do offer some form of trial. I know Microsoft has a free service for TFS that they host (Team Foundation Service) and contains the most up-to-date feature set. That's free for up to 5 users per account - I don't know any other details.

Good luck finding something that meets your needs.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.