Test Manager. QC vs TFS

Question

My department is currently looking at a Quality Assurance test manager. On top of that we currently use Visual Studio and Team Foundation Server (TFS) on the development side of the house. We're looking to implement Quality Center (QC) or TFS as the test manager and potentially across the department as our Application Lifecycle Management (ALM).

My questions:

• What are your recommendations and why?
• What are the advantages and disadvantages of QC and TFS?
• Does Quick Test Pro or Selenium integrate with TFS and/or QC?

Answer 1

If the "development side of the house" is already using TFS, then you would be crazy to consider HPQC.

Both HPQC and TFS use per seat licensing, and you will have to pay double. TFS is a true ALM solution that includes source control, developer work item tracking, build support and everything that the developers need to do their jobs, and in the later versions added testing in their Microsoft Test Manager (MTM) products.

Personally I preferred to "roll my own" testing solutions with TFS instead of using MTM as it suits how I managed my teams better, but I haven't used the latest versions so I can't comment on that.

I doubt you will actually be able to successfully implement HPQC, as the developers won't support you as it will mean that they can't use the tools that they already have invested in and already use for requirements, work item and defect tracking.

TFS has great API support (and HPQC is pretty good), so as long as you are prepared to roll up your sleeves and write code, you will be able to make them integrate.

Answer 2

If you're already using TFS in-house it makes a lot more sense to use the Microsoft toolset for ALM and issue tracking (although I'd look at upgrading to 2013 - TFS management has been made a lot easier compared to 2010, and builds the testing side into the web interface)

That said, Microsoft Test Manager is (in my view) more geared to manual testing although it can be used to manage automation. MTM doesn't run without TFS, and I can't speak to its ability to manage virtual machines used to run automation because I haven't worked with that side of the application.

If you go with TFS and MTM, you'll have the benefit of cross-linking your developer code checkins through to test plans authored in MTM (which adds a lot of traceability) and developers will be able to link test items to specific code routines that exercise them (which is particularly handy for unit tests).

For automation, Microsoft's CodedUI doesn't match the way I prefer to operate all that well, and it does require a decent level of coding knowledge - but it is a powerful tool that is available in Visual Studio Premium and higher licenses (the same license level that includes Microsoft Test Manager).

You'd have the same or greater learning curve with HP QC/QTP and as Bruce said likely face issues with your developers not wanting to change from a product they're comfortable with.

Another benefit of TFS/VS/MTM that's in the later versions (I'm working with 2012) is that work item management can happen via the TFS web portal, through Visual Studio, or through MTM. Each location has its quirks, of course, but I've found I can do most of what I need in all three places - if I'm doing manual tests in MTM and run into a bug, I create it right there, in MTM. If I find a bug when I'm working on automation in Visual Studio, I create the bug from Visual Studio. Integration is as easy as giving the URL of the TFS server.

If you'd like some more information about what TFS can do, I'd suggest taking a look at Channel9's recordings of recent TechEd sessions (fair warning - there's a ton of these, each running about an hour, and you'll probably find a lot of them are useful)

Ritual disclaimer: I do not work for Microsoft. I'm not being paid to blow their horn. I just use their products because that's what the developers use.

Answer 3

If you have VS/TFS in house already, use the skills in the teams to help you set it up. Whilst QC is completely customisable, it is not real easy to customise the items you want

My company historically bought QC back when there was a central test team across the whole group, and it worked well. Since restructuring the test teams have all kept with QC, but the developers in my area all use VS, and the team use TFS for our backlog.

I have only used the TFS test bits in preview, but didnt switch due to the test repository we already have for manual testing, and the fact that we dont have to buy anything to stick with QC, but there is an option to use the synchroniser plugin for QC to keep bugs and requirements across both systems

Whatever you do, make sure you know how you want to structure your tests, and what reporting you need to produce up front as it is really difficult to change QC to get data right unless you configure it properly up front

Automation wise, QC has an open test architecture which in theory can run any automation with the right API, but I can't for love nor money find the documentation on how to use it yet.

If you are looking at integrating QTP, if your software is web based, your automation (and QC by the way) will only run on old Internet Explorer or Firefox versions

Answer 4

We are using QC for several years (8-10) now. And are currently planning to fully switch to TFS in the next 1 1/2 year. As TFS is used by all developers and testers (Scrum) now.

For me working with QC always was overall unpleasant:

• Crashes
• Sometimes lost test step data while writing the test
• Strange formating

Pro's about QC:

• You can make custom workflows
• Good for requirement based development with workflows
• External user access with limitation on what they can see

Cons:

• Old / outdated (Though I have seen a HTML5 client of it, maybe that one works better)
• Also orignally made for not agile development I guess
• Crashes / bug / quirks

TFS - As we are still writing our testplans in QC, I can not say too much about testing with it.

Answer 5

I work for itcentralstation.com, a review site for enterprise tech solutions. I found this review for HP Quality Center where the reviewer writes that the solution has bindings with HP QTP which is a test automation tool, with the addition of BPT (Business Process Testing).

Here are some additional excerpts from the review regarding improvements to the users organization and how the solution could be improved:

Pro:

HP QC makes it easier to maintain the scripts and also organize them into various business areas.

A comprehensive solution aiming for total Test management. It has bindings with HP QTP which is a test automation tool.

Con:

Backup strategy is lacking in QC.

Though we can create QC project files to create a backup, there is size limitation when you restore the backup.

http://www.itcentralstation.com/product_reviews/hp-quality-center-review-32263-by-prashanth-janakiraman?tid=thread-cm008-t017&cop=8136

Answer 6

TFS is just a simple tool to manage objects (called work items) as bugs, test cases etc. It allows some linking between them and you have to plan it well for what you will use it. It is just general ticket database tool accidentally used for testing. Advantage of TFS would be zero expanses as there are free downloadable client. I'm not sure about server licenses.

HP Quality center (HPQC) is well Integrated with Quick Test Pro (QTP) and is made for testing. There are more possibilities / more flexibility. Disadvantage is licensed numbers of paralel users.

I did not personally see Selenium integration for both those tools, but definitelly for HPQC could be coded via the java API.

About Selenium and TFS, there was already a question answered here: (Running Selenium test on TFS)