What are the Differences in Test Management with TFS vs MTM for 2017 vs 2012?

Question

What are the Differences in Test Management with Team Foundation Server, Visual Studio, & Microsoft Test Manager for 2017?

testing activities processes as of 2012 are:

1. Create backlog items (in TFS)
2. Create tasks (in TFS)
3. Design test cases (in TFS, VS, or MTM)
4. Link cases with tasks (in TFS, VS, or MTM)
5. Associate automation script (in VS)
6. Create test plan (in MTM)
7. Execute test cases (in MTM)
8. Check result (in MTM)

What has changed for 2017 besides TFS being renamed to VSTS ?

Answer 1

A lot. The last I heard, MTM will be deprecated once the test-specific functionality of the TFS web portal is available.

• Create backlog items (in TFS) - the UI is a lot better, and there's been a lot of improvement to backlog item management and iteration management.
• Create tasks (in TFS) - see above.
• Design test cases (in TFS, VS, or MTM) - the TFS UI is much better with this, since the UI improvements for tasks and backlog items carry to test cases
• Link cases with tasks (in TFS, VS, or MTM) - The TFS UI is by far the best of the three choices now, in my opinion.
• Associate automation script (in VS) - this hasn't changed.
• Create test plan (in MTM) - TFS now allows creating and editing test plans, with a better UI
• Execute test cases (in MTM) - TFS manual test execution is cleaner and easier on system memory than using MTM. For automated tests, the TFS build and release process works brilliantly, and as long as there are test cases associated to test code, the results will be published and easily accessible. (They are still published without the connection, but a little less straightforward to access)
• Check result (in MTM) - Can be done in TFS web portal much more cleanly using the TFS build/release process - I currently have an automation process doing an overnight build and release, which emails me the outcome of the tests it runs. From there, I can open the results file and check anything I need to check.

TFS has not been renamed to VSTS: VSTS is the Microsoft-hosted version of TFS, and as such is actually more up-to-date than TFS2017. You can find a summary of what is available in which version of TFS vs VSTS here.

(Disclaimer: I am not a Microsoft employee. I use TFS2017 at work)

Answer 2

Microsoft now recommends using the Online TFS Test Hub over Microsoft Test Manager Stand Alone Client for all test management requirements as they’ve been slowly moving all the features from the stand alone client to web @ https://docs.microsoft.com/en-us/vsts/manual-test/mtm/guidance-mtm-usage?view=vsts

• Run with option Data Collectors via the Online TFS Test hub will launch MTM if selected and provides: Event Logs, Action Logs, Screen & Voice Recorder, System Info (additional logging)
• Online TFS Test hub via Web Browser based runner does not give these options
• TFS Online Test Hub now has Charts & Results Tabs

The biggest change is that pretty much everything’s moved to the web and you no longer have to fight your way through MTM’s terrible user interface design. The web interface makes sense. There’s no longer an artificial separation between planning your tests, running your tests, and tracking the results – everything is in one, unified interface.

The second biggest change is that Lab Management is done. The build system that was at the core of Lab Management has changed. The deployment features of Lab Management have been moved in to TFS Release Management. TFS Build vNext and TFS Release Management work in concert to get stuff deployed on to servers and the integration with the QA tools are now entirely optional.

You’ve now got two options for running tests:

1) run test cases using the web interface

2) run test cases using the MTM client.

You still have the option to run tests using the MTM rich test client. This makes sense because the really handy stuff from the MTM Test Runner was the ability to create really descriptive bugs that contained screenshots and video recordings. Microsoft split that test execution functionality into its own EXE so you can still do all that great stuff.

Using the MTM Test Runner also lets you continue to do lightweight test automations using MTM Action Recordings. The idea with Action Recordings is that a non-technical QA person would be able to create automated tests that help speed up the QA testing process and eliminate QA testing tedium. The Action Recordings feature makes for some awesome demos. In real life, it’s still pretty limiting and brittle. It only works with certain types of Windows application technologies and for web apps it only works with Internet Explorer.

If you want to do more free-form exploratory testing, there’s a new Exploratory Testing client for Chrome. It’s pretty good and it gives you the ability to do screenshots and video recording, too. Last time that I checked, it was still in beta and as such it still had some sharp edges. Microsoft is definitely going in a good direction with that plugin though.

Another thing that’s missing (more like deprecated) is the Associated Automation feature. That used to let you hook a Visual Studio Coded UI test directly to a Test Case and then have it run as part of a Lab Management build. Since Lab Management is pretty much done, this means that Associated Automations are just about done. I wouldn’t sweat this too badly though because running just one Coded UI test is kind of a weird choice in the first place and it makes a lot more sense to just hook your entire Coded UI Test suite in to your vNext Build or Release Management flow. If you do that, the results of those tests aren’t tied to a specific Test Case but you still get to see the results in the Test Hub’s “Runs” tab.

Testing activities processes as of 2017 are:

1. Create backlog items (in TFS)
2. Create tasks (in TFS)
3. Design test cases (in TFS, VS, or MTM)
4. Link cases with tasks (in TFS, VS, or MTM)
5. Associate automation script (in VS)
6. Create test plan (in MTM or TFS)
7. Execute test cases (in MTM or TFS)
8. Check result (in MTM or TFS)