I am testing a product which uses very low level file system operations. It is crucial that this product works with external file system devices like Windows phones (using the MTP protocol). There is the MTP Device Simulator and I am wondering if this is a reliable tool for integration testing. If not, do you know of any limitations? Would you rather prefer testing with an actual device? Are there any alternatives?
Emulators/Simulators are sometimes good and sometimes not. You can look at sauce labs https://saucelabs.com/ which has some to offer for various devices and there are some other companies as well. Some utilize actual devices which gets expensive. Frankly I have found through a cost analysis that it's cheaper to buy a few devices to test directly from than to utilize an external company for the real devices. The simulations will test most of what you want to test, but then you will need a real person playing with the device itself to get the usability testing part.
As far as doing the automation testing I would try to get the simulator to work as it would likely cover 80+% of the testing that needs to be done. Check out reviews to be sure other people find that it's accurate and also do a sanity check on a real device to see if there is any visible difference. Each simulator has it's own nuances and issues...some better than others. Try to utilize a simulator instead of emulator as the emulator tends to not account for the hardware specifics much and focuses on the software part, where the simulator tries to account for a simulation of the actual device including hardware specs.
Here is a site that lists a sampling of tools and companies. http://www.testingtools.com/mobile-testing/ Be sure to really research as the mobile automation testing is rather newer and there is alot of competition resulting a wide variety of good and bad. Find your specific niche that is best for your team and don't settle. Always test at least some on the actual device before release as there are always nuances that don't 100% show up in simulation.
As far as the MTP Simulator goes, I personally haven't utilized it, but it's a simulator and part of the windows DEK so I think if you were using a simulator that's probably the best you will get for that device. Apple/Android have been far ahead of Windows on the simulators for mobile devices. A few years back they didn't have anything, so something is better than nothing, but don't ignore the value of actual device testing.