I am currently working on planning a test strategy for test automation for our mobile apps. The problem is that all the automation frameworks I've seen utilize a slightly different context than what the application really uses.

For example, I am evaluating SeeTest. To run automation tests on an non-jailbreak device the developer needs to add some frameworks to the project and actually build a different app the release app. So, when testing an instrumented iOS app with SeeTest, the app under test is not the same as the released app, which means that a full regression should be made on the released build.

This is the same of every framework I have seen.

How did others solve this issues?

3 Answers 3


I worked at a highly-respected mobile development shop for 6 years. We did not trust automation or emulators as acceptable tools to finalize a build. Each build was manually tested on the actual handset it was targeted for before the build was sent to the customer.

That's the problem with mobile development. Unless you find a framework that will run automation using the actual device, you won't have 100% confidence in your build, as you will never really test the deliverable build.

  • Both Android and iOS have automation frame work that runs on the actual device.
    – Guy
    Commented Dec 18, 2012 at 8:06

If you are talking iOS, you should look into the automated testing support supplied by apple in their tool called instruments.

  • I know about instruments and iOS automation FW. Android also has such FW. The problem is that both tests in a slightly different context then the real world. it is like the schrodinger's cat paradox...
    – Guy
    Commented Dec 18, 2012 at 8:10

Have you looked into Device Anywhere and Perfecto Mobile? They are both services for testing against real devices. A warning: These aren't cheap solutions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.