Best practices to automate mobile apps on non-jailbreak device using SeeTest

Question

I am currently working on planning a test strategy for test automation for our mobile apps. The problem is that all the automation frameworks I've seen utilize a slightly different context than what the application really uses.

For example, I am evaluating SeeTest. To run automation tests on an non-jailbreak device the developer needs to add some frameworks to the project and actually build a different app the release app. So, when testing an instrumented iOS app with SeeTest, the app under test is not the same as the released app, which means that a full regression should be made on the released build.

This is the same of every framework I have seen.

How did others solve this issues?

Answer 1

I worked at a highly-respected mobile development shop for 6 years. We did not trust automation or emulators as acceptable tools to finalize a build. Each build was manually tested on the actual handset it was targeted for before the build was sent to the customer.

That's the problem with mobile development. Unless you find a framework that will run automation using the actual device, you won't have 100% confidence in your build, as you will never really test the deliverable build.

Answer 2

If you are talking iOS, you should look into the automated testing support supplied by apple in their tool called instruments.

Answer 3

Have you looked into Device Anywhere and Perfecto Mobile? They are both services for testing against real devices. A warning: These aren't cheap solutions.