Take the 2-minute tour ×
Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It's 100% free, no registration required.

I am looking into the UI test automation options for hybrid apps (Cordova PhoneGap) on both iOS and Android. So far I still struggle with finding any decent tool. I may consider proprietary product (TestDroid, SilkMobile, Ranorex or so..), but first I would really like evaluate available open-source toolset.

Yeah, there are many mobile testing frameworks already such as UIAutomator, Robotium, Espresso for Android, UIAutomation or KIF on iOS. Problem is those tools are closely bound to specific platform.

So I favor idea of multiplatform tools such as Calabash, Appium, MonkeyTalk. The latter two seems to be most popular and feature-rich alternatives.

I tried them both, however, I found none of them mature enough, each of them having a lot of "infancy" issues. Appium's concept is really nice but even now, with version 1.0, it is being struck down by poor documentation, outdated examples and some hybrid app issues (selendroid dependancy - there were reported issues with coordination between Appium and Selendroid API). It is simply painful to setup locally (and I cannot only depend on Sauce Labs as one needs to debug the tests locally).

MonkeyTalk seems more user friendly and better documented, though I experienced some issues as well (crashes, instrumentation failures) + some of the better features are not available for iOS (on-the-fly instrumentation..). The better equiped Pro version is still Beta after all.

It seems there is no ultimate solution for hybrid apps automation as of yet. The whole field of mobile testing looks still, surprisingly, undeveloped. While I favor the idea of having one tool for all mobile test automation, the most critical requirement is still "to have a tool that offers best test automation capabilities for certain app". If this means I need to use two or even three different tools best suiting specific mobile platforms, so be it.

But I cannot be the only one here, trying to solve this dilemma, so I wonder what has your experience been in building the best possible infrastructure for testing mobile hybrid apps?

Please no self-promotion of test automation tools vendors.

share|improve this question
    
I might want to add there already is one fairly straightforward solution for hybrid apps automation, I have been using so far, and that is simply using selenium webdriver with the mobile app running in Ripple UI web emulator. This is not good enough though as you're not testing the real thing, platform-specific issues will not be caught by those tests.. –  VojtaBurian May 15 at 13:12

4 Answers 4

I have used eggPlant by TestPlant with success.

Pros:

  • Can be used on pretty much any platform that you can remote into
  • Has a very good image compare solution with good flexibility/resiliency
  • I have used this with great success on iOS, Android, Linux, Windows, Mac

Cons:

  • Uses image compare
  • Uses a proprietary language
  • Expensive

I have also used Appium, although it is, as you say, a bit underdeveloped. However, the shop I currently work for are making good progress with Appium now on hybrid apps with iOS and Android.

Pros:

  • Free (as free as an economist would allow)
  • Well-supported
  • Nice coordination with Selenium

Cons:

  • New
  • Needs a bit more technical skill to get up and running

Also of note: SOASTA and Perfecto Mobile

As a note, I have worked at many high-performing companies, and have successfully implemented and supported automation efforts.

share|improve this answer
    
Thanks for your input. I might want to take a look at eggPlant later on. But now I am seeking some open-source solution with reliable codebase. Appium might work out once all the configuration issues are overcome. –  VojtaBurian May 14 at 9:08

I am trying to use Appium for automating cordova based apps on iOS and Android. Till now i have achieved no success at all. On Android: I am able to launch the app using Appium but the moment I am trying to switch to WEBVIEW it fails will exception: Not yet implemented.

On iOS 7.1 The App Launch itself fails.

share|improve this answer
    
I now succeed to connect to our hybrid app using following snippet. But this did not work e.g. on Android 2.3.* (webview not found)... app = os.path.abspath(os.path.join(os.path.dirname(__file__),'Application.apk')) desired_caps = { 'platformName': 'Android', 'automationName': 'selendroid', 'app': app, 'app-package': 'com.apppackage.Application', 'app-activity': '.HomeScreenActivity' } self.driver = webdriver.Remote('http://localhost:4723/wd/hub', desired_caps) self.driver.switch_to.context('WEBVIEW_0') items = self.driver.find_elements_by_css_selector('h2.ng-binding') –  VojtaBurian May 14 at 9:09
    
Ravi, this isn't really an answer - it would do better as a comment because you're saying that you have the same problem as the original poster. –  Kate Paulk May 14 at 10:55
    
@KatePaulk, but he does not have enough rate to add comments –  amazpyel Jun 13 at 10:26
    
@amazpyel - perhaps the mods could migrate to a comment? –  Kate Paulk Jun 16 at 11:06

I suggest you to use Calabash if you are talking about open-source frameworks. Calabash develops faster and better than Appium, IMHO. Also, Xamarin cloud is available. Calabash has powerfull documentation and community as same as Appium but I don't have experience with proprietary tools because open-source tool is enough for me at the moment.

And I am agree with you that

there is no ultimate solution for hybrid apps automation as of yet.

share|improve this answer

I read this in hopes but everyone is in the same place :)

This is a newer field (automating mobile devices) and therefore flooded with competition and as stated "infancy" development. Proprietary brands have found their own ways to access real devices and sell those services for top dollar. This is difficult for several reasons.

  1. Most mobile devices belong to a wireless service provider and are sold with agreements.
  2. The devices are more locked down on technology because they are packaged with a wireless carrier and they don't want just anyone getting in there are doing stuff.
  3. There is little money in "emulated" devices so they will come eventually when the devices are out and they won't necessarily perform anywhere near the actual devices specs cause basically they are free and there is no market...why not force the person/company to buy your device and then use it for testing.

Thus companies like Xamarian, Sauce Labs, SeeTest, Appurify, Perfecto Mobile, etc... are competing for the best way to automate this stuff. That is also why various others like Calabash(Xamarian), MonkeyTalk, Appium(Sauce Labs), etc... are working for better free versions while offering paid ones too. Appium is now connected with Sauce Labs btw...

Appium has an appeal because Selenium ties into it directly. It's a pain to deal with, but it should be improving more quickly now that Sauce Labs is involved. Although that will also depend on what they want to sell vs. what they want to provide for free...still no real devices there though.

Out of all of this though I would recommend Appurify http://appurify.com/ as they have the cheaper straight automation approach with Selenium integration. No manual testing on remote real devices though...just automation for a monthly fee. (partnered now with SOASTA...)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.