How do I determine which Android devices to test on so that I get the best coverage?

Question

Android platform is famous for it's saturation and variety. Even if I test my app on 6 different devices (low-end, high-end, gingerbread, honeycomb, ICS, small display, large display, vanilla android, HTC sense, etc), some device-specific bugs still seem to haunt my apps.

Android Market says that there's rough 600 different android devices my app can be used. That sure is lot of devices to test on. I can't afford to buy them all.

Do you have good "Best Practices" for testing device-specific problems on android apps?

Are there any good beta testing services I can post my app and have it tested on 100 different devices? I used Samsung's remote testing service, but works quite poorly on openGL/NDK applications.

Answer 1

Welcome Habba,

This is a variant of the problem that is faced by website developers everywhere. Ultimately it needs to be about where do you want to focus your time and efforts. Just like a website, or java client app, no program will run perfectly everywhere. This is the exact reason why Java was referred to as 'write once, test everywhere'. This exact problem of the regression testing matrix is why apple is very aggressive with it's device upgrade schedule and not supporting most devices that are greater than than three years old.

Looking at the data produced by google, you should be focusing on Froyo and Gingerbread which hold 87.3% of the market.

Then you should focus on the main devices that run those versions of the OS, with mdli and hdpi screens as they have 81.8% of the market.

To expand beyond that, I would use a pairwise technique to work out which other devices to test on.

Answer 2

Welcome Habba!

This is a very troublesome issue that will likely never go away. And I rather like that, as it keeps the market more interesting. There are several strategies I use in my org to approach testing for Android, listed below. But the biggest thing I've come to realize when working with this platform is that there will ALWAYS be some devices that don't handle your application as well. There are too many variables to consider and at some point cost wins out. The best you can do is mitigate damage at that point.

• Place limits early: Depending on context and what you're creating, limit what devices can install the build. The manifest file allows a pretty decent level of control for target devices by SDK (Android Version), resolution, device features, and many others. Android Manifest
• Narrow focus: Once we place limits on the supported devices, we further limit test devices by selecting a low/mid/high range that fits those criteria and focus on those. Then as time permits more devices are cycled in to get some extra coverage.
• Spread the build around: In my org, there are a lot of people that have Android phones/tablets as personal devices. So it would be a shame to let all that go to waste making phone calls/texts or checking Facebook. If you have them check Install from Unknown Sources under Application Settings you can place the .apk online (a simple page w/ a download link) and get your new tester helpers to go and install.
• Emulate: Depending on what you're developing, there are also a lot of solutions using the Android SDK to emulate devices.
• Automate: There are quite a few tools to spread the load around. The SDK/Selenium/etc all have the capacity to automate using the emulator or physical devices. It isn't perfect, but gets at least some of the more obvious problems out of the way leaving time for you to focus on the important stuff.

As for distribution and beta testing, you could do what I mentioned above and just put the .apk online and send the link out. Or there are several paid services that will provide an extensive library of devices and a small army of testers to use them. Services like uTest are great for stuff like this. Not cheap, but they are valuable.

Answer 3

While there are literally thousands of combinations of Android Devices/OS Versions, there's a few that stand out as leaders. Determine what Devices/OS's are most popular. This information is available online. Determine your threshold for testing coverage. ( Example; Only devices with a market share greater than 25%)

Also, make sure to look into any popular ( This is relative and for you to determine) devices that might have some kind of attribute that you'd like to confirm functions properly (abnormal screen resolution?)

The bottom line is that we will never get 100% test coverage on all devices and OS configurations. Even with Emulators, we will always have to pick the most popular.