Software Quality Assurance & Testing Stack Exchange is a question and answer site for software quality control experts, automation engineers, and software testers. It only takes a minute to sign up.
Sign up to join this community
I have an Android app. Occasionally, some user reports an issue that is reproducible only on their particular phone/OS version combination. I cannot possibly get all phone models to try and reproduce the issue. So what is a good strategy to debug/troubleshoot issues that happen only on particular phones?
EDIT: I am not looking to testing on all phone/OS combinations. I am asking: How do I troubleshoot on one particular phone/OS combination, if I don't own that combination?
On Bitbar.com you can rent access to many/most combination of OS and device. I have no affiliation, just considering the service. Not used them yet.
I suggest you to take a look at current market trends and get some idea from usage statistics of over your mobile app customers. This will give the best answer for your problem. Then you will get the information about most-used devices, and operating systems. So, you can plan your testing environment based on customer demand. In this way, you decide the of importance of the problem.
And you may check the cloud services for specific combinations for browser&os&device below.
https://www.browserstack.com/local-testing
https://saucelabs.com/automated_testing/cloud-mobile-app-testing
https://aws.amazon.com/blogs/aws/aws-device-farm-test-mobile-apps-on-real-devices/
I suggest you to take a look at your product analytics for mobile. What does that mean exactly?
Nov 27, 2017 at 7:34
I mean you may take a look at current market trends and get some idea from usage statistics of over your customers. - I think you have not understood my question - this doesn't help me at all.
Nov 30, 2017 at 3:55
"I am not looking to testing on all phone/OS combinations. I am asking if I want to troubleshoot on one particular phone/OS combination, what's the way to do it?"
You are asking something only you can answer: whether it is useful or not to investigate a specific combination or not. Somebody already pointed out, look at the statistics.
That could actually help you:
The question if you should look into it depends also on the possible gain. If you spent weeks to investigate a problem that only happens in 1 of thousands of cases it might be best to leave it and focus on something else. That is a business decision, however.
If it is worthwhile, according to you, to investigate: get that device. There are some providers for online test hardware - never used them - maybe they have the combo you need. And of course the Android emulators might be enough for your combo.
YMMV...
whether it is useful or not to investigate a specific combination or not - I am not asking that question at all.
Jan 14, 2018 at 17:38
All Android device have similarities like OS version, screen density, launcher, etc. You don't need to test all possible combinations and take in account rare exceptions - you need to ensure that your app is compatible with predefined compatibility scope.
Compatibility scope is a part of non-functional requirements. If you have no requirements, you should define the scope on your own. The best approach here is to consider usage statistics from the countries where you potential audience lives.
In the end, you'll get a list of a few devices, not hundreds.
P.S.: Device vendor does not matter here.
I believe you can use an open-source automated testing tool providing the possibility to test on emulators (virtual phones). Some of these tools include: Selenium-based Appium and Selendroid, which is tailored specifically for Android apps with API up to 19, Calabash, Robotium and more.
Even the Android award top developer cannot 100% guarantee their application can run seamlessly on all devices. And it's not possible to test everything with all device-os combinations. The best place to report/search device-specific bugs is probably still via Android Issues Tracker, of cause, as long as issues are found and reported by consumer developer. You can compare or find the issues and fixes using search filters. I think it's practical way to find solutions for device specific bugs that you couldn't replicate due to unavailability of devices.
Unfortunately there is no bullet proof way. If you go to one of the more detailed usage statistics by model reports (I can't provide links, those are usually paid) you will notice that there is a lot of fragmentation and no one sub-model has more than very few usage percentages, and that's before OS version is taken into consideration. Note that for each main model e.g. Galaxy S8 there are more than a few sub models based on radio type, main processor, memory etc.
If your application is fairly simple (no hardware dependency for example, but even this is tricky since some graphical features do use dedicated hardware accelerations) you can probably get away with testing on one type of model out of sub-models based on usage statistics.
One of the many options for remote lab with real devices, or crows sourced testing, are good for covering the top used models, but again you will probably fail to get 100% coverage.
The bottom line is that you simply cannot test all combinations unless you purchase each and every one of them, probably more than one device of each model since re-installing the OS will bias your results at best and is impossible for some phones-OS combinations.
