4

I have an Android app. Occasionally, some user reports an issue that is reproducible only on their particular phone/OS version combination. I cannot possibly get all phone models to try and reproduce the issue. So what is a good strategy to debug/troubleshoot issues that happen only on particular phones?

EDIT: I am not looking to testing on all phone/OS combinations. I am asking: How do I troubleshoot on one particular phone/OS combination, if I don't own that combination?

  • If this was a website, i'd suggest how to figure out browsers via something like google analytics, and then suggest a manual+automated testing option like browserstack and share some of my experience of doing automated tests on it. I'll upvote an answer that looks like that. – Nathan Cooper Oct 27 '17 at 22:12
  • I think some of the extra words may be confusing the answerers. I would interpret your question as "how do I debug/troubleshoot/test issues reported by my users on a particular phone/OS combination that I don't have a physical copy of, to support my existing user base?" -- maybe that last detail would make it clearer and help you get better answers. – c32hedge Jan 15 '18 at 15:42
1

On Bitbar.com you can rent access to many/most combination of OS and device. I have no affiliation, just considering the service. Not used them yet.

1

I suggest you to take a look at current market trends and get some idea from usage statistics of over your mobile app customers. This will give the best answer for your problem. Then you will get the information about most-used devices, and operating systems. So, you can plan your testing environment based on customer demand. In this way, you decide the of importance of the problem.

And you may check the cloud services for specific combinations for browser&os&device below.

https://www.browserstack.com/local-testing
https://saucelabs.com/automated_testing/cloud-mobile-app-testing
https://aws.amazon.com/blogs/aws/aws-device-farm-test-mobile-apps-on-real-devices/

  • I suggest you to take a look at your product analytics for mobile. What does that mean exactly? – user93353 Nov 27 '17 at 7:34
  • I mean you may take a look at current market trends and get some idea from usage statistics of over your customers. – cell-in Nov 29 '17 at 13:58
  • I mean you may take a look at current market trends and get some idea from usage statistics of over your customers. - I think you have not understood my question - this doesn't help me at all. – user93353 Nov 30 '17 at 3:55
  • 1
    I use SauceLabs TestObject. You can 'RDP' into almost any real device they support. It is through a web portal and can be kinda slow, but very handy for these one offs. – kirbycope Jan 15 '18 at 2:07
  • @kirbycope You might add that as your own answer since your comment and Peter's answer seem to be the only places the OP's actual question are addressed. – c32hedge Jan 15 '18 at 15:47
1

"I am not looking to testing on all phone/OS combinations. I am asking if I want to troubleshoot on one particular phone/OS combination, what's the way to do it?"

You are asking something only you can answer: whether it is useful or not to investigate a specific combination or not. Somebody already pointed out, look at the statistics.

That could actually help you:

  • How many people are using that combo?
  • How often is that specific problem reported?
  • How serious does it seem?
  • Could it be dormant in other combo's as well?
  • ... and so on

The question if you should look into it depends also on the possible gain. If you spent weeks to investigate a problem that only happens in 1 of thousands of cases it might be best to leave it and focus on something else. That is a business decision, however.

If it is worthwhile, according to you, to investigate: get that device. There are some providers for online test hardware - never used them - maybe they have the combo you need. And of course the Android emulators might be enough for your combo.

YMMV...

  • whether it is useful or not to investigate a specific combination or not - I am not asking that question at all. – user93353 Jan 14 '18 at 17:38
  • Well.. than you apparently mean something completely different with I am asking if I want to troubleshoot.. – Ray Oei Jan 15 '18 at 13:16
  • If I want to troubleshoot - then how do I go about it? Read the whole sentence – user93353 Jan 15 '18 at 14:40
  • That part is also in my answer: prepare a working emulator for that device, Rent one, Buy one. Simple. – Ray Oei Jan 15 '18 at 16:01
  • Just a friendly note: your tone (intended or not) doesn't really help, I fear, in getting the help you are seeking. – Ray Oei Jan 15 '18 at 16:01
0

All Android device have similarities like OS version, screen density, launcher, etc. You don't need to test all possible combinations and take in account rare exceptions - you need to ensure that your app is compatible with predefined compatibility scope.

Compatibility scope is a part of non-functional requirements. If you have no requirements, you should define the scope on your own. The best approach here is to consider usage statistics from the countries where you potential audience lives.

In the end, you'll get a list of a few devices, not hundreds.

P.S.: Device vendor does not matter here.

  • I am not looking to testing on all phone/OS combinations. I am asking if I want to troubleshoot on one particular phone/OS combination, what's the way to do it? – user93353 Nov 7 '17 at 11:03
0

I believe you can use an open-source automated testing tool providing the possibility to test on emulators (virtual phones). Some of these tools include: Selenium-based Appium and Selendroid, which is tailored specifically for Android apps with API up to 19, Calabash, Robotium and more.

0

Even the Android award top developer cannot 100% guarantee their application can run seamlessly on all devices. And it's not possible to test everything with all device-os combinations. The best place to report/search device-specific bugs is probably still via Android Issues Tracker, of cause, as long as issues are found and reported by consumer developer. You can compare or find the issues and fixes using search filters. I think it's practical way to find solutions for device specific bugs that you couldn't replicate due to unavailability of devices.

  • I am not looking to testing on all phone/OS combinations. I am asking if I want to troubleshoot on one particular phone/OS combination, what's the way to do it? – user93353 Nov 7 '17 at 11:03
0

Unfortunately there is no bullet proof way. If you go to one of the more detailed usage statistics by model reports (I can't provide links, those are usually paid) you will notice that there is a lot of fragmentation and no one sub-model has more than very few usage percentages, and that's before OS version is taken into consideration. Note that for each main model e.g. Galaxy S8 there are more than a few sub models based on radio type, main processor, memory etc.

If your application is fairly simple (no hardware dependency for example, but even this is tricky since some graphical features do use dedicated hardware accelerations) you can probably get away with testing on one type of model out of sub-models based on usage statistics.

One of the many options for remote lab with real devices, or crows sourced testing, are good for covering the top used models, but again you will probably fail to get 100% coverage.

The bottom line is that you simply cannot test all combinations unless you purchase each and every one of them, probably more than one device of each model since re-installing the OS will bias your results at best and is impossible for some phones-OS combinations.

  • Mistake in my question - I am not looking to testing on all phone/OS combinations. I am asking if I want to troubleshoot on one particular phone/OS combination, what's the way to do it? – user93353 Nov 7 '17 at 11:03
  • I don't think there is a much better answer to that, in theory you could have a list of which vendor has what but that's not easily scalable. – Rsf Nov 7 '17 at 11:27
  • How would a list of which vendor has what help me in any way at all? – user93353 Nov 7 '17 at 11:51
  • Vendor as in test labs providing real phones, so If you need a sub model/OS version you know where to find them. – Rsf Nov 7 '17 at 11:53
  • There are no test labs where I am – user93353 Nov 7 '17 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.