We are currently testing a new website on device. However I believe we could test by operating system version and Prominent browser instead. Has anyone else tried this approach rather than testing on every device?
I guess to minimize risks, but limit mobile testing to a limited number of devices.
I would test on something like:
- Each different Mobile OS version still in use significantly
- Latest major browser versions on those OS's
- Most used resolutions of Mobile devices
Still I would wonder how much more confidence I would get when testing with more devices. Maybe a shortlist would be enough.
I would create a shortlist based on last 3 months data from http://gs.statcounter.com/
- Screen resolution
- Browser version
- Operating system
We tried to base our Android tests on marketing surveys, the results were mixed. To begin with a marketing name like 'Nokia 6210' is not good enough, you need a specific model ID since hardware and software have significant variations.
Then we found, as expected, that the market is highly segregated- each model has a really small percentage of market share, and it's far from being statistically significant to choose between a 1.12% (of market share) device and 1.43%.
But at the end you don't have another, better, choice. There are hundreds of sub models, sub OS and browser versions putting a huge price tag on testing, even if you use this or that virtual, remote test services
Testing for all devices is difficult thing. But you can test your app by checking different OS version-wise. If you looking to test on different devices you use third party service provider for cloud device. Try for Testdroid/Bitbar, here you can get free devices for Manual Testing too.