1

Are there any differences for mobile web (not native App, not hybrid) testing between emulators/simulators and real devices?

There are different 3rd party sites under test that contain our custom in-house HTML5 player that needs to be verified, along with all the analytics, on a plethora of mobile devices.

For the analytics stuff, especially on iOS devices, the Man-in-the-middle proxy that we're starting beforehand needs a manually accepted and installed SSL certificate. Because of this, very few cloud-browser providers offer Private devices but they come with large costs.

So, we were thinking of building our own test lab. Naturally, the first choice would be to use iOS Simulators and Android Emulators, but we don't know whether we would get false-positives or false-negatives because of this.

Hopefully someone had this dilemma before and can shed some light into this (even if it's not HTML5, any info would be useful).

Improve this question

2 Answers 2

Reset to default
3

There will be differences between emulators/simulators and actual devices. Android emulators just emulate the screen size and pixel ratio and run a stock version of Android. Many of the major devices run their own custom version of Android. For instance, any Samsung Galaxy runs it's own version of Android which contains Samsung software to promote people using their browser and software. Other low end devices also use their own versions of Android which is designed to run on low memory.

IOS simulators through the browser are not as reliable as the IOS Simulator through XCODE. The XCODE simulator is more reliable as it is the one which IOS developers use during their development process. That being said, when you run the XCODE IOS simulator on a computer, you choose the processor which you want to emulate. Most of the times, you run using and X64 processor which is optimized to run on a computer. It's not the same processor which an actual iPhone will run.

Since, both implementation and hardware is different, there will definitely be false positives. As well as differences in performance.

Also, you should keep in mind when creating your own device farm, iOS limits the number of simulators which you can run on one apple device. You can generally run one simulator per Apple machine. So to build a device farm legally, you are going to need a bunch of mac computers or iPhones.

Improve this answer
1

Although this answer is two years ago, I try to share my lessons learned (it is not directly HTML5 but as you said any info would be useful). In our project we also thougth about the possibility just to use emulators and afterwards we discussed about the pros and cons using real devices vs. emulators. After discussion we decided to use both, but focus also on real devices. Here is the summary:

  • virtual devices / emulators can just emulate e.g. incoming calls or connection losses (WLAN/3G) and hence cannot replace real situations (in our case we had to test with real cars and real devices, testing just with emulators was not possible to genereate a real situation e.g. in the mountains / forest)

  • energy consumption of a device can just be executed on a real device and not on a emulator

  • only on real devices we get the specific HW- and SW package from the manufacturer. Emulators are in most cases "generic" and we don't get the specific HW-and SW package. For example within iOS devices pressing the Home Button has crashed the app and once when testing with emulators/virtual device this would not happened and probably bug would not have been detected (see Example of a user link: Rootless Launcher German and see comment from user Max "Has anyone installed it successfully at P20? It is possible to select launcher on settings but once clicking on Home Button the app crashes. Message appears Smart Recovery, serious error by rootless pixel launcher)

  • executing tests on real devices is generally more efficient that executing test cases on emulators. The behaviour of latency is generally different between both options and hence can have an impact on test result

  • only real devices have real system configurations

  • the installed interfaces and sensors are quite compex and hence are fragile in case of interferences (e.g. GPS). And furthermore only on real devices we are able to test with real edge conditions.

  • for functional test, testing with virtual devices/ emulators are ok, but for measuring/covering the performance and running stability test of real devices are mandatory

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.