We're trying to have automated mobile testing but it's taking a huge amount of effort and I'm not seeing any rewards.
Our site is responsive and has a mobile view. We do not have native mobile apps.
What I have found so far is that nearly all the errors we have found also occur on desktop flows, for example
- A required field wasn't entered
- A field entry used an invalid format
- I got a 404 or 500 error page
- A dropdown list did not have the expected values
- An error field did not have the correct error highlighting
I have not been able to identify any mobile specific issues (issues that would not occur on desktop) that our automation would catch.
Also, the few errors that I have seen such as :
- A field is on the page but not currently visible in the viewport
- A field is to small to tap on with a finger
- Tapping a field twice leads to a click on the next page when finally loaded
are NOT the sort of issues that our selenium based automated mobile testing (using Ruby, Capybara and Appium) would capture.
I am being asked by my organization to do automated mobile testing because this is half of our traffic. I accept this but feel that given the difficulty of setting up mobile testing and the effort involved, I should identify what issues automated mobile testing should be expected to uncover.
I am looking for issues that wouldn't be present when running desktop automation on a narrow window to force the responsive mobile view.
This follows a theme that I have been focused on a lot lately, which is 'How to make sure tests add value'