What issues, when automating responsive mobile sites, don't appear on a desktop

Question

We're trying to have automated mobile testing but it's taking a huge amount of effort and I'm not seeing any rewards.

Our site is responsive and has a mobile view. We do not have native mobile apps.

What I have found so far is that nearly all the errors we have found also occur on desktop flows, for example

• A required field wasn't entered
• A field entry used an invalid format
• I got a 404 or 500 error page
• A dropdown list did not have the expected values
• An error field did not have the correct error highlighting
• etc

I have not been able to identify any mobile specific issues (issues that would not occur on desktop) that our automation would catch.

Also, the few errors that I have seen such as :

• A field is on the page but not currently visible in the viewport
• A field is to small to tap on with a finger
• Tapping a field twice leads to a click on the next page when finally loaded

are NOT the sort of issues that our selenium based automated mobile testing (using Ruby, Capybara and Appium) would capture.

I am being asked by my organization to do automated mobile testing because this is half of our traffic. I accept this but feel that given the difficulty of setting up mobile testing and the effort involved, I should identify what issues automated mobile testing should be expected to uncover.
I am looking for issues that wouldn't be present when running desktop automation on a narrow window to force the responsive mobile view.

This follows a theme that I have been focused on a lot lately, which is 'How to make sure tests add value'

Answer 1

Michael Agree. Faced same condition before. As experienced earlier few things might help you at some extend.

• You need to convey them the pros & cons of the automation testing.

• You need to give them a brief about the use of automation as well. I mean with lesser knowledge one can expect to much from you. You need to take initiatives to keep balance between actual possible things to automate and expectations

• As per my knowledge, In your case of browser automation we can ensure that everything is working well 24*7. This is a primary goal of browser automation.

• Communicate clearly if we are not bothering about things should work ALWAYS then we should think of reduce the dependency & efforts on browser automation

• Limitations: Its really out of scope [as of now at lease] to automate browser to catch images or content overlapping, veriication of image quality and visibility, crosschecking on different versions of browser in [responsive] website

Answer 2

I test mostly CMSs ie corporate website. Usually, the defects I find in devices would be related to styles e.g. string does not wraparound, broken responsive. for that I use browserstack and a person visually checking the logs. sometimes functional issues are found i.e. buttons not clickable on small device.

Other times, i find the 'issues' found are UX related which can only be found manually. e.g. something does not look 'nice', not centred on certain devices and not all

perhaps it would be good to discuss and trash out what type of issues concern your boss/biz rep and take it from there. It could just be that they hate styles issues.

good luck!

Answer 3

Not sure whether you have tried http://galenframework.com . One of my colleagues showcased its capabilities (he is using it for testing the responsiveness) and it captures what an human eye could never capture.

Cons: Pretty easy to develop from a testing point but without the support from UI/UX team, Herculean task.