Browser Testing

Question

We have now been conducting browser testing on our websites for over 4 years and are conducting a review on how we work. We split browser testing down in the following areas.

Functional - All keys processes should be tested to ensure they continue to work. Automated

Rendering - All sites should render in the same way on each browser

Usability - All sites should maintain the same standard of usability across each browser.

We find the quarterly checks on all sites to be time consuming, although have proven useful in maintain the standards of the site.

What does everybody else do? How important do you see browser testing?

Answer 1

Some things I've consider from a strategy point of view in various situations:

Could you move to a 'test based on change' approach? Could you monitor the industry for software upgrades (of for example browsers) and new screens (with different dimensions and characteristics) and thus test in response to change as opposed to periodic testing? Clearly you do monitor change to a degree because you have to upgrade tests to reflect these changes.

How closely do you monitor the system to try and discover issues as they occur. Are all web server and application logs automatically monitored? You can see indications of issues, users taking too long to perform a task, transactions abandoned, etc.

Could you move the testing requirement on to your clients? How easy could you make it for your clients to tell you about issues and problems? And how do you thank them? Entry into competition or a credit perhaps?

Do you actively try to measure the impact the issues you have discovered in the past have had? For example characterising issues by:

• percentage clients impacted
• length of time existed
• loss of revenue
• estimated type of issue: outage, limited access methods, loss of functionality, lack of usability, etc.

It can be interesting and knowing the estimated cost of issues might be useful as part of your review.

Clearly every situation is unique and I know nothing of what you do so just some thoughts that I hope may stimulate further thoughts.

Answer 2

Where I work, there are three web applications that need to be maintained, with a fourth (which is intended to unify all three existing applications) in development. Each has a distinct purpose:

• Application 1 is an enterprise-level employee and payroll management web application used by both internal payroll specialists and externally by customers.
• Application 2 is a much smaller site allowing individuals to view and print their paystubs.
• Application 3 is an internal-facing portal for managing customers, schedules, and the like.
• The new application is starting with App 2's functionality and will gradually integrate with, then replace App 1 and 3.

I'm the one and only test specialist here, responsible for all testing. My team is 10 people: me, 7 developers, and the leads.

My plan is:

• Functional - for the new app, build automated regression around key functionality as it is released and stabilized. For the existing applications, only App 1 gets manual regression testing, and then only for a small subset of key functions (due to time constraints). I plan to automate this as time permits.
• Appearance/Rendering - this is not tested in a systematic fashion. I check all new features in the existing applications for consistency with the rest of the site (using all the supported browsers), then hope nothing changes it.
• Usability - apart from testing that the shortcuts the applications are supposed to allow continue to function, this is not formally tested. The internal users of the applications are quick to notify the team when usability is an issue.

In general, I think you'll find that most places will automate functional testing as much as possible, and manually test appearance/rendering in different browsers. Usability testing will vary depending on whether the users are expected to be trained users or general public - for applications where the user base is the general public the specialist testers can often provide enough information about usability. With a user base of trained power users, it's often better to have them perform the usability tests.

Answer 3

First of all, browser testing is really important to do as it is one of the most common way you get customers into your sales funnel, get people interested in your product and showcase different features of your product.

Secondly, I believe you are already doing the right things in terms of testing them and have a really good testing strategy. Some things that can help in making sure we are making efficient use of our time would be-

• Split the responsibility of the different types of browser testing to different teams/groups. This way, one single team does not need to suffer doing everything

• Make sure you have good tools to support your testing effort and help you in getting faster feedback on your application across different browsers. Check this article which provides some valuable resources on browser testing - https://sdtimes.com/test/3-factors-consider-testing-responsive-websites/

• Use tools like Google Analytics and find out which browsers are commonly used by your customers to access your website. This would help to narrow down the scope of testing in terms of only having to test on a few browsers. For example - If you are testing on Chrome, Firefox, Safari, IE, Edge, Dolphin, Opera etc it is going to be really time consuming. With Google Analytics you may find out that 95% of your customers are coming from a Chrome, Firefox and Safari browser which means you can focus all your testing on these 3 browsers first and then if you have time pick other browsers

• Do combinatorial testing to narrow down the number of browser, OS and device options. You can use a tool like PICT or All Pairs. More tools can be found here - http://www.pairwise.org/tools.asp

• Do Session Based Exploratory Testing to do more focused testing to find rendering issues. These are timeboxed focused session on a particular module of the website. You can read more about it here - http://www.rajsubra.com/2018/06/06/sbtm/

The above approaches will help you get more efficiency in your browser testing effort.

-Raj