How is QA involved in usability testing?

Question

Usability testing provides feedback how well users can accomplish tasks using a GUI.

In many companies there dedicated departments for usability testing, separate from QA testing.

Still, I wonder what low-cost testing approach QA teams performs?

Approaches I tried occasionally, either in industry or in academy, either formally or informally:

• Hallway testing, when you pick a random person from your office to try something with UI
• Task-based testing with think aloud protocol analysis, where you prepare tasks, and listen to the person as she performs the task with your app, listening how she perceives UI and why performs this or the other action
• UX Guidelines compliance, though compliance to UX guidelines does not always gives warrancy that GUI is usable.

However, all those approaches were done usually once GUI was implemented or partially implemented. Can QA be involved in shaping the usability aspect at design phase?

Answer 1

They can - and should be. Sit with the designers as they develop their wireframes and prototypes. Help them out with their scenarios and personas. Make yourself a cardboard iPad and try out the layouts - see here for a real world example

( and an aside, I dont really understand what the difference between QA testing and usability testing is - well I sort of do but... )

Answer 2

I work for a small company with four developers and two testers. Our product is a web-based application that we deploy to Amazon EC2. The application targets holders of a special kind of bank account.

I will describe the process we use with external participants, i.e. with participants who are not our employees (we also run informal usability tests with employees). My process will not work for all companies or all products, but it may work for yours.

We use UserTesting.com, a site that users freelance testers for fifteen-minute testing sessions. You supply the instructions and the kind of tester you are looking for in terms of age range, kinds of computer/mobile device, or anything else you can think of. (We only accept testers with a specific kind of bank account.) For $39, a tester will follow your instructions and give you verbal and written feedback on their experience. You also get a video recording of the entire session. We point the tester to a non-production installation of our site that we maintain specifically for testing. (It too is deployed at Amazon EC2.) When we aren't testing, we shut the test site down, so we only pay for it when we need it.

For each round of testing, we pick a new/updated feature or an area that seems to generate a lot of support calls. We design some specific tasks that we want to test. We craft our scenarios and instructions carefully because the tester won't have us around to ask questions. The instructions are designed to be completed in fifteen minutes or less. We typically use 3-5 testers per round of testing.

For new features, we run the test as soon as the feature is stable enough that the tester is not at risk of being hamstrung by a bug. It is ok to tell the tester, "This is a new feature and it isn't finished yet but we are looking for early feedback."

We often have some tester feedback within an hour of posting a test. I often have feedback from 3-5 testers within a morning or an afternoon.

You have to be careful about how you use test results. They will frequently identify some kind of difficulty, but just because a single tester finds something hard or confusing does not necessarily mean you need to make any changes in your product. Instead, you need to consider the circumstances and look for trends.

These tests are not designed to be statistically significant. Instead, the goal is to pick a feature, point the testers at it, and just see what happens. It is an informal, qualitative process, and yet we have initiated a lot of improvements in our site this way.

Because these tests are informal and inexpensive, they can be executed frequently and quickly, which means we are more likely to use them than we would if there were a lot of process/bureaucracy involved.

I read a few books in usability testing before going down this path. One I recommend is Rocket Surgery Made Easy, By Steve Krug. It is short, easy to read, and practical.

Answer 3

I have worked in two different scenarios: a large project to develop a scheduling app for a government agency and an established commercial product. In both, usability was integrated into the testing but in very different ways.

For the large development project, we invited local representatives from the customer into our test lab during development (i.e. an open test lab). We would do various sessions, sometimes as independent evaluations and sometimes as paired testing.

In my current job, my desk sits next to the lead support person. I regularly overhear calls from customers and often get involved in discussions when an issue crops up.

Whatever the method, hearing the issues directly from users as a part of day to day testing is extremely important and productive, no matter how you classify them.

Answer 4

I've been involved with all of the above.

In my first QA role, I managed a team that was responsible for ensuring compliance to UI Standards.

At another company, we did somewhat formal Usability Testing, bringing in existing customers to explore the application's usability in an inexpensive manner: http://www.allthingsquality.com/2010/04/inexpensive-usability-testing.html

But in other companies where I have worked, User Experience was much more of an early-stage design task. There, the scheduling and staffing were such that QA wasn't very involved with UX.

Answer 5

My answer is 2 parts, first in direct response to your question: You should definitely keep usability in mind while testing your product. Just because the product matches the specification, does not mean that it is intuitive, simple and concise. If it's difficult for you to understand or use, then you can pretty much bet it will be even more difficult for someone who is not an expert like you. Once you have an early version or prototype, you can start doing some of the testing mentioned in other answers.

The second part of my answer is a little indirect, but I think very useful. Once you have released your site to the public, you can execute site optimization tests as described in this whitepaper: http://webtrends.com/files/whitepaper/Whitepaper-OnlineTestingForSiteOptimization101-Webtrends.pdf. Full disclosure, I work for Webtrends. This testing is usually not initiated by QA (but could still be recommended by them). This type of testing allows you to provide different content to different users who visit your site and test which content works better. For example, if the goal is to get people to complete a form, you can provide 3 different versions of that form and find out which version performs the best. Large revenue generating sites that use this optimization technique often end up increasing click-through rates (and thus revenue) by large percentages. Some of our customers have increased over 50% just by re-ordering elements, changing text, changing fonts, colors, etc.

Answer 6

I think they can. It is correct that QA usually handles the major part of testing i.e. the various types of testing. Regarding UI what I have seen is when UI development is going on, usually the opinion of QA is taken into consideration.

QA knows :

• the system's purpose as in how the system will be acting under normal and unexpected situations.

• the type of user that will be using the system

This is because the QA has some experience on which and what parts the user may got stuck and may become somewhat frustrated or helpless. In those cases, the QA suggests the UI developer about the overall look and feel of the app.

Those opinions are many times correct and taken into consideration.