Usability Testing – The Definitive Guide

This guide will be the only guide you’ll ever need for understanding and getting along with usability testing. 

This guide covers the following topics:

• • What is Usability Testing?
  • Importance of Usability Testing
  • Types of Usability Testing
  • Advantages of Usability Testing
  • Disadvantages of Usability Testing
  • Questions to ask in Usability Tests
  • Misconceptions related to Usability Testing
  • How is Usability Testing different from User Testing?
  • Challenges of Usability Testing
  • Conducting Usability Tests
  • Tips for Usability Testing
  • How can zipBoard help?

Ready to dive deep into usability testing? Read on!

What is Usability Testing?

User feedback is an important aspect of developing products that people love and share. Asking others to evaluate your product is common in every field especially design and user experience. Usability testing is all about getting people to interact with your website or app and observing their reactions and behavior to improve the user experience and design of your product. 

In other words, it is a method of evaluating the user experience by determining whether their actual users are easily able to use the product, app, or website. Designers and UX researchers conduct usability tests after every iteration of their product – from initial stages to the release. 

Why is Usability Testing important?

Usability testing is done by real-life users in order to address the usability issues of a website or a product. It is basically done to know about the issues that designers, marketers, and product managers are unable to identify. 

Usability testing is important to:

• • Understand how real users use your product or website
  • Catch minor bugs that may have gone undetected during the design review and development stages
  • Identify gaps in the market, new ideas, and product features
  • Ensure that users don’t get stuck with complex workflows
  • Deliver the best-in-class user experience
  • Validate product prototypes

What are the types of Usability Testing?

Usability testing can be broadly classified into 3 domains – moderated vs unmoderated, remote vs in-person, and comparative vs explorative vs assessment.

Credits – hotjar.com

Moderated vs Unmoderated

Moderated testing involves the presence of a moderator that guides the testers with respect to the testing workflows, answers their queries, and asks follow up questions. This type of testing provides in-depth results but it can be very expensive and time-consuming. Moderated testing should be done to analyze user behavior. 

Unmoderated testing does not have a moderator to supervise the testing process. The testers test on their own sitting in a lab or their homes. This type of testing may result in no or less feedback as the testers may give up on the tests due to technical difficulties, especially during testing complex workflows. Unmoderated testing should be done to observe and measure behavior patterns.

In-Person vs Remote

In-person tests have to be conducted in the presence of a moderator and it provides extra data points like body language and facial expressions. However, it is costly, time-consuming, and it requires space in order to conduct the tests. The number of tests is comparatively low when compared to remote testing.

Remote tests are done over the internet or phone. It is significantly cheaper and more tests can be conducted when compared to in-person tests. However, the number of data points obtained via remote testing are less when compared to in-person testing. Remote testing is recommended when you have to collect a large sample of data. It is not recommended for testing complex workflows but a good way of capturing remote feedback.

Explorative vs Assessment vs Comparative

Explorative tests are open-ended and are usually done during the early stages of product development. It requires the testers to brainstorm, give ideas and opinions about the product concept and usability. It helps in addressing gaps in the market and identifying potential new features and ideas.

Assessment tests are aimed at evaluating a product’s general functionality. It tests user satisfaction and how well they are able to use the product.

Comparative tests involve asking users to choose a solution from two or more solutions. These types of tests are primarily used to compare a website or a product with its competitors.

What are the benefits of Usability Testing?

No matter where you are in your development stage, your website, app, or your product can benefit from usability testing. From prototyping to building a full-fledged product, you must incorporate usability testing. 

Here are some of the benefits of usability testing:

Prototype Validation

Validating what you’re building is essential for the usability of your product and helps in resource optimization. That’s why it is important to build a prototype first instead of locking down the features of your final product. During the prototyping phase of product development, testing helps validate your product idea. It helps you plan future development and reduces unwanted overheads and wastage of resources. Moreover, it helps in avoiding product failures.

Credits – usabilitygeek.com

Testing at the prototyping stage will help you understand whether the product you’re building is important to the users? It will also help you figure out if you really should go ahead and build the complete product? Based on the response of users during the prototyping stage, you will get an idea of what to build next and what all features to include in your product.

Identifying Issues With Complex Workflows

There may be complex workflows involved in your product especially if you’re building an e-commerce product or if you have a multi-step signup page. You do not want users churning off during these complex workflows. Usability testing helps you understand the reasons for user drop-offs during each step of your multi-step process. It helps you better conversions leading to improved user experience. 

Catching Minor Errors

No matter how much vigorously you test your product or use web app bug management tools, minor bugs always seem to go undetected. Usability testing helps in figuring out these small errors essentially saving you from future embarrassment. A new set of eyes is required to pick up broken links, spelling and grammatical mistakes, broken layouts for a few particular screen resolutions, and much more which the developers and testers generally tend to miss out on. One or two minor bugs do not hamper the user experience much but collectively, these small bugs raise a red flag against the trustworthiness and professionalism of the business.

Get Stakeholder Buy-In For Change

Developers and testers may find some product features or workflows messy but they do not generally get the stakeholder buy-in to change it. If you see that the users are actually struggling while getting their hands around a feature or a workflow, you can easily persuade the organization to make changes to the same. Analyzing and reporting the key struggles of the users is highly important in order to get the stakeholder buy-in for change. Gartner expects 89% of businesses to compete for customer experience.

Unbiased Feedback Resolves Team’s Internal Debates

Usability testing can resolve your team’s internal debates. Accurate and unbiased user feedback who (unlike your team) are not emotionally attached to your product will give you a clear picture of what’s working, what’s not, and what to work on next. Analyzing how users use your product help in not having unnecessary debates thereby improving team communication. Let users guide you towards building an engaging and useful product.

Ultimately Provide World-Class User Experience

Research shows that, on average, for every $1 invested in improving user experience brings $100 in return. Usability testing helps you build user-friendly products that are aimed at delivering the highest order of user experience. Users remember the experience and a world-class experience is essential for a successful product. Testing at every stage of development is critical for ensuring the delivery of unforgettable customer experiences. With testing, users are better able to reach their goals resulting in the business meeting its targets.

What are the disadvantages of Usability Testing?

No doubt that usability testing is highly advantageous on multiple grounds, it comes with its own set of disadvantages.

• • Costly & Time Consuming – Usability testing can be expensive, time-consuming, and labor-intensive to run and maintain. 
  • No Real-World Scenarios – The tests are done in laboratories and do not really represent the real-world scenarios of how the users might use your product. The tests might miss out on some of the key reactions from real users of your product.
  • Analysis Is Time Consuming – The data obtained from the usability tests can be fairly complex and its analysis is time-consuming.
  • Significant Commitment Required – The process of usability testing requires testers to be fully committed to the task at hand. 
  • Moderated Usability Testing Produces Biased Results – During moderated usability testing, the moderators ask the testers to follow particular guidelines which result in biased test results as the testers do not get the freedom to test the product on their own.
  • Testers Give Up During Unmoderated Usability Testing – Without the presence of a moderator, the testers may get confused and not take the tests seriously. They may not think out loud resulting in limited or no qualitative feedback. Testers may face technical issues especially during complex workflows resulting in them giving up on the tests.

Questions to ask in Usability Tests

In order to get actionable insights from usability testing, you must ask the right questions. Writing these questions sound easy at first but in reality, it takes a lot of time and effort to come up with the right questions. Some testers will interpret questions differently. Not only the questions have to be right but the testers and their experience with products.

Credits – hotjar.com

Usability tests are run to validate a prototype, to find underlying issues with multi-step complex workflows, to gather unbiased user opinion, and to get feedback in order to improve the overall user experience. 

Here are some example questions you must ask the testers in a usability test:

• • Why will they visit your website? What are they trying to achieve? Other websites that they might visit before visiting your website?
  • What is stopping them from completing this task? What else is required on this web page? Is the wording easy to understand? Is the navigation too complex and confusing?
  • What convinces them to take this action? What do they like the most about the experience on this web page?
  • Would they visit the site again? Do they understand why the site exists?

Usability testing has four stages – screening, pre-test, test, and post-test. To collect quantitative data, use multiple choice questions and scale based questions (scale from 0 to 10). To collect qualitative data, ask open-ended questions, like – what do they think about this feature? 

A good practice to follow while gathering qualitative data is to follow a “yes” or “no” type of a question with additional open-ended questions called probes. Example: Do you like this new feature? If yes – why? If no – why not?

Let’s have a look at the various questions to be asked during each of the four phases:

Screening

Before proceeding with the tests, you must screen your testers based on the target audience or user personas. Ask basic demographic and experience questions during the screening period to figure out potential testers. 

Examples of screening questions assuming the website sells online programming courses:

• • What’s your age?
  • What’s your profession?
  • Have you ever visited [your website]?
    
  • What’s your highest education?
  • Have you ever taken an online course?

Pre-Test

Once you’ve selected the testers, you could again interview them prior to the actual usability test in order to learn more about their knowledge and expertise on the subject. 

Examples of pre-test questions assuming the website sells online programming courses:

• • Have you ever taken a programming course?
  • Do you know basic programming?
  • Which programming language are you most comfortable with?
  • Have you ever used a similar site before?
  • Why would you buy the course from our website and not from a similar website (say Z)?

Test

This is the actual testing stage. You must collect data that clearly answer why testers make certain choices while navigating through the tasks. The test may be highly interactive as the tester may talk about their thought process throughout the testing process with every step they complete. Conversely, the testers may work independently and answer the questions after each section is complete. 

Examples of test questions assuming the website sells online programming courses:

• • What do you think of ______? 
  • Which of these approaches do you like the most? Why?
  • I noticed you did ______. May I know why?
  • Was there any other way to _______?
  • What are your thoughts on the signup experience?
  • Were you able to find your course easily?
  • What are your thoughts on the checkout experience?
  • What about the total number of hours per course?
  • Is the course content clear and self-explanatory?
  • What are your thoughts on the length of each section in a course?

Post-Test

After the test is over, you get a chance to ask questions related to the overall experience. Common questions during this phase include:

• • What was the best/worst thing about ______?
  • What’s your overall impression of ______?
  • How is _____ different from our website?
  • What are your thoughts on changing _____?

Misconceptions related to Usability Testing

There are some user experience tools that must not be related to Usability testing. Some of these tools are as follows:

Surveys

Surveys may be used in conjunction with usability testing but in general, they do not come under usability testing because they do not allow you to observe the interaction of users with your product. They are good at taking feedback from users. Feedback Survey tools: HubSpot forms, JotForm Survey Maker, SurveyMonkey, Typeform, and more.

Credits – feedbacq.co

Heat Maps

A heat map allows you to visualize how users move around the web page. The most popular parts of a web page are the hot regions while the least popular parts are cold. Heat maps basically display how users move around a web page or website. Using heat maps is not usability testing. Tools for heatmap generation: Crzyegg, Mouseflow, NotifyVisitors, Hotjar, and more.

Credits – b2bquotes.com

Focus Groups

A focus group is basically a group of users gathered together in order to discuss a specific topic. Focus groups are used to know people’s opinions about the product or service. They are not used for testing purposes.

A/B Testing

A/B testing helps in validating whether a certain approach is working or not by experimenting with multiple versions of a web page to see which one is most effective in driving results. It does not validate user behavior, unlike usability testing. Tools for A/B testing: Google Analytics & Google Optimize, Optimizely, VWO, Adobe Target, and more.

Credits – optimizely.com

User Acceptance Testing

It is the last phase of the software testing process – a technical quality assurance test where users go through a set of steps. It cannot evaluate if the product is user-friendly. Learn more about user acceptance testing here.

How is Usability Testing different from User Testing?

Most developers, designers, and product managers often get confused between user testing and usability testing. While user testing is done before the development of the product, usability testing is done after the product has been developed. 

Credits – theuxblog.com

There’s a belief that user testing is done to validate the demand for the product whereas usability testing is done to evaluate how users use the product and its user experience (website review). 

For user testing, the most important question is – “Will users use the product or the newly added feature?” whereas, for usability testing, the most important questions are – “Will the user be able to use the product?”.

Note that user testing is done by focus groups.

What are the challenges of Usability Testing?

Usability testing is not a piece of cake. It takes efforts and resources to evaluate the user experience of your product. Let’s look at a few challenges of usability testing:

• • Setting clear objectives at the start – There may be a long list of issues that may be needed to address. It may be challenging to set clear objectives at the start. Having clear objectives helps in avoiding the wrong questions being asked and wrong testers being recruited. If the objectives are not clear at the start, it can result in inappropriate and meaningful recommendations.
  • Finding the right testers – When you are conducting usability testing, you want to have the testers aligned with what you are building. If you have a clear picture of who your users are, then only you will be able to find the right testers who will point you in the right direction. 
  • Finding enough testers – Another challenge is to find the right testers in adequate numbers as it is tough to find the right testers who are willing to participate in the testing process. The sample size of testers must not be so small that it produces biased or inappropriate results.
  • Setting the right tasks – Once you onboard enough number of right testers, you must provide them with the right set of tasks. If you want value for money from your usability testing efforts, do not give the testers a list of predetermined tasks. Instead, tell them upfront what is important to you so that they explore the product in line with it. 
  • The pressure to deliver – In fast-paced environments, there is significant pressure to meet deadlines, pushing features into production in a rush sometimes self-validating the features in order to keep the development train running. 
  • Get organizational buy-in – In order to get the buy-in from your organization, you must keep the relevant stakeholders engaged right from the word go. Ask the stakeholders and decision-makers to have a look at some portion of the research in real-time. 

How to conduct a Usability Test?

Before conducting the test, you must decide on the testing method that’s right for you based on what you want to achieve, how much time and money you want to invest, and the audience you want to test.

Credits – hotjar.com

Let’s dive deep into the five-step process of the most complex testing method- moderated usability testing.

Step 1 – Planning

It is a crucial part of the testing process as it dictates how you proceed further and the results you obtain. The planning stage deals with:

• • Defining the problems/areas you want to focus on
  • Type of users you want to test
  • Questions you want to ask
  • The location where testing will be done
  • Timeline of the testing process
  • Moderators who will run the sessions
  • Session recording techniques

Step 2 – Recruiting

This stage deals with recruiting testers which largely depends on your testing goals and planning. You must find the right testers and in adequate numbers to not obtain inappropriate results. Recruit testers by hiring an agency, using your website or social media channels, or simply by listing jobs on job portals. You may also recruit your customers or users for the testing process.

Step 3 – Designing The Test

This stage deals with planning the scenarios your participants will go through and assigning the tasks they need to complete in order to obtain actionable insights. 

Step 4 – Execution

This stage deals with executing the various testing scenarios with testers. If you’re going for moderated usability testing, make sure the participants are physically comfortable with the testing setup and also that they know what will happen during the session. Before recording the session, seek the permission of the participants. Make them as comfortable as possible by engaging in friendly conversations before starting with the session. 

Collect pre-test data before transitioning into the first task. If a participant is getting frustrated with one scenario, switch them to an easier scenario. Have someone else take notes for you so that you are fully committed to engaging with the participants. Reserve some time in the end for follow up questions for gathering overall feedback.

Step 5 – Analysis

This stage deals with analyzing results from the collected data. Do this as soon as the session is complete as the observations will be fresh in your mind. Using the data collected, try to find out critical and frequent problems for further examination. Prioritization of issues is essential.

Tips for Usability Testing

• • Set clear criteria for the recruitment of participants.
  • Keep stakeholders in the loop right from the beginning.
  • Make participants aware that they are not being tested.
  • Let someone else take notes during the session so that you can fully commit to moderating the session.
  • Before the testing begins, make participants as comfortable as possible.
  • Do the analysis as soon as you’re done collecting data from the tests.
  • Clearly state the objectives to the participants.
  • Follow yes/no type of questions with open-ended questions like why/why not.
  • According to Usability.gov, have more than 12 participants in order to find almost 100% of usability issues.
    
  • Ask participants to think aloud during the test. Ask them to say whatever they are looking at, doing, or feeling.
  • Do not interrupt participants when they are thinking aloud.
  • If the participant loses focus, make them refocus on the task.
  • To quantify your results, use this scale – Strongly agree, Agree, Neither agree nor disagree, Disagree, Strongly disagree.
  • For list based questions, ask the participants to list down the answers in order.
  • Keep the tests short – 15 minutes to 30 minutes.

How can zipBoard help?

zipBoard centralizes the feedback and internal tasks at one place enabling remote visual collaboration. It helps teams better manage feedback from participants and reproduce bugs and issues with screencast and annotation tools that let stakeholders share contextual feedback without leaving their content. It enables better remote collaboration with contextual feedback and task tracking. zipBoard provides a better way to explain issues using screen recording and annotations makes the review process more collaborative and handoffs easier. 

zipBoard is free for one review file/URL (1 project) and unlimited collaborators. The paid plan starts at $99/month (billed annually) for up to 20 review files/URLs and unlimited projects.