Before we talk about Qualitative Usability Tests
We first need to talk about what a Usability Test is in general.
A Usability Test is a method used to evaluate how usable or intuitive your product is. This is done by observing and analysing representative users as they perform specific tasks in your product. Usability Tests allow you to identify any usability problems, collect qualitative and quantitative data, and determine your users’ satisfaction with your product or proposed designs early on in the design cycle.
There are two types of Usability Tests; Quantitative Usability Tests and Qualitative Usability Tests. Quantitative usability testing is focused on collecting UX metrics (like time on task or task success) through controlled, specific tasks. A qualitative usability test has more open-ended tasks and prioritizes observations, like identifying usability issues or user insights.
For this method, we are going to focus on qualitative usability tests, but you can find information on quantitative usability tests here.
What is a Qualitative Usability Test?
A Qualitative Usability Test is a method that prioritizes observation when participants perform open-ended tasks in a system as you collect feedback on usability issues or user insights.
Qualitative Usability Tests are a type of Formative User Research, which basically means that the results from these tests inform how the design will evolve during the design process, however, they can also be performed on your existing product in order to inform the next round of design changes.
Since the goal of this method is to guide your designs, qualitative usability tests are effective even with smaller sample sizes as you can get a lot of contextual feedback and quickly iterate to achieve your goals.
Some cons of qualitative usability tests are that they can be time-consuming, recruiting can be a challenge, and they can take a long time to analyse. Remote moderated testing can help make scheduling easier, and there are paid services out there to help with that, however, they can be quite expensive.
What do you need for a Qualitative Usability Test?
How do you conduct a Qualitative Usability Test?
Step 1: Decide what you want to test and why
As mentioned above, qualitative usability tests can be run on early designs or your finished product. You just need to know what you want to test and why. For example, are you looking to evaluate certain areas of your existing product to uncover new potential features or fixes? Or are you looking to evaluate your early designs to see if you’re on track and to guide the next round of quick iterations?
Step 2: Decide if you want moderated or unmoderated
Your next decision is to decide whether it will be moderated or unmoderated. A moderated test involves the active participation of a trained facilitator, while an unmoderated test is completed by test participants in their own environment without a facilitator present.
Unmoderated tests are faster, and allow for greater sample sizes in a short amount of time, but there are several cons; you can’t ask follow ups, the data has higher variability, and there is a risk of “Cheater” participants (which are participants that are only in it for compensation and do not provide substantial feedback).
Step 3: Create your test plan and tasks
Next, you need to create your test script and tasks. Come up with as many tasks that you want feedback on and that will fit within your timeframe. Your tasks here should be open-ended and the flows your participants interact with should support the entire task and any possible edge cases.
Some examples of qualitative tasks would be:
- Find a pair of runners that you like and pay for them using our site
- Find a way to get help on the site
- Use this dashboard to analyze your data
Step 4: Schedule and recruit participants
Once you know what you want to test, you will want to find participants. Ideally they are representative of your target audience and have the right amount of experience for the flows you are testing. For example, if you’re testing for feedback on a new sign-up flow, you may want participants that have never used your product before. If you’re testing an advanced feature, you may need participants with a lot of experience with your product.
Qualitative usability tests are effective with small sample sizes (around 5 participants per round). The logic behind this is that after around 5 participants, you will start getting too much repetitive feedback, and it will no longer be worth your time and effort. Instead, you’re encouraged to make changes to your designs, then do another round of tests. Repeat this process until you feel you’ve reached your goal (or until you run out of time).
This makes scheduling easy, since you only need 5 per round, but depending on how many rounds you do, scheduling could become difficult.
Step 5: Have them go through the tasks and note your observations
Now you can have them go through the tasks while you record and document it. During the tasks, ask them to “think out loud”. Encouraging participants to talk through what they are looking for and why will unlock deeper insights into what they want, which will give you plenty of ideas on how to address it. If they start to get quiet, you can always ask “what are you thinking now?”.
Step 6: Ask questions
In between each task, or once all tasks are complete, you can bring in some other research methods, like Quantitative Surveys and Qualitative Surveys, to get some additional feedback on the specific task or overall flow. You can even interview them with another set of questions, however this can only be done in moderated tests.
When listening to their answers, it’s important to prioritize their actions in the tasks, rather than what they said in their responses. For example, they may tell you that they found the task easy, but in reality, they may have struggled for a long while at various points in the task.
Step 7: Analyze and report your findings
Pull out the main takeaways from each task then cross reference those takeaways with the results from the other participants. This will surface any patterns or trends in the feedback and possibly point to deeper insights. It will also allow you to make design proposals based off of user data, instead of just your opinions.
Once you have this all in an easy-to-digest report, share your findings with your team and stakeholders. Make sure to include the main insights at the top, alongside any proposals you have on how to improve the designs before your next round of Usability Tests, or before you hand the designs over to the dev team.