Usability testing involves different techniques to ensure that users of a system can execute intended tasks efficiently, effectively, and satisfactorily.
In usability testing, prospective users perform tasks typical of a user group in an ordinary environment with a website, system, or physical product.
Over time, it has evolved from experimental psychology methods to less controlled and more qualitative tests.
Quantitative vs Qualitative Usability Testing
The main differences between quantitative and qualitative testing lie in how the data is collected. Generally, usability testing involves two types of data: qualitative and quantitative.
It is, therefore, vital to understand the distinctions between the two and how to employ each appropriately. The end goal is to gain valuable insights; however, their approach varies.
With qualitative testing, data about behaviours and attitudes are directly collected by critically observing what users do and how they react to a product.
In comparison, quantitative testing accumulates data about users’ behaviours and attitudes indirectly. For instance, quantitative data is typically recorded automatically while participants complete the tasks.
Qualitative usability data examples range from product reviews, user comments, descriptions of the issues experienced, facial expressions, and preferences, etc.
On the other hand, quantitative data typically constitutes statistical data that is quantifiable in numerical terms. For example, how long it took for someone to complete a task, the percentage of a group that clicked a section of a design, etc.
Moderated vs Unmoderated Usability Testing
Moderated usability testing is administered in-person or remotely by a trained moderator. The moderator introduces the test to participants, responds to their queries, and asks follow-up questions. On the other hand, unmoderated usability testing is performed without direct supervision.
Moderated testing typically produces in-depth results because of the direct interaction between moderator and test participants. Moderator is able to probe and follow up with questions to further uncover the underlying motivation or reasons for the participant’s actions.
However, it can be considerably more expensive to organise and run. Conversely, unmoderated testing is cheaper. However, participant answers can provide superficial answers at times.
Explorative vs Comparative Usability Testing
Explorative usability testing is open-ended and involves participants being asked to brainstorm, give opinions, and freely express ideas and concepts.
The information is usually collected in early product development to help researchers pinpoint market gaps, identify potential new features, and iterate new ideas.
Comparative usability testing involves asking users to choose between solutions they prefer to compare a website or app with its primary competitors.
Usability Testing Methods
1. Guerrilla testing
Guerrilla testing is arguably the simplest usability testing method. Essentially, guerrilla testing is as simple as going to a public place like a café and asking people about their thoughts on a prototype.
Basically, test participants are chosen randomly and asked to perform quick tasks, often in exchange for a small gift (like a free coffee). It’s a low-cost approach that works best in the early stages of product development.
2. Lab usability testing
Lab usability testing needs a trained moderator and a suitable place for testing.
This test approach is suitable when you need in-depth information on how real users interact with a product, and the issues they may face.
This method enables you to collect comprehensive and qualitative information. However, it can be expensive to organise and execute since it requires a controlled environment, hiring of test participants and trained moderators.
3. Contextual inquiry
This usability testing method helps a product team obtain information about the user experience from the real users. This method is perfect for attaining rich information about users— for example, their workspace, personal preferences, and habits.
In this approach, users are first asked a specific set of questions about their experience with a product. Subsequently, they are then critically observed and questioned while working within their own environments. For example, a finance clerk is observed on how she use the new accounting software at her office where she usually works.
4. Online or remote usability testing
Usability testing is carried out with participant by connecting with them over the internet. This method of usability testing is able to save time and lessen the need for respondents travelling in person to the usability testing lab. It also provides the added advantage which the participant can join the usability test within the comfort of their home. Screen sharing is usually done online so that the UX researcher can see what the respondent is seeing during the usability test.
5. Card Sorting
Card sorting is a usability testing process where participants demonstrate how they expect a website to look in terms of navigation. This testing method helps UX designers and UX researchers to discover whether their navigational structure matches what users expect.
7. A/B Testing
This usability testing method typically involves critically comparing two versions of an application or website against each other to assess discrepancies. Usually, product managers use statistical analysis to fully determine which of the two versions works better.
The article is a part of our comprehensive guide on “Usability testing”.
During usability testing, the moderator typically collect qualitative data like behavioural observations and participant comments. UX Researchers would also sometimes collect quantitative data like task times and success rates.
By critically analysing this data, the UX researcher seek to identify usability problems within a mobile application or website.
In today’s blog post, we shall delve into usability testing for mobile applications and its underworkings.
What is mobile usability testing?
Mobile usability testing is essentially usability testing performed on mobile products and applications running on mobile platforms.
This usability testing exercise attempts to determine how end users might perceive a mobile app or product, either in a usability laboratory or in a field setting.
Is mobile app usability testing important?
Mobile app usability tests seek to observe test-subject users while utilising a specific app. The core purpose is to measure the application’s user-friendliness to better support a brand’s key commercial objectives.
Mobile usability testing also ensures an app adds value to a business, thus meeting the expectations of the final users.
Generally, ensuring good usability for mobile apps helps improve customer satisfaction, decrease time spent on customer support, and increase overall sales and revenue.
Furthermore, a well-tested and improved application attracts visitors’ attention and helps set up trust and cooperation between the app owner and the app’s potential users.
How to run effective usability testing for mobile apps
Step 1: Define objectives
Before commencing any usability testing exercise, you should set your objectives and goals straight. In essence, you should clearly define the questions you want to answer with the usability test. Or even clarify the hypothesis you want to test with the usability test.
As you define the test’s objectives, ensure you explore these important areas:
- The app’s roadmap (for example if something is important to be tested or it will already be removed in the next update)
- The potential impact of the test
- Existing users and markets for whom the app is targeted
- The app’s existing competitors
- Timing and scope
Step 2: Design the tasks
Once the test objectives have been defined, you then need to set the tasks of the usability test. Tasks should be one sentence long and consist of the interactions to be performed by the test users, for example:
- Register an account
- Upload a photo
- Accept a friend request
- Sign into an account
However, rather than directly asking the test user to execute a task, tasks should be converted into task scenarios. These provide more context to participants about why they are doing the task. And thus, facilitate more natural interactions similar to what an ordinary user will perform with your app.
Consequently, task scenarios that are defined should always be:
- Realistic, actionable and without any obvious clues about how to perform the steps.
- Sequentially ordered to ensure a smooth flow of the test session.
- Tied to one or more objectives.
Step 3: Prepare the usability test documents
When conducting usability testing for mobile apps, there are several documents you typically require. For example:
- Consent forms (for minors and even adults)
- Post-test questionnaire
Ensure these documents are well prepared to suit your user group and their nuances.
Step 4: Prepare the test participants
Mobile usability testing typically involves real users undertaking realistic tasks that the mobile app is intended to achieve.
As you prepare to test with real users, there are several considerations one can take when vetting participants:
- Choose users who are a fair representation of the target audience.
- Users own a mobile device with the exact operating system (including the appropriate version/s) that is being targeted.
- Participants must be available at the time, place, frequency of the intended usability tests.
- Users must agree to the compensation terms that you’re offering (if any)
- Participants must be ready to sign a usability test participation consent form
Step 5: Choose a mobile application usability testing methodology
Principally, there are two main methods for conducting usability testing of mobile applications. These are:
- Laboratory-based usability testing
- Remote usability testing
In this step, weigh the pros and cons of each approach and then pick one that will fit your needs.
Step 6: Reporting the results of the usability test
After collecting your usability test data, the next step is to compile, organise and analyse it to draw meaningful conclusions.
The data can be split between quantitative and quantitative data. For instance, quantitative data encompass completion rates, task times, success rates, satisfaction ratings and error rates.
Or qualitative data like problems experienced, answers provided in the questionnaire, post-test interviews and debriefing sessions.
This step basically involves careful analysing notes on recordings, transcripts, and other information you might have gathered. It then moves on to presenting it in a way that delivers actionable recommendations.
Usability testing is vital to the success of mobile apps. Mobile apps that users perceive as easy to learn, user-friendly and less time-consuming tend to be more profitable and popular.
Furthermore, usability testing should occur on a regular and scheduled basis, especially when introducing new design features, or updates to improve functionality.
The article is a part of our comprehensive guide on “Usability testing”.
Usability is essentially the measurement of a product’s usefulness from the users’ perspective. In principle, usability can be segmented into three fundamental elements: efficiency, effectiveness, and user satisfaction.
As a research methodology, the lack of usability testing during the development of a product often leads to user dissatisfaction and rejection. Another research methodology that serves a similar purpose of ensuring product utility is a field study.
What is the difference between field studies and usability testing?
Field study encompasses all studies of users in their natural environments (usually conducted in the user’s context and location as opposed to your office/lab).
For example, homes, workplaces, neighbourhoods, parks, streets, and shops. Essentially, localities and areas in which one’s product might eventually be utilised are the best places to conduct these studies.
On the other hand, usability testing evaluates a product by directly testing it with representative users.
During a usability test, participants attempt to complete typical tasks while testers/observers watch, listen and take notes.
Overall, the core goal of usability testing is to identify usability problems, collect qualitative and quantitative data to determine the participant’s satisfaction with the product.
When to choose a field study?
Field studies deliver a complete, unbiased picture of what prospective users actually do with a product. They provide context, especially if a product is designed to function in a particular context.
In such instances, conventional lab testing might not give you accurate results. So, when your usability research cannot fit in a lab, you may need to carry these out.
Furthermore, field study is well suited for practical products. For instance, if a device is engineered to function as a core component of the navigational system of an oil tanker. Then visiting the oil tanker for final testing makes sense.
Some of the examples of field studies include user’s office/home visit, ethnographic research, and contextual inquiry.
How to conduct a field study?
1. Preliminary planning
This period involves working with participants. You don’t necessarily require a specific set of research questions but defined study topics. Field studies are strictly observational, as researchers aim to be as unobtrusive as possible.
2. Direct observation
This part of a field study involves watching users critically observe how they behave and why. Ideally, the test subjects do not care that you’re watching, and act exactly as if you are not there.
3. Participant observation
Here, the researcher can join the group of people being studied and records data as field notes or diary entries, after finishing observations for the day.
Benefits of field studies in UX design
Field studies enable you to understand your users in-depth, under realistic conditions. Field studies also allow you to discover social defects and understand environmental factors before releasing products.
Generally, field study advantages can be summarised as:
- It yields highly detailed data.
- It emphasises the role and relevance of social context.
- It can help uncover social facts that may not be immediately obvious or that research participants may be unaware of.
Potential pitfalls of field studies in UX design
The key disadvantage of field studies is their expense. Furthermore, most field studies can’t usually be automated or sped up with technology. This is because they typically rely on old-fashioned theories of patience and observation.
Additionally, if your study design requires a large number of observers, then a field study might not be ideal. Similarly, if your product is to be used in rare, unpredictable circumstances (for example, first-responder mobilisation after an earthquake), then you can’t do a field study.
Further, if your study involves collecting sensitive, confidential information, you might require the more controllable circumstances of a user research lab.
When to choose usability testing?
Usability testing is ideal when:
- Seeking to uncover any issues within your design, workflow, or process.
- Validating if a design works for participants – especially if there is debate on a project team about how something is ‘utilised by users’.
- Seeking insights into both good and poor issues with an interface.
- Seeking different perspectives and mental models on an interface.
You can choose to conduct usability testing:
- Before significant design decisions are made.
- In high-risk, low-certainty situations.
- When it’s time to evaluate and iterate.
- After product launch.
Benefits of usability testing in UX Design
- Improved user experience and the product utility.
- Discovery of hidden usability issues.
- It ensures that the application’s functionality matches the requirements.
- It identifies changes required to improve user performance and satisfaction.
- Helps to analyse product performance to determine if it meets the defined usability objectives.
Disadvantages of usability testing in UX design
- There is sometimes uncertainty about what to test
- There can be testing fatigue
- There is sometimes too much feedback gathered
Field Studies vs Usability Testing Cost
Field studies are expensive and can be highly time-consuming. This is so because of the need to travel, the number of hours researchers are required to commit, and the complex analysis that open-ended, unstructured research dictates.
On the other hand, usability tests are relatively inexpensive and easy to conduct, especially if conducted remotely. However, some of these can only be conducted in a specially designed laboratory, enabling facilitators to interact with, and observe users. Such tests can involve some cost, but they will still be cheaper than field testing.
In summary, usability testing is an inexpensive means of gathering valuable feedback from representative users.
In contrast, field studies allow user researchers to gain first-hand experience and knowledge about the users and the processes they study.
The article is a part of our comprehensive guide on “Usability testing”.
Choosing an appropriate usability testing tool is vital to executing an effective user experience. Typically, the choice of tool depends on the type of usability testing method you’ll employ.
It is true that you can run usability testing without a specialised tool, for example, when doing qualitative, in-person usability testing. However, if you are planning remote usability testing sessions, or seeking to collect quantitative data, then a dedicated tool is necessary.
What are usability testing tools?
Usability testing tools are dedicated solutions that essentially streamline usability testing activities. These solutions can be utilised to gather actionable insights into how prospective users might perceive and experience your product.
For example, they can be used to put a prototype or website in front of real users. Thus, enabling website owners to analyse how users accomplish a given task or to identify possible frustration points.
Read our article on “How to conduct usability testing for websites?”
Generally, usability testing tools smoothen the process of accessing and tracking users’ feedback. They also help with the analysis of that feedback to enable product owners to make data-informed decisions and deliver the best user experience.
Top 10 Usability Testing Tools
Maze is a rapid user testing platform for collecting both qualitative and quantitative usability data. This platform seamlessly integrates directly with Adobe XD, Marvel, Sketch Figma, InVision.
Furthermore, Maze allows users to create and run in-depth usability tests, then share those tests with testers via a link. Maze includes varied features like task analysis, guerrilla testing, multiple path analysis, heatmaps, A/B testing, and wireframe testing.
Maze also generates an instant usability test report for each test that users can share with anyone via a link.
This reporting functionality presents usability testing results like completion rates, misclick rates, and time spent. Additionally, Maze allows users to run research surveys and test their information architecture using Card Sorts and Tree Tests.
This is a user experience (UX) screen recording tool for UX designers and product managers who seek to analyse how users interact with their applications.
Lookback’s testing capabilities enable designers to view what users see and get their reactions in real-time. This is either recorded or in-person.
For instance, if you set up a remote test, participants shall receive a link to download your app. Then they get started with a live session or self-test, and you can even communicate directly with them. Subsequently, user recordings automatically appear in a dashboard where one can organise them into groups and create highlights.
As one of the most popular usability testing tools in the market, Userlytics a UX research service that offers picture-in-picture user recording (webcam view + screen and audio recording).
It allows webmasters to go in-depth with the criteria and traits they seek from participants. Userlytics also enables users to create screening questions that disqualify users from the study. The platform goes deep into segmentation for the studies, thus rendering more accurate results.
Furthermore, Userlytics also offers advanced quantitative tools like card sorting and tree testing that can be integrated with one’s qualitative usability tests.
4. Crazy Egg
Crazyegg is a click-based user experience tool with features like ‘Heatmap’ logs that show where each visitor clicked on your webpage. Or even a ‘scroll map’ that shows how far down the page each visitor usually scrolls.
It also offers an overlay feature that breaks down the number of clicks on each page element. It also presents detailed insights about visitor sources, search terms, and other components.
Optimizely is a popular A/B Testing platform that enables users to track visits and conversions. The tool boasts a range of features like mobile website testing, geotargeting, cross-browser testing, visitor segmentation and multivariate testing.
The Qualaroo usability testing tool prompts users to answer targeted questions and surveys in real-time on test sites. This unique tool has the ability to integrate with other tools, like Salesforce and Marketo.
It also has exit surveys that webmasters can utilise to discover why site visitors don’t convert. Additionally, it has a ‘Skip logic’ capability to analyse visitors’ responses before targeting them with a custom follow-up question.
7. Feedback Army
This user testing tool utilises Amazon’s Mechanical Turk service to solicit user responses. Its three-step process is quicker than setting up your own Mechanical Turk test.
It basically involves:
- Uploading the URL of the page that you seek to test.
- Setting three to six questions about the page.
- Waiting for your ten responses to arrive.
8. Userzoom Go
Userzoom Go is a comprehensive user research system that features a dedicated testing platform, recruiting services, and automated reporting capabilities.
Essentially, it focuses on enabling study organisers to focus on the tasks and questions while it handles it recruits, sets up, and carries out the tests.
The Loop11 usability testing tool focuses on unmoderated remote tests. It offers a set of tools like A/B testing, testing of live websites or prototypes, heat maps and clickstream analysis.
Because Loop11 doesn’t maintain an active participant database, users are required to bring their own participants.
HotJar offers heatmaps as its key feature. Its data visualisation has earned it positive reviews with users as it presents data in an easy manner to draw actionable insights.
Furthermore, its heatmaps, move maps and scroll maps can be easily downloaded with one click. It also offers a recording feature for users to see the exact behaviour of participants. Additionally, it provides an option to create surveys easily.
The article is a part of our comprehensive guide on “Usability testing”.