UX Research Flashcards
What presentation on UX research should consist of
- Why we needed this research
- Key findings
- Influence on design
What is saturated answer to research question
It means that new interviews will not give us any new information
What is a good qualitative research
When as a result we clearly understand what to do
Why do we need research?
To understand where we now and which plan do we need to build the better reality
Qualitative or quantitative to choose
Both. Quantitative on their own explain nothing and qualitative prove nothing
Specifics of qualitative and quantitative research
- Quantitative — all about numbers, answer to ‘What’
- Qualitative — pains, needs, reasons, insights/ answer to ‘Why’
Quantitative or Qualitative?
Quant — more respectful, numbers are easy to understand
Qual — more people-oriented, why better understand the reasons behind
Phrase about Quant and Qual
Quant research alone explains nothing, Qual research alone proves nothing
Which type of research is better for a designer?
Qualitative, because it’s more people-oriented and gives insights
Types of qual research
- Usability testing (can be also quant)
- In-depth interviews
- Observations
- Context interviews (while the user is using some interface)
- Focus-groups (not so good for designers)
- Diary testing (when we want analyse experience for the period, long-term usability)
- Workshops (participative design)
Who is SME
A subject-matter expert (SME) is a person who has accumulated great knowledge
Sources for qual study
- Users (real and potential)
- Not-users (used and quitted or didn’t come)
- SME (subject matter experts)
- Owner (product owner)
Triangulation
For:
- Credibility
- Complexity
- Depth
- Width
Triangulation examples
- Start with quant study (or google analytics), discover problems, build hypotheses
- Continue with qual, go deeper
- Then back to quant, to check/prove
- In-depth interview (calm and long), context interview (quicker and more hot), open survey (anonymous)
- Triangulate people (researchers) to avoid confirmation bias
The difference between research and study
Research is a discipline, study is a specific research for the specific project
Divergent and convergent thinking
Phases of this process are either diverging or converging. During a diverging phase, you try to open up as much as possible without limiting yourself, whereas a converging phase focuses on condensing and narrowing your findings or ideas.
A landscape of user research methods
The Attitudinal vs. Behavioral Dimension
Kind of contrasting “what people say” versus “what people do” (very often the two are quite different).
About card sorting
Card sorting is a UX research method in which study participants group individual labels written on notecards according to criteria that make sense to them.
Provides insights about users’ mental model of an information space and can help determine the best information architecture for your product.
Let’s imagine that you’re designing a car-rental site. Your company offers around 60 vehicle models that customers can choose from. How would you organize those vehicles into categories that people can browse to quickly find their ideal car rental? Your company might use technical terms such as family car, executive car, and full-size luxury car. But your users might have no idea of the difference between some of those categories. This is where card sorting can help: ask your users to organize vehicles into groups that make sense to them, and, then, see what patterns emerge.
Conducting the card sorting
Generally, the process works as follows:
- Choose a set of topics. The set should include 40–80 items that represent the main content on the site. Write each topic on an individual index card.
Tip: Avoid topics that contain the same words; participants will tend to group those cards together.
- User organizes topics into groups. Shuffle the cards and give them to the participant. Ask the user to look at the cards one at a time and place cards that belong together into piles.
Some piles can be big, others small. If the participant isn’t sure about a card, or doesn’t know what it means, it’s ok to leave it off to the side. It’s better to have a set of “unknown” or “unsure” cards than to randomly group cards.
Notes:
There is no preset number of piles to aim for. Some users may create many small piles, others may end up with a few big ones. It all depends on their individual mental models.
Users should be aware that it’s OK change their mind as they work: they can move a card from one pile to another, merge two piles, split a pile into several new piles, and so on. Card sorting is a bottom–up process, and false starts are to be expected.
- User names the groups. Once the participant has grouped all the cards to her satisfaction, give her blank cards and ask her to write down a name for each group she created. This step will reveal the user’s mental model of the topic space. You may get a few ideas for navigation categories, but don’t expect participants to create effective labels.
Tip: It’s important to do this naming step after all the groups have been created, so that the user doesn’t lock herself in to categories while she’s still working; she should be free to rearrange her groups at any moment.
- Debrief the user. (This step is optional, but highly recommended.) Ask users to explain the rationale behind the groups they created. Additional questions may include:
- Were any items especially easy or difficult to place?
- Did any items seem to belong in two or more groups?
- What thoughts do you have about the items left unsorted (if any)?
You can also ask the user to think out loud while they perform the original sorting. Doing so provides detailed information, but also takes time to analyze. For example, you might hear the user say, “I might put card Tomatoes into pile Vegetables. But wait, they are really a fruit, they don’t really fit there. I think Fruits is a better match.” Such a statement would allow you to conclude that the user did consider Vegetables a decent match for Tomatoes, even though Fruits was even better. This information could push you into crosslinking from Vegetables to Fruits or maybe even assigning the item to Vegetables if there are other reasons leaning in that direction.
If needed, ask the user for more-practical group sizes. You should not impose your own wishes or biases upon the participant during the original sorting (steps 1–3), but once the user’s preferred grouping has been defined, and after the initial debrief, you can definitely ask the participant to break up large groups into smaller subgroups. Or the opposite: to group small groups into larger categories.
Repeat with 15–20 users. You’ll need enough users to detect patterns in users’ mental models.
Open Card Sorting vs. Closed Card Sorting
- Open card sorting is the most common type of card sort and what we described above. Generally, when practitioners use the term card sort, it’s implied that it will be an open card sort. In an open card sort, users are free to assign whatever names they want to the groups they’ve created with the cards in the stack.
- Closed card sorting is a variation where users are given a predetermined set of category names, and they are asked to organize the individual cards into these predetermined categories. Closed card sorting does not reveal how users conceptualize a set of topics. Instead, it is used to evaluate how well an existing category structure supports the content, from a user’s perspective. A critique of the closed card sort is that it tests users’ ability to fit the content into the “correct” bucket — to users, it can feel more like solving a puzzle than like naturally matching content to categories. The method does not reflect how users naturally browse content, which is to first scan categories and make a selection based on information scent. Instead of closed card sorting, we recommend tree testing (also known as reverse card sorting) as a way to evaluate navigation categories.
Moderated vs. Unmoderated Card Sorting
- Moderated card sorting includes step 4 in the process outlined above: the debrief (and/or think-aloud during the actual sorting). This step is a highly valuable opportunity to gain qualitative insights into users’ rationale for their groupings. You can ask questions, probe for further understanding, and ask about specific cards, as needed. If it’s feasible for your schedule and budget, we recommend moderating your card sorts to get these insights.
- Unmoderated card sorting involves users organizing content into groups on their own, usually via an online tool, with no interaction with a facilitator. It is generally faster and less expensive than moderated card sorting, for the simple reason that it doesn’t require a researcher to speak with each user. Unmoderated card sorting can be useful as a supplement to moderated card sorting sessions. For example, imagine a study involved highly distinct audience groups, and the research team decided to run a card sort with 60 users: 20 users for each of 3 different audience groups. In this case, it can be cost-prohibitive to run 60 moderated card-sorting sessions. Instead, the team may decide to do a small study of 5–10 moderated sessions for each audience group, followed by unmoderated card sorting for the remaining sessions.
Paper vs. Digital Card Sorting
- Paper card sorting is the traditional form of card sorting. Topics are written on index cards and users are asked to create their group on a large workspace. The biggest advantage to paper card sorting is that there is no learning curve for the study participants: all they have to do is stack paper into piles on a table. It’s a forgiving and flexible process: users can easily move cards around or even start over. It’s also easier for people to manipulate a very large number of cards on a big table than it is to manipulate many objects on a computer screen that often can’t show everything within a single view. The downside of paper card sorting is that the researchers have to manually document each participant’s groups and input them into a tool for analysis.
- Digital card sorting uses software or a web-based tool to simulate topic cards, which users then drag and drop into groups. This method is generally the easiest for researchers, because the software can analyze the results from all the participants and reveal which items were most commonly grouped together, what category names users created, and the likelihood of two items being paired together. The downside is that the usability of the tool can impact the success of the session — technology problems can cause frustration or even prevent users from creating the exact groups that they want.
Card sorting VS tree testing
Card sorting is invaluable for understanding how your audience thinks, but it does not necessarily produce the exact categorization scheme you should follow.
For example, participants in a card sort often create a generic category to hold a few items which don’t seem to fit anywhere else; this is understandable, but if you were to actually include an “other stuff” category in your menu, the same users would avoid it like the plague.
(Website visitors are notoriously reluctant to click on vague labels because they quite rightly suspect they’ll have to do a lot of work to sift through the content.)
For best results, a card sort should be followed up by a tree test to evaluate the proposed menu structure.
What is a tree testing
A tree test evaluates a hierarchical category structure, or tree, by having users find the locations in the tree where specific tasks can be completed.
Tree testing is incredibly useful as a follow-up to card sorting because it:
- Evaluates a hierarchy according to how it performs in a real-world scenario, using tasks similar to a usability test; and
- Can be conducted well in advance of designing page layouts or navigation menus, allowing inexpensive exploration and refinement of the menu categories and labels.
To conduct a tree test, you don’t need to sketch any wireframes or write any content. You only need to prepare two things: the tree, or hierarchical menu, and the tasks, or instructions which explain to study participants what they should attempt to find.
Risks during tree testing
Your tree should be a complete list of all your main content categories, and all their subcategories. Even if you are interested in testing only a specific section of the tree, excluding the other sections is risky because it assumes that users will know which section to go to. For example, if your website had both a Products and a Services category, and you chose to test only the Products tree, you would miss out on finding whether your audience understands the difference between these two categories.
Competitive tree testing
If you are considering different labels for the same tree category, you may want to test two different trees in order to compare how the terms perform. Such a test is especially easy to do with Userzoom’s tree-testing tool, which allows you to randomly assign participants to different versions of the tree, in a manner similar to an A/B test on a live website. If you do test multiple trees, avoid showing the same user two alternative trees in the same session — users’ behavior when interacting with the second tree would be skewed by their experiences with the first one.
Testing locations via tree test
There’s no need to prepare and test a separate tree if you just want to compare different locations for a label — such as whether tomatoes should be placed under Fruits or Vegetables. Instead of testing two different trees for each location, you can test a single tree and compare how many users clicked Fruits vs. how many clicked Vegetables. (You’ll also be able to tell which category they tried first, if they clicked on both.)
Digital tools for tree testing
Userzoom and Treejack are both good options for conducting tree testing.
Tree testing tasks
Ideally you should include tasks which target:
- Key website goals and user tasks, such as finding your most important product (Success rates in your primary navigation tasks can serve as a baseline against which you can compare secondary tasks, and a reference point for future testing.)
- Potential problem areas, such as new categories proposed by stakeholders or participants in a card sort
Tree testing task phrasing
- Find information about starting a business.
- You are moving to Santa Fe next year, and once you arrive you would like to supplement your income by opening a side business providing lawn-care services. Find out what regulations you will need to follow.
- You are considering opening a lawn-care service. See if there are any resources on this site that can help you begin the process.
3rd is ❤️
Tree testing limitations
Tree testing is often executed as a remote, unmoderated study. After recruiting representative users, you simply send them a link to the study, and the testing tool walks them through the process of completing the tasks using their own computer. The testing tool is much better than a human would be at keeping track of exactly which categories users click on.
However, this format does not capture the full context of user behavior (such as comments made while performing a task) and you can’t ask personalized follow-up questions.
To minimize the effects of the format, conduct at least a few moderated pilot sessions before collecting the bulk of your data. In these moderated sessions you can ensure the task wording is understandable and also get a chance to pick up on nuances that might otherwise be hard to spot in the quantitative data. For example, in a recent tree test we noticed in the pilot testing that many users avoided a certain category for the first half of their session, because the label was so broad that they feared the contents would be overwhelming. This trend wasn’t noticeable in the quantitative results due to the task order randomization, but it was quite obvious as you sat through each session and saw task after task where users ignored an obvious choice. That insight alone made the pilot testing a day well spent.
You can also partially compensate for the inability to ask follow-up questions by including a short survey after the tree test. Rather than asking users to recall any labels they found confusing, provide them with a list of labels and ask them to check which were difficult to understand. This question can be followed up with an open-ended question inviting users to share any further comments and feedback, to elicit unexpected assumptions or misunderstandings that may not be apparent from the click history.
Generative research methods
Research goal:
Find new directions and opportunities
Field studies, diary studies, interviews, surveys, participatory design, concept testing
Formative research methods
Research goal:
Improve usability of design
Card sorting, tree testing, usability testing, remote testing (moderated and unmoderated)
Summative research methods
Research goal:
Measure product performance against itself or its competition
Usability benchmarking, unmoderated UX testing, A/B testing, clickstream / analytics, surveys
Usability testing (aka usability-lab studies)
Participants are brought into a lab, one-on-one with a researcher, and given a set of scenarios that lead to tasks and usage of specific interest within a product or service.
Fild studies
Researchers study participants in their own environment (work or home), where they would most likely encounter the product or service being used in the most realistic or natural environment.
Amounts of users for each type of study
