Research Methods

Question 1

Focus Groups

Answer 1

Guided group discussions of 6-12 people
Often used to collected attitudes and relevant experiences related to a specific product, service, marketing campaign, or brand.

Advantages
Good for desires, motivations, values, memory
Participants can bounce ideas off one one another

Limitations
Have to be careful to reduce risk of group think, conversation being dominated by one stakeholder

Question 2

RITE (Rapid iterative testing and evaluation)

Answer 2

Iterative usability method in which the assets are modified or changed between tests based on each session’s feedback.

Advantages
Great for early stage, lean feedback
Encourages team alignment in design decisions

Limitations
Requires strong team commitment
If testing higher fidelity or functional prototypes - making quick changes can be a challenge
Fixing one problem might introduce several new problems

Question 3

Literature Review

Answer 3

Provides a concise summary of existing research about a research questions. Can also extend to more detailed synthesis that re-organizes literature and creates an argument.

Advantages
Relatively quick, great return on investment

Limitations
External validity
Paywalls for sources

Question 4

Structured interviews

Answer 4

Interviews with fixed sequence of questions that are not flexible either by moderator or participant.

Advantages
Can be more efficient in moderation required
Continuity in questions allows for session comparability, and metrics to be collected

Limitations
May be more difficult to build rapport
Limits surfacing of unexpected stories

Question 5

Intercept Study

Answer 5

A quick way to get a product or concept tested in the wild when you have a specific, narrow set of topics to explore

Advantages
Best for short studies with very little screening required

Limitations
Risky with unreleased products or related assets
Probing not allowed

Question 6

Dyad (Paired) Interviews

Answer 6

An interview with 2-3 participants in a session together who have a pre-existing relationship.

Advantages

• Captures social dynamics that would otherwise be missed with an individual participant; they can build off of each others answers and keep each other honest
• Easier to build rapport and comfort with participants; especially proven effective with young children and teens

Limitations

• Cancellation, snowball sampling
• Careful about what you introduce (e.g., couples, arguments)

Question 7

Unstructured Interviews

Answer 7

Interviews with the most conversational flexibility; often the researcher has a set of questions or topics to address, but the conversation may change direction so long as the topics are relevant.

Advantages

• Great for exploration, deep understanding, especially when tackling spaces, experiences, cultures, or settings that aren’t yet fully understood
• Often easier to build rapport

Cons

• Inability to compare sessions or quantify
• Setting stakeholder expectations (don’t know fully what we’ll have answered or surfaced)

Question 8

KANO model

Answer 8

Method for measuring the effect of different product or service attributes on user satisfaction. Includes a functional and dysfunctional scenario, leades to placement into one of 5 categories (e.g., tablestake, performance, delighters)

Pros:

• Ability to categorize and prioritize product features / roadmaps
• Provides a metric for the interest level of a feature

Cons:

• Short shelf live; expectations change quickly in fast moving markets (what delights today may be a tablestake tomorrow)
• Features need to be non-technical and easy for participants to comprehend
• Issues with hypotheticals in surveys

Question 9

Playtest

Answer 9

Observing participants interacting with a product or service in an unstructured fashion. Typically used with live builds that have already been extensively QAed.

Advantages

• Allows the holistic study of the natural context of use (progression, likeability), and to surface unexpected painpoints or attitudes.
• Efficiently run a study with a greater sample size

Limitations

• Not great at more deeply understanding “why,” since researcher intervention in the study is limited
• Lots of concurrent and retrospective moderation required; need lab space and technical support for a wider range of participants

Question 10

Remote, unmoderated research

Answer 10

Method that automates the collection of peoples attitudes and behaviors. A collection of methods (e.g., diary studies, click testing, usability evaluation, card sorting)

Advantages

• Can conduct studies with slightly higher statistical power
• Reduces researcher hours and resources
• Avoids some forms of researcher bias
• Perhaps more ecological validity than lab-based testing

Limitations

• Inability to incorporate probes
• All assets need to be heavily QAed

Question 11

Individual-centered behavioral mapping

Answer 11

Method that maps the activities and travel of a specific individual or group over time and location.

Advantages:
- Great for illuminating social behaviors and interactions (rather than, say, an office or set space)

Limitations:
- Slightly intrusive, need to allow time for participants to become less reactive to being followed

Question 12

Cognitive Walkthrough

Answer 12

Usability inspection method that examines the learnability of a new product or system from the perspective of a first-time or one-time user.

You break down the “first session,” including all onboarding, into a series of steps. A team of evaluators then evaluate each step according to 4 key questions (e.g., will users want to produce the action of the step? will users discover the signifier for the affordance (e.g., button, label)?).

Advantages:

• Speedy, lightweight method for quick evaluation
• Helps to prioritize topics for usability testing, since you often won’t have time to evaluate each step

Limitations:

• Complementary to usability testing, not meant to replace
• Requires “thinking caps” and role playing, which may be challenging depending on evaluators

Question 13

Contextual Inquiry

Answer 13

An ethnographic method that places researcher on-site as a participant in research. The researcher observes and interviews (asks questions), in an apprentice-like fashion, in order to understand communication flows, tasks sequences, tools and artifacts, and physical environments.

Advantages:
- Allows for the examination of tacit knowledge and underlying (and invisible) work structures

Limitations:
- Need to CONSTANTLY check your interpretation to ensure it aligns with the participants

Question 14

Crowdsourcing

Answer 14

A quantitative method where you allow a large population of people to voluntarily opt-in to complete microtasks.

Microtasks can range from classifying data, answering questions, to providing feedback.

Advantages:

• Quickly and cheaply collect a large volume of data
• When testing hypotheses, representation doesn’t matter (as much)

Limitations:
- Limited ability to collect demographic information, intentions, expertise. Thus, harder to generalize to an overall population.

Question 15

Narrative Interview

Answer 15

Researcher asks a series of open-ended questions that direct the participant to stories of lived experiences.

Follow-up guidance: who, what, when, where, how. Less of a question-answer format, and more of a narrator-listener format.

Advantages
- A way to collect rich experiences from participants when time constraints (e.g., no time for longitudinal methods or direct observation)

Limitations
- Requires rapport and sensitive handling

Question 16

Desirability Testing

Answer 16

Iterative method where participants are provided with a fixed vocabulary to select from in order to describe a design. Participants typically select 3-5 cards, and are asked to explain their choices.

Advantages

• Can build consensus when selecting alternative design directions based on how well each elicits intended emotional responses
• Helps non-designers express their feelings towards a product or service

Limitations

• Need clear design hypotheses and goals, from which ideal attributes can be selected
• Interpretations of vocabulary words can vary; good to hear explanations and pick unambiguous terms

Question 17

Diary Studies

Answer 17

Longitudinal research method where you expose participants to a series of questions and tasks over the course of ~1 week. Participants submit entries throughout the course of that time.

Advantages

• A way to collect in-situ feedback without observation
• A useful way to collect infrequent experiences, or experiences that may be too sensitive for intrusive observation

Limitations

• Ensuring participant adherence
• Lacks the richness of observation
• Best paired with other qualitative methods to be able to use probes

Question 18

Experiments

Answer 18

A method used to test for causal relationships between variable. Typically you have a control group, followed with 1 or more treatment groups. All other factors should remain constant between the groups, aside from the experimental variable.

3 types of experiments: lab-based, internet-based, and field-based experiments

Advantages:

• The best way to determine causal relationships
• Can be used both to test hypotheses and mitigate risks when developing product updates

Limitations

• Consider the estimated effect size; if small, then consider the extensive sample required to detect the effect.
• If lab or field-based, consider the drawbacks; need for consistency in protocol, whether you need to conduct it blindly, whether you can actually control assignment of participants to groups.

Question 19

Difference between exploratory and generative research

Answer 19

Foundational / Exploratory / Pathfinding

• Foundational research, sometimes referred to as “exploratory” or “pathfinding” research, centers on a problem or topic that hasn’t yet been clearly defined. It is critical in all research fields, helping teams generate initial understanding, establish priorities, and anticipate and adapt to changes in the competitive landscape.
• User-centered, rather than product or paradigm centered

Generative Research

• Typically comes into gear when you’ve defined a problem space and you’re now looking to prioritize relevant opportunities and solutions
• Very slightly more product centered, since you’re looking for viable product solutions ultimately

Question 20

Two kinds of generative research

Answer 20

• Projective, Constructive

Projective

• Expressive exercises used at very early stages of design - thoughts, feelings, desired elicited
• Artifact creation can support engaged and comfortable conversation (e.g., collage, drawing, diagramming)

Constructive

• Used at later stages of design and concept development, after parameters have been set for ideating.
• E.g., we’re going to design a calendaring system - flexible modeling can be used to determine appropriate components; set of components will be varied, fixed, and not too wide as to overwhelm them.

Question 21

Heuristics evaluation

Answer 21

A usability inspection method where you inspect a set of designs according to a set of selected usability heuristics (principles).

Advantages:

• Lightweight, efficient
• Can identify major usability painpoints prior to a usability test (similar to RITE method)

Limitations:

• Ideally requires 3-5 evaluators who are trained on heuristics
• Don’t just call out violations; call out good usage of heuristics as well

Question 22

Field Research (In-Home Visits)

Answer 22

A method for collecting in-situ feedback within the contexts that people may use or consider products of interest.

Often includes exercises such as home tours, out of box experience, card sorts, generative methods. Debriefs critical between sessions

Advantages

• Participants in their own turf (ecological validity, rapport, comfort)
• Deep, in-situ observation with ability to probe; can cite artifacts and “show” you what they mean
• Stakeholder buy in and participation (developing empathy through immersion)

Limitations

• Time required (planning, stakeholder preparation, travel, participant recruitment)
• Lighter methods to get in-situ feedback; know when this commitment is required
• Personal safety risk (e.g., doing them alone, alergens, etc)
• Observer effect (participants have no preconception of what a home visit entails - may act differently when being observed; place recording devices discretely)

Question 23

Love Letter and Breakup Letter

Answer 23

Participants are recruited to write letters to product in personified form, and express feelings of dislike and like toward the product.

Advantage

• Can elicit deeper feelings about a product that can help to ground how products and brands should position themselves against empathy and needs
• Works great as part of other qualitative methods or in group workshops

Limitations
- Important to capture their feelings (not just the letter) through video and audio