Week 4 User Study Flashcards
Why Do a User Study?
Best way to learn about your design
Get feedback from users
Learn about things you had not
considered
Understand users, and what they
want/need from your design
Types of Data
Quantitative and Qualitative
What is Quantitative Data?
Expressed as numbers (size,
magnitude, amount)
Measurable, statistics
Can test specific
characteristics or
hypotheses
What is Qualitative Data?
Detailed descriptions,
subjective, “rich”
Represents themes, patterns,
stories
Can be observed but not
measured
Open-ended enquiry
Comparison of Qualitative VS Quantitative Examples
Quantitative
* Number of clicks
* Performance scores
* Task durations in seconds
* Likert scale responses
* Number of successful trials
* Number of computers you own
Qualitative
* Explanation of why someone chose an item
* Description of a process
* Story describing someone’s experiences with
technology
* Someone’s understanding of how a system
works
* Area of expertise
* Favourite operating system
Subjective data
an individual’s opinion, preference, experience, or judgment; not from some external measure.
Quantitative: The user rates this system
8/10 for ease of use
Qualitative: The user thinks computers are
too expensive
Objective data
‘external to the mind’ and concern facts and measurements.
Quantitative: The user took 45 seconds to
complete the task
Qualitative: The user owns an Apple phone
Data Gathering How to?
Ask the user: Questionnaire, Interview
Observe interaction: Direct observation, Think aloud, Co-discovery learning
Measure performance: Time, Physiological data, #errors
Questionnaire: Design Considerations?
Responses can be influenced
by order of questions
Provide clear instructions
Balance between whitespace
and being compact
Wording is very important (Negative, positive, neutral)
Questionnaire Types:
Multiple Choice, Likert Scale, Semantic, Open-Ended
“Conversations with a Purpose” - Pros and Cons
Pros:
excellent for pursuing specific issues,
address specific questions of interest,
more flexible than questionnaires,
Cons:
accounts are subjective,
prone to rationalization,
time consuming to conduct and
to analyze,
participants may have difficulty,
evaluator can bias the interview,
Interview Types
Structured
Unstructured
Semi-structured
Unstructured
More like a conversation, can often go into depth
Semi-structured
Some pre-determined directions but
flexible as situation evolves
Structured
Pre-determined set of questions
* Aka, a standardized interview
Focus Groups (Group Interviews) and Pros and Cons
2 – 10 people interviewed
at one time
A skilled moderator is critical
Usually recorded
+ can accommodate diverse
and sensitive issues
+/- opinions developed within
a social context
+ good way to identify
“proto-users”
- some interviewees may
dominate
Direct Observation
User is given the task
* and evaluator just watches
the user without interruption
Problem
* no insight into the user’s
decision process, opinion…
Think-Aloud
Most widely used method in
industry
User are asked to say what
they are thinking/doing
Gives insight into what the
user is thinking
Think-Aloud Problems
Awkward for the user
“Thinking” about it may alter
the way people perform their
task
Hard to talk when
concentrating on task
Co-discovering Learning
Normal conversation
between the two
users
Removes
awkwardness
Provides insights
Recording Observations Techniques
Paper and pencil (or laptop)
Audio recording
Video recording
Recording Observations - Pros
Possible to do in a real-life
context or in a lab
Rich data
See what’s happening for
yourself
Recording Observations - Cons
Observer’s presence can
disrupt work
Can be difficult to analyze or
reproduce
Potentially time-consuming
Performance Data
Timestamps, timing
User clicks, selections, data
entries
Scores
Task outcomes
Data capture
Physiological Data
Eye-tracking
Heart rate, breathing rate
ECG (cardio), EEG (brain)
Skin conductance
Motion
Requires specialized equipment
Types of Study
Laboratory
Field
Web
Extra: Crowd-Sourced Study
Lab-Based/Controlled Study
Characteristics
* Controlled environment
* Direct observation
Used for
* Measuring performance & improvements
* Discovering usability problems
* Gaining confidence of design before riskier testing
Potential Drawbacks
* Realism can be difficult to achieve
* Novelty effects (hey, this is cool!)
Field Study
Characteristics
* Deployed in regular setting
* Maybe more ecologically valid
* More expensive and time-consuming
Used for
* Evaluate real-world usage
* Gather long-term data
Potential Drawbacks
* Riskier for security, privacy, reliability perspectives
* Influenced by outside factors, unsupervised
* Data much more “messy
Web-Based/Remote Study
Characteristics
* More natural setting
* Uncontrolled (realistic) environment
* Can prompt to complete tasks, screenshare, videoconference
Used for
* More convenient than lab studies
* Middle ground between lab and field studies
Potential Drawbacks
* More difficult to observe
* Technical challenges
Crowd-Sourced Study
Characteristics
* Fast and cheap data collection through online crowdsourcing
Used for
* Quickly gathering large amounts of data
* Studies that can be broken into small tasks
Potential Drawbacks
* Data quality/reliability
* Need fully automated study protocol
Triangulation
Explores the same topic in multiple ways
* to gather more holistic
understanding, increase
validity, credibility
Uses multiple
* Data sources (people, places, times)
* Data collection methods
* Evaluators
Potential Drawbacks
* Time-consuming data analysis
Usability Testing
Goals & questions focus on how well users perform tasks
with the product
Comparison of products or prototypes is common
Focus is on time to complete task, number & type of
errors
Data collected by video & interaction logging
Testing is central
Satisfaction questionnaires & interviews provide data
about users’ opinions
Experiment
Predict the relationship between
two or more variables
Independent variable is
manipulated by the researcher
Dependent variables are what you
are measuring
Validated statistically & replicable
Experiment Design (The two things)
Between-Subjects & Within-Subjects
Different Participants / Between-Subjects Design
Each participant is tested
on one condition only
Pros
* No ordering or training effects
Cons
* Need more participants,
* Chance of variations between subjects
Same Participant / Within-Subjects Design
Each participant is tested
on all conditions
Pros
* Need fewer participants
* Can see variation in performance across conditions per
participant
* Eliminates variation of participants between conditions
Cons
* Ordering effects
* Training effects
Developing a Study Protocol.What is needed/needs to be set up
Type of study
Evaluation measures, conditions,
instrumentation
Tasks, scenarios
Scripts and instructions
Questionnaires, Interview scripts
Hardware and software
(Study Protocol) Tasks
Be specific and realistic
Use scenarios if possible
Ask user to complete tasks
(Study Protocol) Instructions and Interaction
Instructions matter,
can easily bias user behavior
Users help evaluate/test thesystem
You are grateful for
any feedback – good
or bad!
(Study Protocol)Instrumentation for Recording User Behavior
Screen recording
(e.g., Camtasia)
Noldus Observer,
Morae Observer
Eye-tracking,
physiological measures
Video
(Study Protocol)Instrumentation for Prototypes
AS in what to record
Log user actions, timing info
Test for accuracy and format of
logs!
Include testing-specific prompts
and messages
(Study Protocol)Participants
Representative users
How many? 5-10, 20+ for significant experiment
Remuneration
* What is reasonable?
* Do not want to influence behavior
Ethical Treatment of Participants
An evaluation can be a
distressing experience
Participants should always
be treated with respect
Before the Session (userstudy)
Don’t waste user’s time
Make users comfortable
Maintain privacy
Inform user
Users must volunteer
University Research Ethics Board
Any research involving humans
must be cleared
Document the study protocol
Document the purpose of study
Submitted to the Research Ethics
Board
How Do We Decide on a Type of Study?
Work backwards from your goals to figure out what you need to do:
What are your overall goals at
this point?
What are the questions you want
to answer?
What kinds of data do you need
to collect?
What type of study fits these
criteria?
- What are the Study Goals? (Choosing a Type of Study)
What are your overall goals at this point in the process?
May be an iterative
process
May have goals at
different levels
- What are the Evaluation Questions? (Choosing a Type of Study)
What are your overall goals at this point in the process?
Current study goals
What kinds of
questions address
these goals?
- What Kinds of Data Do You Need? (Choosing a Type of Study)
Qualitative, Quantitative, Performance, all pertaining to specific needs for the type of study. As in what questions do you need to ask to get the data
- What Type of Study Fits these Criteria?
(Choosing a Type of Study)
What are the types that fit? Field, Lab, Web, Crowd Source
Now What?(Choosing a Type of Study)
On what do you want to focus?
Make a specific plan for the
user study
* Type of study (considering time, resources, expertise)
* Protocol
* Instruments
* Data analysis
Realism?
* Will results apply in real world?
Generalizability?
* Will results apply to other situations?
