Week 4 User Study

Question 1

Why Do a User Study?

Answer 1

Best way to learn about your design

Get feedback from users

Learn about things you had not
considered

Understand users, and what they
want/need from your design

Question 2

Types of Data

Answer 2

Quantitative and Qualitative

Question 3

What is Quantitative Data?

Answer 3

Expressed as numbers (size,
magnitude, amount)

Measurable, statistics

Can test specific
characteristics or
hypotheses

Question 4

What is Qualitative Data?

Answer 4

Detailed descriptions,
subjective, “rich”

Represents themes, patterns,
stories

Can be observed but not
measured

Open-ended enquiry

Question 5

Comparison of Qualitative VS Quantitative Examples

Answer 5

Quantitative
* Number of clicks
* Performance scores
* Task durations in seconds
* Likert scale responses
* Number of successful trials
* Number of computers you own

Qualitative
* Explanation of why someone chose an item
* Description of a process
* Story describing someone’s experiences with
technology
* Someone’s understanding of how a system
works
* Area of expertise
* Favourite operating system

Question 6

Subjective data

Answer 6

an individual’s opinion, preference, experience, or judgment; not from some external measure.

Quantitative: The user rates this system
8/10 for ease of use
Qualitative: The user thinks computers are
too expensive

Question 7

Objective data

Answer 7

‘external to the mind’ and concern facts and measurements.

Quantitative: The user took 45 seconds to
complete the task
Qualitative: The user owns an Apple phone

Question 8

Data Gathering How to?

Answer 8

Ask the user: Questionnaire, Interview
Observe interaction: Direct observation, Think aloud, Co-discovery learning
Measure performance: Time, Physiological data, #errors

Question 9

Questionnaire: Design Considerations?

Answer 9

Responses can be influenced
by order of questions

Provide clear instructions

Balance between whitespace
and being compact

Wording is very important (Negative, positive, neutral)

Question 10

Questionnaire Types:

Answer 10

Multiple Choice, Likert Scale, Semantic, Open-Ended

Question 11

“Conversations with a Purpose” - Pros and Cons

Answer 11

Pros:
excellent for pursuing specific issues,

address specific questions of interest,

more flexible than questionnaires,

Cons:
accounts are subjective,
prone to rationalization,
time consuming to conduct and
to analyze,
participants may have difficulty,
evaluator can bias the interview,

Question 12

Interview Types

Answer 12

Structured
Unstructured
Semi-structured

Question 13

Unstructured

Answer 13

More like a conversation, can often go into depth

Question 14

Semi-structured

Answer 14

Some pre-determined directions but
flexible as situation evolves

Question 15

Structured

Answer 15

Pre-determined set of questions
* Aka, a standardized interview

Question 16

Focus Groups (Group Interviews) and Pros and Cons

Answer 16

2 – 10 people interviewed
at one time
A skilled moderator is critical
Usually recorded

+ can accommodate diverse
and sensitive issues
+/- opinions developed within
a social context
+ good way to identify
“proto-users”
- some interviewees may
dominate

Question 17

Direct Observation

Answer 17

User is given the task
* and evaluator just watches
the user without interruption

Problem
* no insight into the user’s
decision process, opinion…

Question 18

Think-Aloud

Answer 18

Most widely used method in
industry

User are asked to say what
they are thinking/doing

Gives insight into what the
user is thinking

Question 19

Think-Aloud Problems

Answer 19

Awkward for the user

“Thinking” about it may alter
the way people perform their
task

Hard to talk when
concentrating on task

Question 20

Co-discovering Learning

Answer 20

Normal conversation
between the two
users

Removes
awkwardness

Provides insights

Question 21

Recording Observations Techniques

Answer 21

Paper and pencil (or laptop)
Audio recording
Video recording

Question 22

Recording Observations - Pros

Answer 22

Possible to do in a real-life
context or in a lab

Rich data

See what’s happening for
yourself

Question 23

Recording Observations - Cons

Answer 23

Observer’s presence can
disrupt work

Can be difficult to analyze or
reproduce

Potentially time-consuming

Question 24

Performance Data

Answer 24

Timestamps, timing
User clicks, selections, data
entries
Scores
Task outcomes
Data capture

Question 25

Physiological Data

Answer 25

Eye-tracking
Heart rate, breathing rate
ECG (cardio), EEG (brain)
Skin conductance
Motion
Requires specialized equipment

Question 26

Types of Study

Answer 26

Laboratory
Field
Web
Extra: Crowd-Sourced Study

Question 27

Lab-Based/Controlled Study

Answer 27

Characteristics
* Controlled environment
* Direct observation
Used for
* Measuring performance & improvements
* Discovering usability problems
* Gaining confidence of design before riskier testing
Potential Drawbacks
* Realism can be difficult to achieve
* Novelty effects (hey, this is cool!)

Question 28

Field Study

Answer 28

Characteristics
* Deployed in regular setting
* Maybe more ecologically valid
* More expensive and time-consuming
Used for
* Evaluate real-world usage
* Gather long-term data
Potential Drawbacks
* Riskier for security, privacy, reliability perspectives
* Influenced by outside factors, unsupervised
* Data much more “messy

Question 29

Web-Based/Remote Study

Answer 29

Characteristics
* More natural setting
* Uncontrolled (realistic) environment
* Can prompt to complete tasks, screenshare, videoconference

Used for
* More convenient than lab studies
* Middle ground between lab and field studies

Potential Drawbacks
* More difficult to observe
* Technical challenges

Question 30

Crowd-Sourced Study

Answer 30

Characteristics
* Fast and cheap data collection through online crowdsourcing
Used for
* Quickly gathering large amounts of data
* Studies that can be broken into small tasks
Potential Drawbacks
* Data quality/reliability
* Need fully automated study protocol

Question 31

Triangulation

Answer 31

Explores the same topic in multiple ways
* to gather more holistic
understanding, increase
validity, credibility
Uses multiple
* Data sources (people, places, times)
* Data collection methods
* Evaluators
Potential Drawbacks
* Time-consuming data analysis

Question 32

Usability Testing

Answer 32

Goals & questions focus on how well users perform tasks
with the product

Comparison of products or prototypes is common
Focus is on time to complete task, number & type of
errors

Data collected by video & interaction logging
Testing is central

Satisfaction questionnaires & interviews provide data
about users’ opinions

Question 33

Experiment

Answer 33

Predict the relationship between
two or more variables

Independent variable is
manipulated by the researcher

Dependent variables are what you
are measuring

Validated statistically & replicable

Question 34

Experiment Design (The two things)

Answer 34

Between-Subjects & Within-Subjects

Question 35

Different Participants / Between-Subjects Design

Answer 35

Each participant is tested
on one condition only

Pros
* No ordering or training effects
Cons
* Need more participants,
* Chance of variations between subjects

Question 36

Same Participant / Within-Subjects Design

Answer 36

Each participant is tested
on all conditions

Pros
* Need fewer participants
* Can see variation in performance across conditions per
participant
* Eliminates variation of participants between conditions

Cons
* Ordering effects
* Training effects

Question 37

Developing a Study Protocol.What is needed/needs to be set up

Answer 37

Type of study
Evaluation measures, conditions,
instrumentation
Tasks, scenarios
Scripts and instructions
Questionnaires, Interview scripts
Hardware and software

Question 38

(Study Protocol) Tasks

Answer 38

Be specific and realistic
Use scenarios if possible
Ask user to complete tasks

Question 39

(Study Protocol) Instructions and Interaction

Answer 39

Instructions matter,
can easily bias user behavior

Users help evaluate/test thesystem

You are grateful for
any feedback – good
or bad!

Question 40

(Study Protocol)Instrumentation for Recording User Behavior

Answer 40

Screen recording
(e.g., Camtasia)
Noldus Observer,
Morae Observer
Eye-tracking,
physiological measures
Video

Question 41

(Study Protocol)Instrumentation for Prototypes

AS in what to record

Answer 41

Log user actions, timing info

Test for accuracy and format of
logs!

Include testing-specific prompts
and messages

Question 42

(Study Protocol)Participants

Answer 42

Representative users
How many? 5-10, 20+ for significant experiment
Remuneration
* What is reasonable?
* Do not want to influence behavior

Question 43

Ethical Treatment of Participants

Answer 43

An evaluation can be a
distressing experience

Participants should always
be treated with respect

Question 44

Before the Session (userstudy)

Answer 44

Don’t waste user’s time
Make users comfortable
Maintain privacy
Inform user
Users must volunteer

Question 45

University Research Ethics Board

Answer 45

Any research involving humans
must be cleared

Document the study protocol
Document the purpose of study
Submitted to the Research Ethics
Board

Question 46

How Do We Decide on a Type of Study?

Answer 46

Work backwards from your goals to figure out what you need to do:

What are your overall goals at
this point?
What are the questions you want
to answer?
What kinds of data do you need
to collect?
What type of study fits these
criteria?

Question 47

1. What are the Study Goals? (Choosing a Type of Study)

Answer 47

What are your overall goals at this point in the process?

May be an iterative
process
May have goals at
different levels

Question 48

2. What are the Evaluation Questions? (Choosing a Type of Study)

Answer 48

What are your overall goals at this point in the process?

Current study goals
What kinds of
questions address
these goals?

Question 49

3. What Kinds of Data Do You Need? (Choosing a Type of Study)

Answer 49

Qualitative, Quantitative, Performance, all pertaining to specific needs for the type of study. As in what questions do you need to ask to get the data

Question 50

4. What Type of Study Fits these Criteria?
   (Choosing a Type of Study)

Answer 50

What are the types that fit? Field, Lab, Web, Crowd Source

Question 51

Now What?(Choosing a Type of Study)

Answer 51

On what do you want to focus?
Make a specific plan for the
user study
* Type of study (considering time, resources, expertise)
* Protocol
* Instruments
* Data analysis
Realism?
* Will results apply in real world?
Generalizability?
* Will results apply to other situations?