Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

SCC 202 HCI > User Feedback > Flashcards

User Feedback Flashcards

1
Q

Qualitative vs Quantitative

A

  • Qualities
  • Studying the dynamic and negotiable
  • Open-ended
  • Nuances

Analysis: Thematic patterns
-> “Findings”

  • Numerical quantities
  • Studying the fixed and measurable
  • Categorical

Analysis: Statistics
-> “Results”

Qualitative != Assesing quality

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Qualitative Methods

A
  • Are helpful for understanding the user’s perspective
  • Can be used to evaluate why parts of a design (do not) work
  • But are often used for understanding the use domain before creating a system
    • Domain = The users’ area(s) of expertise (e.g. firefighting, teaching etc.)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What can be evaluated qualitatively?

A
  • Experience
  • Usability issues
  • Contextual fit
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Evaluating Experience

A
  • Hedonic experience
  • How do users experience the system?
    Pleasant/stressful/helpful/chaotic/…
  • Do people want to use the system (for the things it was designed for)?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Evaluating Usability

A
  • Does the system help people or do they experience a need to do workarounds?
    • Are the included features appropriate and sufficient?
    • Does the structure of information make sense to users?
  • What problems/issues do users encounter when using the system?
  • What about the system works well for users?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Evaluating Contextual Fit

A
  • How well does the system work in the intended use situations?
  • How well does the system fit into the domain practice?
    • Does the system disrupt tasks or routines?
  • How does the system indirectly impact people that users work with, or who are otherwise impacted by users’ work?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Qualitative Evaluation with Users

A

Qualitative methods can center around…
- Users reporting on their experience
- You observing the user
- … or a combination

We will get into three examples of methods:
1. Interviews
2. The Think Aloud Protocol
3. Diary studies

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

User Feedback

A
  • Introduction to Qualitative Methods
  • Types of qualitative methods
    • Interviews
    • Focus Groups
    • Think Aloud
    • Focus Shift Analysis
    • Diary Studies
  • Metrics & Measures
    • Self/User Reported
    • Questionnaires
    • Rating Scales
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

Interviews #1

A
  • A purposefully one-sided conversation
  • An interviewer has an agenda and directs the conversation
  • Three types:
    • Unstructured: Open-ended (“Tell me about…”), good for exploring topics and reactions to new design ideas
    • Structured: Similar to questionnaires (“Which of the following…”), good for getting feedback about particular aspect of a design
    • Semi-structured: A mix of open-ended and closed, good for in-depth coverage of the same topics with each participant
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Interviews #2

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Focus Groups #1

A
  • Interviews are often one-on-one
  • Focus group: A group of users are interviewed together
  • Each group usually consists of similar kinds of users
    • E.g. for BB: students in one group, administration in another
  • Good for collecting multiple viewpoints
  • Common form of interview in product design
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Focus Groups #2

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Interviews - Data

A
  • Created during the interview:
    • Audio/video recording
    • Notes
  • After the interview:
    • Reflection notes
    • Transcripts of the audio
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Interviews - Analysis

A

Coding: Grouping parts of the data into themes

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Think Aloud #1

A

The participant carries out a pre-defined task using the system
- During the task, the participant explains what they are thinking and doing, e.g.
- “I’m pressing the search field and typing in…”
- I can’t seem to find the menu. Maybe up here…” moves cursor

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Think Aloud #2

A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Think Aloud - Data

A

Created during the study:
- Video recording
- …and/or combined audio and screen recording
- Notes

After the study:
- Transcripts of audio, aligned with video/screen recoding
- Potentially, only critical incidents are transcribed

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Think Aloud - Analysis

A
  • Coding (e.g. clustering errors into types)
  • Looking for critical incidents
    • E.g., points of error/confusion/silence
    • In some cases it may be particularly interesting if a user makes an error without noticing
  • Codes or critical incidents may be distilled into a list of issues.
  • Results are often similar to what can be discovered with an expert walkthrough…
    But can include surprising insights related to the user’s domain knowledge
19
Q

Think Aloud - Focus Shift Analysis

A
  • Analyzing what the user is focusing on can help you identify breakdowns
    • Breakdown: When the user’s actions are directed at the system rather than at the task
    • Example: Needing to figure out which paper tray to tell the printer to use, rather than specifying that you want the print to be A3
  • In a focus shift analysis, the transcript is mapped to the objects the participant’s focus was directed at
    • Objects= files, windows, UI elements, hardware,…
    • Which objects are relevant depends on what you are evaluating
20
Q

Diary Studies #1

A
  • Participants fill out information during their day-to-day activities over a period of time.
    • Feedback activities: Fillin in a form
    • Elicitaion activities: Capturing media, such as photos
  • Participant can be reminded to participate with automated e-mail or text messages.
  • The diaries can be used to as outset for interviews
21
Q

Diary Studies #2

A
22
Q

Diary Studies - Data

A

Data from diary studies varies a lot:
- Text
- Photos
- Audio
- Annotations on images, maps, etc.
- …

23
Q

Data Studies - Analysis

A
  • Open/closed coding (like interviews)
  • Identifying elements of particular interest to follow up on
  • …
24
Q

Pitfalls #1

A

Conversations are social situations -> People will behave like in any other social situation

Participants in a user study
- … Want to be polite / feel like you know more about the system than them

Ways to mitigate:
- Explain the participant’s role of domain expert (you want to learn from them, not the other way around)
- Let someone other than the person who made the system conduct the study (and let the participant know this)
- Use mock-ups, make it clear that system is not finished

25
Q

Pitfalls #2

A
26
Q

Self-Reported Metrics

A
  • Data reported by the users themselves is important as it provides information on their satisfaction with a system, and perception of the interaction with it
  • Self-reported data can be
    • Qualitative: for example by asking users open-ended questions about their experience
    • Quantitative: By using questionnaires as instrument for collecting quantitative data from users
  • The general format for self-reported metrics is to give users a question (or statement) and ask them to select an answer on a scale
  • Self-reported metrics are rarely useful just by themselves
    • Combined with performance data, such as task success and times
    • Combined with qualitative feedback, which can provide explanations for ratings
27
Q

User-reported data

A
  • Data reported by the users themselves is important as it provides information on their satisfaction with a system, and perception of the interaction with it
  • Subjective feedback by users
  • Self-reported data collected with questionnaires
    • Asking users a pre-defined set of questions
    • Similar to structured interview, but on paper or on a computer
  • Use of rating scales for quantitative analysis
  • Also qualitative data that can be coded for quantitative analysis
28
Q

Questioaires #1

A

-Questionnaires are a method for data collection from study participants
- In general, the term refers to collection of data by giving users a set of pre-defined questions similar to a structured interview but on paper or on a computer

  • Useful for gathering data from a larger number of people
  • Can only gather data you know about (unlike observation and interviews that can uncover data)
29
Q

Questionnaires #2

A

In quantitative research, the term refers to instruments that measure specific phenomena (perceptions, attitudes, …) by asking people questions that are carefully designed to meet three criteria
- Validity: the question measures what is intended to be measured
- Reliability: users will consistently answer the question in the same way
- Sensitivity: the questions detects meaningful differences

30
Q

Questionnaires #4

A

Igroup Presence Questionnaire (IPQ)
- Develop for virtual reality experiences
- Measuring the user’s sense of presence in the virtual environment
- 14 items on 3 factors:
- Spatial presence: Sense of being physically present in the VE
- Involvement: Measuring attention devoted to the VE
- Experienced Realism: measuring subjective experience of realism

31
Q

Open & Closed Questions

A
  • Open-ended questions (“can you suggest any improvements”)
    • Good for general subjective information
    • Difficult to analyse
  • Closed questions - single or multiple choice
    • Restrict responses by supplying alternatives
    • Easy to analyse
    • Watch out for ‘hard-to-interpret’ responses
    • Alternative responses should be
      • Mutually exclusive
      • Exhaustive
32
Q

Other Data Collection

A
  • Collection of demographic information on users, and any information about users that is relevant for a study and analysis of the results
    • Age, gender
    • e.g., prior experience with the type of interface or application
    • e.g., handedness
  • Collection of qualitative feedback
    • For example using SEQ (Single Ease Question) combined with asking users to give a reason for their rating
    • Post-test feedback / comments
      • e.g., “Can you suggest any improvements”
33
Q

Questionnaire Guidelines

A
  • Always collect demographic data: age and gender
  • Concise: keep questions simple and as short as possible.
  • Relevance: each question must be relevant to your study goal.
  • Precision: don’t use vague terms.
  • Avoid ‘loaded’ or ‘leading’ questions that hint at the answer you want to hear
  • Avoid ‘and’ questions -> split.
  • Avoid negative questions (and double-negatives!)
  • Avoid jargon and abbreviations
34
Q

Rating Scales

A
  • The most common rating scale are Likert Scales, composed of statements to which respondents rate their agreement.
  • Developed by Rensis Likert, 1932, as general psychometric scale.
  • A Likert item can be a positive (“The labels used in the interface are clear”) or a negative statement (“ I found the navigation options confusing”)
  • Respondents specify level of agreement with a statement on a symmetrical agree-disagree scale.
  • The original Likert scale has 5 points, each with a response anchor:
    1 - Strongly disagree; 2 - Disagree; 3 - Neither agree nor disagree; 4 - Agree; 5 - Strongly agree
  • The range captures the intensity of the subjects’ feeling for a given item
35
Q

Likert Scales

A

Likert Scales
- User judge a specific statement on a numeric scale
- Usually agreement or disagreement with a statement
- Provides quantitative data
- Typically 5-point or 7-point scales

Also other types of scales, e.g., semantic differential

36
Q

Rating Scales # 1

A
37
Q

Rating Scales #2

A

Statements for Likert scales need to be worded carefully, using unmodified adjectives.
- Modifiers such as “very”, “extremely”, “absolutely” bias the response
- e.g., “the UI is extremely easy to use” makes strong agreement less likely than “the UI is easy to use”

38
Q

Post-task / Post-test rating

A
    • In usability evaluation, questionnaires and ratings are categorized as as post-task versus post-test
  • Post-task ratings are completed immediately after finishing a task, to capture impression of the tasks, and are often just a single task-difficulty question: e.g. 7-point “Single Ease Question” (SEQ)
  • Post-test questionnaires are administered at the end of a session, after completion of all tasks with an interface, to capture how users perceive the usability of the interface as a whole
  • Post-task and post-test rantings can be complementary
39
Q

System Usability Scale (SUS)

A
  • Widely used scale
  • Developed by John Brooke, 1986
  • 10 statements
    • 5 Worded positively, 5 negatively
  • Responses converted to score 0…4
    • Added up
    • Multiplied by 2.5
    • Total out of 100
40
Q

Usefulness, Satisfaction and Ease-of-Use Questionnaire (USE)

A
41
Q

NASA-TLX

A
  • NASA-TLX (Task Load Index) is a post-task
    questionnaire for complex interfaces
  • Developed by NASA for measuring the perceived workload of highly technical tasks of aerospace crew
  • 6 Questions on an un-labelled 21-point scale, from Very Low to Very High
  • Complex to score
  • In HCI it is common to just adopt the mental demand and physical demand questions into custom questionnaires
42
Q

PSSUQ / CSUQ

A
  • Post Study System Usability Questionnaire (PSSUQ) and Computer System Usability Questionnaire (CSUQ)
  • Like the SUS for post-test rating of any type of interface. Originally PSSUQ, minor changes in CSUQ
  • 16 items, 7-point scale, positively worded
  • Provides overall usability score, but also scores subfactors: System Usefulness; Information Quality; Interface Quality
  • High sensitivity: able to detect differences across a large number of variables (different user groups, types of systems used, years of experience, etc.)
  • Effective at smaller sample sizes (because of higher sensitivity)
  • Strong correlation with SUS
43
Q

User Feedback - Key Points

A
  • Some aspects of people‘s interaction with technology cannot be measured
  • Qualitative studies are useful for obtaining rich data from a smaller number of subjects
  • Qualitative methods can be used for evaluation
    • … but are often also used to understand
      the domain and the people who will be
      using a system before design and
      construction of the system begins
  • Qualitative approaches are not by definition better than quantitative approaches (or vice-versa) – What is appropriate depends on what you want to find out…
  • Qualitative and quantitative data can complement each other

SCC 202 HCI (20 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions