Lecture 13 (Evaluation Techniques) Flashcards

1
Q

Why involve humans?

A

Computer applications are used by humans so it is important to evaluate them

  • Computer graphics are inherently human-centered
  • Presence of images, animations, and interaction that will influence behavior
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

What is Evaluation?

A
  • Tests usability and functionality of a system
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Where does Evaluation occur?

A
  • In the lab, field, and/or in collaboration with users
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

When should Evaluation take place?

A
  • Evaluation should be considered at all stages of the design lifecycle
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

What are the goals of evaluation?

A
  • Test hypotheses on users
  • Assess extent of system functionality
  • Assess effect of different features on the user
  • Identify specific problems and attempt to solve them
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

What are the two Evaluation Styles?

A
  • Laboratory studies

- Field studies

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

What are the pros and cons of Laboratory studies?

A

Pros:

  • Specialist equipment available (e.g. eye-tracking software)
  • Uninterrupted/controlled environment

Cons:

  • Lack of context of real-life use (not as realistic)
  • Difficult to observe several users cooperating
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

When is it appropriate to conduct Lab studies for evaluation?

A

If system location is dangerous or impractical for constrained single user system to allow controlled manipulation

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

What are the pros and cons of Field studies?

A

Pros:

  • Natural environment
  • Context retained (though observation may alter it)
  • Longitudinal studies possible

Disadvantages:

  • Distractions
  • Noise
  • Generally less controlled
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

When is it appropriate to conduct Field studies for evaluation?

A

Where context is crucial for longitudinal studies

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

What are the three Design Evaluation Techniques?

A
  • Cognitive walkthrough
  • Heuristic evaluation
  • Review-based Evaluation
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

What is Cognitive Walkthrough?

A

A design evaluation technique

  • Evaluates design on how well it supports users in learning the task
  • Usually performed by experts in cognitive psychology
  • The expert “walks through” the design to identify potential problems using psychological principles
  • Forms used to guide analysis
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

What does a Cognitive Walkthrough consider?

A
  • What impact will interaction have on the user?
  • What cognitive processes are required?
  • What learning problems may occur?
  • Does the design lead the user to generate the correct goals?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

What is Heuristic Evaluation?

A

A design evaluation technique

  • Use this when actual user testing isn’t able to be conducted
  • Used to debug the design
  • Usability criteria (heuristics) are identified
  • Design examined by experts to see if these are violated
  • Needs high level of expertise
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

What is Review-based Evaluation?

A

A design evaluation technique

  • Uses literature to support or refute parts of the design
  • Use this when actual user testing isn’t able to be conducted
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

What are the three Observational Methods?

A
  • Think Aloud
  • Cooperative Evaluation
  • Post-task Walkthroughs
17
Q

What is Think Aloud?

A

Observational method

  • User observed performing a task
  • User is asked to describe what they are doing and why, what they are thinking of etc.
18
Q

What are the pros and cons of Think Aloud method?

A

Pros:
- Simplicity

Cons:

  • Subjective (users might have varying levels of experience)
  • Selective
  • Acts of describing may alter task performance (user might not be good at multitasking)
19
Q

What is Cooperative Evaluation?

A

Observational method

  • Variation on Think Aloud (user collaborates with evaluator)
  • Both user and evaluator can ask each other questions throughout
    • e.g. Why did you make this mistake? What went wrong?
20
Q

What are the pros and cons of Cooperative Evaluation?

A

Pros:

  • Less constrained and easier to use
  • User is encouraged to criticise system
  • Clarification is possible

Cons:
- More interruption

21
Q

What is Post-task Walkthroughs?

A

Observational method

  • User reacts on action after the event
  • Used to “fill in” intention
  • Necessary in cases where Think Aloud is not possible
22
Q

What are the pros and cons of Post-task Walkthroughs?

A

Pros:

  • Analyst has time to focus on relevant incidents
  • Avoid excessive interruptions of task

Cons:

  • Lack of freshness (users may forget things)
  • May be post-hoc interpretations of events (users might fake answers if they forget things)
23
Q

Methods of Recording Data in evaluation?

A
  • Paper and Pencil (cheap)
  • Audio (good for think aloud)
  • Video / Screen capture (accurate and realistic, needs special equipment, obtrusive)
  • Computer logging (automatic and unobtrustive, requires large amounts of data to analyze, also difficult to analyze)
  • User notebooks (subjective, useful insights, good for longitudinal studies)
  • Mixed use and practice
24
Q

What are some Physiological Methods for evaluation?

A
  • Eye Tracking

- Physiological Measurements

25
What is Physiological Measurements in evaluation? How may they be helpful?
- Emotional response linked to physical changes - These may help determine a user's reaction to an interface e. g Heart activity, activity in brain, etc.
26
What are some Query Techniques for evaluation?
- Interviews | - Questionnaires
27
What are Interviews? And what are the pros and cons?
Query Technique - Analyst questions users one-on-one with prepared questions Pros: - Cheap - Issues can be further explored - Can identify unanticipated problems Cons: - Very subjective - Time consuming
28
What are Questionnaires? And what are the pros and cons?
Query Technique - Set of fixed questions given to users Pros: - Quick and reaches a large user group - Can be analyzed more rigorously - Can measure something specific - Measurements can be compared Cons: - Less flexible - Can't go back to user and ask why they answered something - Can't identify unanticipated problems - Needs careful design of the questionnaire
29
How to plan out a Questionnaire?
- Decide what you want to measure - Find standardized questionnaire or create your own - Deploy and collect data - Analyse data
30
How to choose an evaluation method?
When in process - Design vs Implementation Style of evaluation - Laboratory vs field How objective - Subjective vs Objective Types of measures - Qualitative vs quantitative Level of information - High level vs Low level Level of interference - Obtrusive vs unobtrusive Resources available - Time, users, equipment, expertise