Lecture 13 (Evaluation Techniques) Flashcards
Why involve humans?
Computer applications are used by humans so it is important to evaluate them
- Computer graphics are inherently human-centered
- Presence of images, animations, and interaction that will influence behavior
What is Evaluation?
- Tests usability and functionality of a system
Where does Evaluation occur?
- In the lab, field, and/or in collaboration with users
When should Evaluation take place?
- Evaluation should be considered at all stages of the design lifecycle
What are the goals of evaluation?
- Test hypotheses on users
- Assess extent of system functionality
- Assess effect of different features on the user
- Identify specific problems and attempt to solve them
What are the two Evaluation Styles?
- Laboratory studies
- Field studies
What are the pros and cons of Laboratory studies?
Pros:
- Specialist equipment available (e.g. eye-tracking software)
- Uninterrupted/controlled environment
Cons:
- Lack of context of real-life use (not as realistic)
- Difficult to observe several users cooperating
When is it appropriate to conduct Lab studies for evaluation?
If system location is dangerous or impractical for constrained single user system to allow controlled manipulation
What are the pros and cons of Field studies?
Pros:
- Natural environment
- Context retained (though observation may alter it)
- Longitudinal studies possible
Disadvantages:
- Distractions
- Noise
- Generally less controlled
When is it appropriate to conduct Field studies for evaluation?
Where context is crucial for longitudinal studies
What are the three Design Evaluation Techniques?
- Cognitive walkthrough
- Heuristic evaluation
- Review-based Evaluation
What is Cognitive Walkthrough?
A design evaluation technique
- Evaluates design on how well it supports users in learning the task
- Usually performed by experts in cognitive psychology
- The expert “walks through” the design to identify potential problems using psychological principles
- Forms used to guide analysis
What does a Cognitive Walkthrough consider?
- What impact will interaction have on the user?
- What cognitive processes are required?
- What learning problems may occur?
- Does the design lead the user to generate the correct goals?
What is Heuristic Evaluation?
A design evaluation technique
- Use this when actual user testing isn’t able to be conducted
- Used to debug the design
- Usability criteria (heuristics) are identified
- Design examined by experts to see if these are violated
- Needs high level of expertise
What is Review-based Evaluation?
A design evaluation technique
- Uses literature to support or refute parts of the design
- Use this when actual user testing isn’t able to be conducted
What are the three Observational Methods?
- Think Aloud
- Cooperative Evaluation
- Post-task Walkthroughs
What is Think Aloud?
Observational method
- User observed performing a task
- User is asked to describe what they are doing and why, what they are thinking of etc.
What are the pros and cons of Think Aloud method?
Pros:
- Simplicity
Cons:
- Subjective (users might have varying levels of experience)
- Selective
- Acts of describing may alter task performance (user might not be good at multitasking)
What is Cooperative Evaluation?
Observational method
- Variation on Think Aloud (user collaborates with evaluator)
- Both user and evaluator can ask each other questions throughout
- e.g. Why did you make this mistake? What went wrong?
What are the pros and cons of Cooperative Evaluation?
Pros:
- Less constrained and easier to use
- User is encouraged to criticise system
- Clarification is possible
Cons:
- More interruption
What is Post-task Walkthroughs?
Observational method
- User reacts on action after the event
- Used to “fill in” intention
- Necessary in cases where Think Aloud is not possible
What are the pros and cons of Post-task Walkthroughs?
Pros:
- Analyst has time to focus on relevant incidents
- Avoid excessive interruptions of task
Cons:
- Lack of freshness (users may forget things)
- May be post-hoc interpretations of events (users might fake answers if they forget things)
Methods of Recording Data in evaluation?
- Paper and Pencil (cheap)
- Audio (good for think aloud)
- Video / Screen capture (accurate and realistic, needs special equipment, obtrusive)
- Computer logging (automatic and unobtrustive, requires large amounts of data to analyze, also difficult to analyze)
- User notebooks (subjective, useful insights, good for longitudinal studies)
- Mixed use and practice
What are some Physiological Methods for evaluation?
- Eye Tracking
- Physiological Measurements
What is Physiological Measurements in evaluation? How may they be helpful?
- Emotional response linked to physical changes
- These may help determine a user’s reaction to an interface
e. g Heart activity, activity in brain, etc.
What are some Query Techniques for evaluation?
- Interviews
- Questionnaires
What are Interviews? And what are the pros and cons?
Query Technique
- Analyst questions users one-on-one with prepared questions
Pros:
- Cheap
- Issues can be further explored
- Can identify unanticipated problems
Cons:
- Very subjective
- Time consuming
What are Questionnaires? And what are the pros and cons?
Query Technique
- Set of fixed questions given to users
Pros:
- Quick and reaches a large user group
- Can be analyzed more rigorously
- Can measure something specific
- Measurements can be compared
Cons:
- Less flexible
- Can’t go back to user and ask why they answered something
- Can’t identify unanticipated problems
- Needs careful design of the questionnaire
How to plan out a Questionnaire?
- Decide what you want to measure
- Find standardized questionnaire or create your own
- Deploy and collect data
- Analyse data
How to choose an evaluation method?
When in process
- Design vs Implementation
Style of evaluation
- Laboratory vs field
How objective
- Subjective vs Objective
Types of measures
- Qualitative vs quantitative
Level of information
- High level vs Low level
Level of interference
- Obtrusive vs unobtrusive
Resources available
- Time, users, equipment, expertise
