Usability & UX evaluation slides information Flashcards
Define Usability
“The user can do what he or she wants to do, that what he or she expects, to be able to do it, without hindrance, hesitation or questions.
What are the Usbality measures
Efficiency (Time on task)
Effectiveness (Task completion)
Satisfaction (No discomfort, positive attitude)
Name aspects of Bailey’s human performance model
- Somebody (human) -> Detach
- Somewhere (context) => there are more places than you know
- Something *activity) -> Does not exist in a vacuum
All these factors are in motion
Define User Experience
“A person’s perception / responses to the use, or anticipated use of a product, system or service”
Alternate defenition User Experience
The consequences of a user’s internal state (predispositions, expectations, motivation, mood) a systems characteristics (complexity, purpose,usability, functionality, etc.) And the context within which the interaction occurs
What is the difference between UX & Usability?
- Usability is a part of UX
- User Experience is an extension of the satisfaction part of Usability
The UX in time model
When:
- Before Usage, Anticipated UX, Imagining an Experience
- During Usage, Momentary UX, Experiencing
- After Usage, Episodix UX, Reflecting on an experience
- Over time, Cumulative UX, Recollection of periods of use
Aspects you need to watch out for with Usefulness
- Designing the right thing vs. Designing things right (optimizing Usability)
- Danger of early usability testing
- Apply the right method for the right phase (Concept drawings do not provide deep insights, Task-centered evaluations focus on the negative)
Why evaluate?
- Informed design decisions
- Identify design problems and fix them
- Money (cost reduction, profit gain)
- Create feeling of involvement
- Generation of scientific research/ intermediate level knowledge
Factors to inform your research design
- Timing
- Goal (Benchmark, high-level issue)
- Desired output (Qualitative, Quantitative)
- Subject (Prototype, product)
- Available budget
- Level of interaction with participants
- Task-Driven or Natural form (free-form)
How to use evaluations to obtain input for user research
- Evaluations of existing (competitor) products can be part of research
- Other than unearthing existing issues, evaluations can give input for future directions
- Shared methods
Focus of your investigation is either on:
- Behavior
- Attitude
The product feature your investigating is either
- Interface
- Information architecture
Choices for research
Lab vs field testing
Remote evaluations - (a)synchronous
Incidental - Long term
Informal evaluations are..
- Fast & Cheap
- Less planning
- Loose Recruitement (even colleagues)
- Less structured
- Less formal output
Setting up a study holds the following considerations
- Planning (which approach, methods at what time), triangulation)
- Participants (Sampling, depends on question, sample size)
- Ethics (Consent, Right to witdraw, Information is confidential)
Intermediate level knowledge
Represents interesting and important knowledge in design research
Examples:
- Design methods and tools
- Design guidelines
- Usability heuristics
- Patterns
- Strong concepts
- Experiential qualities
- Criticism
- Annotated portfolios
What are design research directions?
- A holistic vision for UX evaluation (Hedonic & Pragmatic)
- Inspection methods for Hedonic attributes (Psychophysiology)
- Core skills needed for evaluation ( What do we all need as interdisciplinary field
- Learn from evaluation in practice)
Triangulation
= An approach to data collection and analyses that uses multiple methods, measures or approaches to look for convergence on product requirements or problem areas
Benefits of triangulation
- More in-depth understanding
- More richness, varied set of data
- More convincing, persuasive recommendations
- Reduce ‘innaproppriate certainty’ that not much is wrong with a design
- Prioritizing requirements
What can you triangulate?
Any source of data (usability tests, interview, think aloud)
Triangulation methods
Between methods:
- Research methods (implicit & explicit)
- Qualitative & Quantitative
Within methods:
- Observer
- User groups
- Geographic
Sequential triangulation
Sequential (between studies)
- Dairy studies, followed by interview
- User research, followed by evaluation of a prototype
Concurrent triangulation
Concurrent ( In same study more methods)
- Eye tracking and a survey
What are the performance metrics
- Time on task
- Task success
- Errors
- Click paths
What are the advantages of using performance metrics?
- Statistically valid
- Clear to explain
- Easy to compare
What are Challenges with performance metrics
- Beware of number fetishism
- Cause & effect implications
- Covariates
When is a task not succesful?
- Person gives up
- Time limit reached
- Researcher intervenes
- Incorrect answer
Levels of succes
Usability perspective:
- Complete succes: with or without assistanc
- Partial success: with or without assistance
- Failure: User thought it was complete but it wasn’t, user gave up
UX perspective:
1 = no problem (user completed task without difficulty)
2= Minor problem (completed task, with slight detour or small mistakes)
3= Major problem (completed task, but with major problems)
4 = Failure / gave up
Different types of task succes:
a. Binary succes ( 1= succesful, 0= failure)
b. Levels of succes (
c. Task failure
Types of task failure
- Giving up
- Moderator ‘calls’ it
- Too long
- Wrong
What can time on task mean
Short time on task: Not hindered by issues
Long time on task: engagement
Ask how the user perceived the passage of time
When to use ‘error’ metric?
- Significant efficiency loss
- Significant economic loss
- Error would result in task failure
How to fix slips? (Typo, accidental misclick)
- Reduce number of actions required
- Work on error handling and prevention
How to fix mistakes (clicking something unclickable, entering wrong data, performing wrong action)
- Provide better support
- Hints
- Fix User Interface problems
- Fix Usabillity study scenario / tasks
What do we measure with the metric ‘click path’?
- Number of actions needed to complete a task
- First click tests
- ( may be relevant to identify ideal path)
What are the challenges of user testing?
- Walk up and use: natural first interaction, you may - need familliarity or expertise
- Standardized tasks: Assign variance to what we want to test
- Scale of the problem: User contexts differ greatly, need often be practical
- Usability trap: Only creating what a usability test can measure (avoid complexity)
What is log data?
Being able to reconstruct (parts of) the interaction of one particular system (while also being able to generalize and find patterns)
What are advantages of log data?
- Relatively non invasive
- Natural product use
- Suitable for measure over time
- Customizable ‘events’
What are forms of logging?
- Primitive form: Keyboard, mouse logging
- Identifying tasks
- Deriving performance measures
Analytics
- A/B testing
- A + B testing
- Heatmaps and scrollmaps (Hotjar)
- Video recording (Hotjar)
Advantages of analytics compared to Usability
- Generally collected remotely and ansynchronous
- Larger sample size (more anonymous)
- Over time (returning users, learnability, active users, adoption)
- Allows you to discover patterns
- Natural product use
Differences between logging and analytics
- Very similar, but then with standardized measures
- Logging can be part of a usability study and an anlytics approach
What are emotional & implicit measures?
- Relatively objective measures (spontaneous, subconscious)
- Decreases the effect of social desirability
Why do you need to handle implicit measures with care?
- Noisy, sensitive to external influences
- Open to interpretation - What not why
Types of implicit measures
- Psychophysiology ( EEG, HR, fMRI)
- Eye tracking
- Facial behavior analysis
- Pose, motion tracking
What is eye tracking?
Focus on patterns of saliency of particular features,
doesn’t help to determine usability problems, but helps to determine what lead to that problem
How does eye tracking work?
The light reflected from the cornea and the center of the pupil are used to inform the eye tracker about the movement and direction of the eye
What is facial behavior analysis ?
The analysis of strong emotional reactions
Mostly primite emotions (hapiness, sad)
Done with facial reader
When to use facial behavior analysis?
- When strong emotions occur
- When the temporal pattern of emotion is important
- When strong emotional reactions matter in the usability of the product (if you want to avoid strong emotions for work for example, or want to trigger for advertisement)
What are challenges of facial behavior analysis?
- Limited to basic emotions and extreme cases
- Unclear what triggered a change in emotion
What are challenges of Pose and Motion trackers?
- Raw data, no basic interpretation
- Pose and motion are complex constructs
The pyramyd of measures
What people: Say/think, do/use, know/feel/dream
Techniques: Interviews, observations, generative sessions
Knowledge: explicit, observable, tacit, latent
Determinants of type of qualitative studies
- Interaction with the user (observation, interview)
- Location of data collection (lab, field)
- Number of participants (individual, group)
- Modality of interaction ( written, direct oral)
- Freedom in interaction
The goals of qualitative studies
- Get an overview of usability issues that are currently present in the product (based on do/use)
- Identify the user’s experiences with the product, and possible improvements
Potential solutions to gather insights into usability
- There is no foolproof way to identify all the usability flaws
- Use mix of methods
- Concentrate on the most important features
- Think about the goals of the evaluation
- Use multiple evaluatios (within-methods triangulation)
- Reflect on your practices
What are qualitative say&think methods
- Individual interview
- Contextual interview
- Laddering
- Focus group
- Think aloud
What are qualitative do & use methods?
- Controlled observations
- Naturalistic observations
- Dairy studies
- Expert review: Holostic approach
- Expert review: Cognitive walkthrough
What is an Individual interview?
A talk one on one with a user, f2f or through phone or video. This interview can discover beliefs, attitudes, desires and reactions towards a concept.
What are advantages of an Individual interview?
- Deep understanding of needs and wants
- Adaption
What are dissadvantages of an Individual interview?
- Costly
- Explicit knowledge only (just say/think, no do/use)
What is an contextual interview?
Watch people’s behavior in their own environment
And ask questions while observing.
This type of interview is done when context is important.
What are the advantages of a contextual interview
- Natural behavior.
- Real life task/environment
- Explicit & observable knowledge
What are the dissadvantages of a contextual interview?
- Costly
- Poor control conditions
What is laddering?
Laddering is a qualitative interview technique to discover meaningull information about a product or service. This is done by looking at Attributes-Consequences-Values. This technique is done by keep asking ‘why’.
What are the advantages of Laddering?
- Deep understanding and getting to high-level goals and underlying values.
What are the dissadvantages of Laddering?
- Can feel tedious/childish
- Not easy to conduct
What is a focus group?
A group of people that haves moderated discussions (N=5-10), often informal and semi-structured. This type of research is done to discover attitudes, beliefs, desires and reactions.
What are the advantages of a focus group?
- Cost effective
- Discover what the user feels, needs, value
What are dissadvantages of a focus group?
- Explicit knowledge only
- Requires an advanced moderator
- Poor method for Usability testing
- Group dynamics
What is good to do during a Focus Group, and what is good not to do?
- It is good to ask open-ended questions
- Balance participation
- Let participants use their own words
- Not good to have a stiff and formal presentation
What is think aloud?
- Users verbalize the whole proces while performing tasks, this technique is used to uncover what they really think.
What are advantages of think aloud?
- Cost effective
- Flexible (both for prototypes and finished products)
- Real time measure of experience
What are dissadvantages of think aloud?
- Can feel unnatural
- Chance of bias
- Individual differences
- Affects other measures (eye tacking)
What are the strategies of thinking aloud?
- Rigid: No interference/neutral reminder (keep talking)
- Speech/clues: Acknowleding contributions, ask for clarification, encouragement
What is co-discovery?
It is the method of think-aloud with two people, more natural to share thoughts
What is a controlled observation?
- Observing users interact with a product or service in a lab environment
- They are asked to reach a specific goal, or do a specific task
- Context for quantitative measures (usability issues)
What is a naturalistic observation
- Observing users interact with a product ‘in the wild’
- This technique is used when the context is impotant
What are the advantages of a controlled observation?
- Control over the condition
- Structured approach
- Quick
What are dissadvantages of a controlled observation?
- Pre-defined tasks only
- Unnatural context/behavior
What are advantages of a naturalistic observation?
- Natural product use
- Ecological validity
- Observe how the product is used in daily life
What are dissadvantages of a naturalistic observation?
- Little control over conditions
- Costly
- Hawthorne effect (show what you think the observer wants to see)
What can you observe?
- Space
- Actor
- Activity
- Object
- Act
- Event
- Time
- Goal
- Feelings
- Interaction (user-product, social)
- Use in context (routine, appropriation)
What is an Expert Review: Holistic evaluation?
- Inspection method: Analysis by an expert
- Comparing against heuristics
- Holistic approach
What is an Expert review: Cognitive walkthrough?
- Inspection method: Soecific tasks, focus on ease of completion
- Particular useful for ‘walk-up and use system
Advantages of Expert review:
- Flexible
- Quick
Disadvantages of Expert review:
- Better to use more than one expert (=costly)
- No real user, using the product
- Often focused on minor usability issues
What questions can a expert aks him/herself during a cognitive walkthrough?
- Will users want to produce whatever effect the action has?
- Will users see the (UI) controls
- Will users know that the control produces the effect they want
- After the action is taken, will users understand the feedback
Two types of Qualitative data coding:
- Manual coding (computer-aided)
- Computerized analysis
What are the different types of Manual coding?
- Data driven coding (bottom up)
- Theoretically driven coding (top down)
- Combination of bottom up and top down coding
What is a sentiment analysis?
- Detect the polarity (positive/negative) of an opinion with a certain degree of confidence
- Uncover themes of sentiment in large scale data (e.g. twitter posts, reviews)
What are the steps of systematic coding?
- Transcription of verbal data
- Immerse yourself in the data, read the transcript several times
- Chunking into single meaningful units
- Close reading and detailed coding
- Analysis and sorting of similar codes into categories
- Recode text
- Finalizie coding system
What are bio signals?
- Brain activity
- Heart rate
- Skin conductance
- Skin temperature
- Muscular activity
Which two constructs are determinants for emotion?
- Arousal: Intensity of emotion
- Valence: Whether this arousal is positive or negative
The history of psychophysiological measures, can you name different steps?
- Behaviorism: Has a strong influence, Classical conditioning
- Stimulus response model: Who says what to whom with what effect
What are assumptions on information processing?
- Humans have innate capacity (inherited tendencies)
- Humans are active information seekers
- Knowledge is stored in the brain (cognitive system requires representation)
- Humans are dynamic systems
- Systems are divisible
What are the kinds of self-report measures?
- Surveys & questionnaire
- Dairy Studies
- UX curve
- Sentence completion
Why would you write your own survey questions?
- Helpful for demographics
- For behavorial and attitudinal metrics
Why would you use a survey/questionnaire?
- Profile your use group
- Assess values, needs, beliefs
- Evaluate design (Usability/UX)
- Can use this anywhere in the process
What is a dissadvantage of surveys/questionnaire?
- Prone to social desirability
- After the fact
What type of questions can be in a survey?
- Open questions (free form, listings)
- Likert scale
- Semantic differential scale
- Frequency scales
- Paired comparison scale
Best to use standardized scales
What is a dairy study?
Tracking everyday prolonged use, follow a specific type of (long-term) experience. Which can be done remote using paper or technologies.
What is a UX curve?
- Letting participants draw a curve to describe their experience about a product changed over time
- It is a retrospective method (memory rather than in-situation experience)
- Different scales for different aspects of UX (attractiveness, ease of use)
- Reveal impactful experiences over time
What types of data can you present?
- Task performance (Often in graphs, task succes, time on task, errors)
- Attitude (self-report, interviews)
- Usability issues (Number of unique issues, frequency of issues per participant
What are the two types of EDA?
CNS: Measures direct brain activity (EEG, fMRI)
PNS: Measures evoked response (heart rate, skin conductance)
What are different types of PNS responses?
- Autonomic (involuntary: Sympatetic, parasympatetic)
- Somatic (voluntary)
- Electrodermal activity ( Sympatetic activation in autonomic branch of the periphieral nervous system
What are different types of Electrodermal activity:
- Tonic activity (baseline of continuous activity)
- Phasic activity (temporary evkoed response (to stimuli))
- Nonspecific activity ( temportary evoked response (without stimuli))
