Usability & UX evaluation Article information Flashcards
What are Annotated portfolio’s?
They are a way to communicate design research
- This is done to suggest that design can become productive as research
How are Annotated portfolio’s made?
- By selecting a collection of designs
- Representing them in an appropriate medium
- And combining the design re-presentations with brief textual annotations
The essence of design research
“The essence of research is to produce knowledge, and the essence of design is to produce artifacts.”
Purposes of research
- Primary: Knowledge production
- Secondary: Changing society, curing illnes and furthering economic growth
What happends with abstraction of artifacts?
- Has to take place in high order for an artifact to reach a high level of knowledge that is applicable across a broader range of scenario’s.
- Abstraction adds knowledge value to an artifact
What is intermediate level knowledge?
Represetens interesting and important knowledge n design
- Heuristics
- Methods and Tools
- Guidelines
- Criticism
- Experiential Qualities
- Patterns
- Strong concepts
- Annotated portfolio’s
Why can Annotated portfolio’s be seen as intermediate level knowledge?
- Annotated portfolio’s can be seen as abstractions in the sense that more abstract entities have wide scope of applicability, which makes it broader knowledge also applicable to other scenario’s
- Even though an annontationn are non-abstractions from the original design, they may still be appropariate as abstractions by the receivers of the annotated portfolio.
What are the five phases of evaluation history?
- The system reliability phase (1940’s-50’s) -> Concerns of evaluation were on system realibility (minimizing system fault time, quickly repairing errors)
- The system performance phase(1940’s-60’s) -> Evaluation focuses on issues related to the system performance (THe processing speed)
- The user performance phase (1960’s-70’s) -> Users and their performance brought a new type of evaluator *experimental psychologist) Tried to enhance productivity through performance based metrics
- Usability phase (1970’s-2000s) -> Evaluation for the ease of use of computers systems developed.
- User experience phase (2000s-Present) -> The user experience paradigm emerged for design and evaluation, We non consider ‘non-utilitarian’ aspects of computer use
In the “Usability phase” what were the dimensions for user performance?
- Learnability
- Throughput
- Attitude
In the “Usability phase” what were the performance metrics?
- Time on task
- Error rate
- Accuracy
- Task completion (Task succes)
- Satisfaction
Fameworks developed in the “User Experience phase”
- Hassenzahl model (Pragmatic & Hedonic)
- Normans levels of emotional design
- Folizzi and Battarbee ‘User interactions that influences UX’
Explain the Hassenzahl model:
- Pragmatic attributes -> Ability to help us achieve behavorial goals (usability metrics)
- Hedonic attributes -> Ability to evoke pleasure, allow self-expression, and provoke memories (identification, stimulation, satisfaction)
Explain Normans levels of emotional design
- Viceral
- Behavioral
- Reflective
Explain Forlizzi and Battarbee ‘User Interaction that influence UX’ model:
- Fluent
- Cognitive
- Expressive
Difference between UX evaluation and Usability evaluation
UX evaluation focuses on Hedonic attributes, Usability evaluation on pragmatic attributes
The present state of evaluations has common approaches, name these:
- Metrics
- Post-use usabillity questionnaires
- Eye tracking
- Mouse-tracking (click-paths)
To test Hedonic attributes, which approaches are used?
- Post-use questionnaires
- Interviews
- Focus groups
- Self-report techniques (emocards, personal meaning maps)
- Facial expressions
- Psychophysiological reactions
What are inspection methods?
They aim to replace user feedback with expert judgment and focus exlusively on pragmatic attributes
What are Inspection methods?
- Early methods include guidelines and checklists
- Common method is Expert reviews: Heuristic evaluation & Cognitive walkthrough
What are Field methods?
- Represents techniques conducted in natural settings
- Many of these methods can be adapted to adress either pragmatic or hedonic attributes
- Common field methods
What are common field methods?
- Behavioral observations
- Collages or artifacts
- Log analysis
- Dairy studies
- The experience sampling method
- Living laboratory (A/B testing)
What can be stated about the future of evaluations?
- Evaluators have been foced to dapt to the chaning use context of technological innovations
- Because of the current speed, and complexity of the changes, evaluators struggle to keep pace
On what do most of the discussions around the future of evaluation evolve now?
- Criticism on current methods
- Concerns raised about method reliability
- Potential harmfulness in certain situations
Five potential research directions for future design research:
- Create a more Holostic vision for UX evaluation
- Develop inspection methods for Hedonic attributes
- Examine core skills needed for evaluation
- Investigate informal approaches to evaluation
- Learn from evaluation in practice
User Experience defenition =
A person’s perceptions and responses that result from the use or anticipated use of a product, system or service.
User experience design shift
- UX has been extended to incorporate henoic qualities such as”Aesthetics, stimulation, and Identification
- UX is also a dynamic concept influenced by contextual aspects.
- UX offers a much more holistic and dynamic take on interaction with product that just usability
- UX is a multidisciplinary field with: cognitive science, design, psychology, engineering
Four triangulation opproaches (Denzin)
- Data Triangulation: The use of a mix of data sources in a study
- Investigator Triangulation: A number of researchers investigating the same phenomenon
- Theory triangulation: The use of a number of theories used for interpreting results of a study
- Methodological Triangulation: The use of more than one method to study a phenomenon
Two overarching types of employing two or more methods:
- Sequential: First either qualitative or quatitative is used, and the other type is used in a following study to explain, explore or validate results.
- Concurrent : Two or more methods are emplyed within the same study to cross-validate findings
Guidelines for performance metrics
- An adequate sample size is required
- Avoid overrelying on performance metrics when your goal is to simply uncover basic usability problems
- Performance metrics tell the what, but not the why
Five types of performance metrics:
- Task succes: Binary succes, levels of succes, task failure
- Time on task: measures how much time is required to complete a task
- Errors: reflect the mistakes made during a task
- Efficiency: The amound effort a user expends to complete a task
- Learnability: improvement over time
What is important to think about when you use ‘Task succes’ as a metric
- Every task a user is asked to perform must have a clear end state or goal
- To determine whether a task is succesful, succes criteria must be defined
Types of task failure
- Giving up
- The moderator ‘calls’ it
- Too long
- Wrong
Level of succes
Usability (Complete succces, partial succes, failure)
UX (1= complete succes, 2= Minor problems, 3= Major problems 4= Failure/give up)
What are ways of giving assistance?
- Moderator takes participant back to the home page/ reset task
- Moderator asks the participant probing question or restates the task
- Moderator answers a question or provides information that helps the participant complete the task
- Participant seeks help from an outside source
Common approaches to ending an unsuccesful task:
- Tell the user they should stop until they reach the point they would normally give up
- Apply a three strikes and you’re out rule
- Call the task after a predetermined amount of time has passed
What doest Time on task measure well?
The efficiency of using a problem, this is only when a short time on task is desirable
How to improve the validity of measuring Time on task right?
- Two moderators can measure time at the same time
- The video can be recorded and include time stamps
Errors, when to measure?
- When an error will result in a significant loss in efficiency
- When an error will result in significant costs to your organization or the end user (parkmobile)
- When an error will result in task failure
What type of errors are there?
- Entering incorrect data into a form field
- Making wrong choice in drop down list
- Taking incorrect sequence of action
- Failing to take key action
What issues can occur when counting errors?
- Make sure you do not count errors double
- Sometimes an error rate is not enough, you also need to look at the type of error
Besides time on task, how can efficiency be measured?
By looking into the effor required to complete a task
What types of effort are there?
- Cognitive effort: Involves finding the right place to perform an action, deciding what action is necessary and interpreting the results of the action
- Physical effort: involvves the physical activity required to take action (e.g. moving a mouse)
what is the beheavior lostness?
Lostness consists of:
- The number of different pages visited to perform the task
- the total number of pages visited while performing task, including revisits to same page
- The minimum number of pages that must be visited to accomplish task
What is Learnability?
The extent to which something can be learned efficiently. It’s an essential metrics when you need to know how someone develops over time
What are the approaches to measuring learnability?
- Trials within the same session: tasks are performed one after another without breaks, this doest not take memory loss into account
- Trials within the same session with breaks in between each task (break is a distractor)
- Trials between session: The participant performs tasks over multiple sessions
What are the key metrics according to ISO?
- Effectiveness (accuracy and completeness to which users achieve specified goals)
- Efficiency (resources expended in relation to the accuracy and completeness wiith which users achieve goals
- Satisfaction (the comfort and acceptibility of use)
What is the metric to measure Effectiveness?
- Completion rate
- Number of errors
What is the metric to measure efficiency?
- Task time (in combination with task completion)
- Overall relative efficiency (The relative efficiency uses the ratio of the time taken by the users who successfully completed the task in relation to the total time taken by all users. ) ‘
What is the metric to measure satisfaction?
- Task level satisfaction (after task completion user should fill in a task how difficult that task was)
- Test level satisfaction (is measured by giving a formalized questionnaire to each test participant at the end of the session) This serves to measure their impression of the overall ease of use.
What number of participants for different studies?
- Qualitative user testing: 3-5 users
- Quantitative: 20 users for each design
What are examples of heuristics?
- Visbility of system status
- Match between system and the real world
- Aestetic and minimalist design
- Flexibility and efficiency of use
- Help and documentation
What is laddering?
A technique that is particularly helpful in eliciting goals and underlying values and therefore, possibly helpfull during early stages of user experience research
What is the Means End Chain theory?
Provides a framework for captuaring qualitative laddering research data in consumer space and a model for assessing cnsumer values and behavios:
- Attributes
- Consequences
- Core values
What is a hierarchical value map?
All the major means end and atribute-consequence-value connections and describes individuals behavior based on their core values.
Types of eye movements in eye tracking
- Saccades
Observed while conducting search tasks (jerky eye movements, both eyes) - Smooth pursuit
much slower eye movements, used to track slowly and regularly moving visual targets - Compensatory
eye movements stabilize the image on the retina when the hear or trunk is moved. - Vergence
rotational eye movements of the eyes in opposite direction, in response to focusing error from visual targets moving closer or further away. - Miniature
refers to eye movement that are under 1 degree of visual angle and include small corrective notions - Nystagmus
refers to a unique sawtooth-pattern of eye movement caused by a rapid back-and forth movement of the eye.
Avantages of eye tracking
The ability to record the micro flow of visual attention in a non-intrusive way
Dissadvantages of eye tracking
- Sensitive for failure
- Strong individual differences
Eye-mind hypotheses, what you need to know
- Tasks should be designed to require endcoding and processing of visual information to achieve well-specified task goals
