Design Evaluation

Question 1

Why evaluate design?

Answer 1

• Improve usability
• Reduce flaws after launch

Question 2

What part of the design is being evaluated?

Answer 2

• All levels, parts, and attributes

Question 3

Where is the design being evaluated

Answer 3

• Laboratory (for controlled scenarios and internal validity)
• Natural Setting (for realistic scenarios and external validity)

Question 4

When to evaluate design?

Answer 4

• Anytime
• Formative process (before formation — qualitative)
• Summative process (after formation — quantitative)

Question 5

What are evaluation issues?

Answer 5

• Surroundings (Environment must be considered in the evaluation_
• Bias (Undiversified sample group)
• Hawthorne Effect (Reactivity due to awareness of being observed)

Question 6

What is Analytical Evaluation?

Answer 6

• Evaluation that doesn’t involve users
• Predict user behaviour and identify usability problems
• Discover design issues before usability testing

Question 7

What is Heuristic Evaluation?

Answer 7

• Fast usability engineering
• Simple prototypes
• 3-5 Users (testers)

Question 8

What steps are in Heuristic Evaluation?

Answer 8

1. Evaluators evaluate the interface multiple times — compare components with usability principles
2. Evaluators aggregate their findings
3. Reveals usability problems

Question 9

What is Usability Testing?

Answer 9

• Identify problems in a product’s design and learn about user behaviour and preference
• Measure performance and satisfaction with a system
• Laboratory setting

Question 10

What components are in Usability Testing?

Answer 10

• Participants
• Tasks
• Facilitator

Question 11

What steps are in Usability Testing?

Answer 11

1. Prepare tasks, find participants, and set up test materials
2. Invite participants, and observe and ask questions
   
   1. Give scenario verbally and textually
   2. Give tasks one at a time
   3. Give short breaks if tasks are long
   4. Take notes and collect data (audio and video)
   5. Debrief participants after tasks
3. Analyze data, find problems, summarize results, and make recommendations

Question 12

How do you effectively collect data?

Answer 12

• Think-Aloud Protocol
  
  • During testing — participants talk about how they feel during tasks
  • After testing — participants recall how they feel after a task
• Audio and video recording
• Questionnaires
  
  • Start with general demographic information
  • Use open-ended questions

Question 13

What type of questions are there?

Answer 13

• Likert Scale
• Semantic Scale
• Ranking Questions
• Open-ended Questions

Question 14

What is the UEQ?

Answer 14

• 26 pairs of contrasting attributes to be answered
• Semantic scale questions
• Comes with data analysis tools

Question 15

What is the UEQ-S?

Answer 15

• Shorter version of UEQ
• 8 pairs instead of 26
• Aggregates into 3 scales:
  
  • Pragmatic quality — practical, goal directed
  • Hedonic quality — appeal, non-goal directed
  • Overall

Question 16

What is qualitative data/analysis?

Answer 16

Observational-based
- Facilitator takes notes and recordings
- Participants write/voices comments

Question 17

What is Quantitative Data/Analysis?

Answer 17

Assessment-based
- Measures from participant tasks and questionnaires

Question 18

What should you look out for during Qualitative Analysis?

Answer 18

• Frequently occurring problems
• Workarounds
• Enjoyment

Question 19

What do you do with results from Qualitative Analysis?

Answer 19

• Prioritize problems based on severity and use usability goals to redesign
• Highlight the good results
• Provide recommendations, not rules (consider constraints such as branding and standards)