Evaluation and Ethics Flashcards
Quantitive vs Qualitve Evaluation
Quantitive: Collects measurable data, statistical analysis, comparing systems, evaluating hypothesis
Qualitative: Finds patterns, gives deeper understanding, motivates new theories
User studies
Procedure: Users complete tasks in a lab
Goal: Compare systems, is hypothesis true, how good is usability of a system
Pro: Controlled, statistics, objective
Cons: No why, expensive, superficial
Data Collection Studies
Procedure: Identify data and then collect it and then analyze it
Goal: How is X rated, How many X exist, How many people read website X
Pros: No participants, often uses public data, cheap, automatable
Cons: Not always ethical, potentially lower quality, cant get specific data
Online Surveys
Procedure: Identify questions, run survey, analyze data
Goal: How many passwords do users have, How to people rate system, what are characteristics of users
Pros: easy to reach participants, fast, scalable
Cons: not controlled, cheating participants, might not reflect whole picture
Interviews
Open ended: some questions are planned, interviewer can change course
Structured: Rigid script, only react if script includes interruption
Pros: great to explore, get rich data and insight, doesnt require loads of participants
Cons: not generizable, requires communications skills, hard to define saturation
Interview Questions
Closed: yes/no, multiple choice, easy to analyze
Open: Asking for responses, good for exploring
Tasks: Complete this sentence
Focus group vs co-creation
Focus group: Group interview that simulates conversation in labotory setting
Co-creation: collaborative creation process among multiple people used for research
Pros: Good for exploring, group dynamic, does not require many people
Cons: Group dynamics, hard to schedule, not generizable, analysis might be costly
Ethics
Risk assesment, informed consent.
Deception study requires more careful understanding and debriefing
