Lecture 4 - Evaluation Flashcards
Formative Evaluation
Not public, just a draft with all the information that needs to be evaluated
- Exploratory
- Informs the design process
- Gives insight into design problem and solution
Summative Evaluation
- Conclusive
- Assesses the success and quality of the design
Quantitative
- About result of Human Robot Interaction
- Systematic
- Empirical/numerical data
Qualitative
- About the process of HRI
- Meaning of / description of
- Non-numerical data
Simulation Types
- Artificial interactions
- Mock-up interactions
Simulation
They evaluate:
- Technical performance
- System behavior
User Study
- Real people /End users
- Essential for user-centered design
User study - In the lab
- More control
- Lower validity
User study - In the wild
- Less Control
- More validity
Types of evaluation
- Exploration
- Comparison
- RCT
- Pilot
- Hybrid
Exploration Goals
- Gain insight into processes
- Generate research questions/hypotheses
Exploration Characteristics
- Formative
- Qualitative
- One condition
- Less strict
- Low-medium # participants
Comparison Goals
- Validate design
- Measure the effect of the design
Comparison Characteristics
- Compares robot with and without designed behavior
- Summative
- Quantitative
- Strict procedure
- Medium-high # participants
RCT Goals
- Validate Solution
- Measure effectiveness
RCT Characteristics
- Compare solution to control group
- Summative
- Quantitative
- Very strict procedure
- High # participants
Pilot Goals
- Evaluate proof of concept
- evaluate research set-up
Pilot Goals
- Formative
- Less strict
- Low # participants
Hybrid
Save time and resources. It is a mixture of other study type goals and characteristics.
How to choose a research question?
- Design objective
- Use case objective
How to choose a hypothesis?
- Claims
- Evidence-based
Independent Variables
Variables that the user control. There are levels
Dependent variables
Things that the user can measure. Ratios, Cardinal or Nominal.
Between Subject setting
different people test each condition so that each person is only exposed to a single user interface. AB test is an example of this.
Within-subjects setting
The same person tests all the conditions
Types of settings
- Between subject vs within subjects
- One measurement vs repeated measurements
- Randomization
- Counter balancing
- balancing user characteristics
What to do with participants
- Who, why, where and when.
- They need informed consent
- amount of participants
- Recruitment and preparation
Stepwise procedural description of the experiment
Description of the process from the moment the participant is collected to the moment the participant is escorted out of the room.
Measures and Instruments
- Biographical Data
- User experience: Measure for effectiveness, efficiency, satisfaction.
- Psychological constructs
- Interview
- Observation
- Logs
- Others
Types of observation
- Human/automatic
- Present/remote
- Direct/delayed
- Observers independent/dependent, trained, mechanical turk
Types of results
- Factual: No interpretation of the results, no speculation
- Results and analysis method
- Metrics and statistics: Cover assumptions using statistical tests.
- Tables & figure
Discussion
- Interpretation, implication, and argumentation of results
- Lesson learned?
- Limitation to the study
