Lecture 4 - Evaluation Flashcards
Formative Evaluation
Not public, just a draft with all the information that needs to be evaluated
- Exploratory
- Informs the design process
- Gives insight into design problem and solution
Summative Evaluation
- Conclusive
- Assesses the success and quality of the design
Quantitative
- About result of Human Robot Interaction
- Systematic
- Empirical/numerical data
Qualitative
- About the process of HRI
- Meaning of / description of
- Non-numerical data
Simulation Types
- Artificial interactions
- Mock-up interactions
Simulation
They evaluate:
- Technical performance
- System behavior
User Study
- Real people /End users
- Essential for user-centered design
User study - In the lab
- More control
- Lower validity
User study - In the wild
- Less Control
- More validity
Types of evaluation
- Exploration
- Comparison
- RCT
- Pilot
- Hybrid
Exploration Goals
- Gain insight into processes
- Generate research questions/hypotheses
Exploration Characteristics
- Formative
- Qualitative
- One condition
- Less strict
- Low-medium # participants
Comparison Goals
- Validate design
- Measure the effect of the design
Comparison Characteristics
- Compares robot with and without designed behavior
- Summative
- Quantitative
- Strict procedure
- Medium-high # participants
RCT Goals
- Validate Solution
- Measure effectiveness
RCT Characteristics
- Compare solution to control group
- Summative
- Quantitative
- Very strict procedure
- High # participants
Pilot Goals
- Evaluate proof of concept
- evaluate research set-up
Pilot Goals
- Formative
- Less strict
- Low # participants
Hybrid
Save time and resources. It is a mixture of other study type goals and characteristics.
How to choose a research question?
- Design objective
- Use case objective
How to choose a hypothesis?
- Claims
- Evidence-based
Independent Variables
Variables that the user control. There are levels
Dependent variables
Things that the user can measure. Ratios, Cardinal or Nominal.
Between Subject setting
different people test each condition so that each person is only exposed to a single user interface. AB test is an example of this.
