Lecture 2: Causal inference with experimental method Flashcards
What is an experiment?
- Golden standard
- Includes intervention/treatment
- Keep everything constant except for the treatment itself
- Clean design? -> simple t-test/ANOVA
Random Assignment
- Equal chance
- Characteristics of participants between two groups do not differ (limit confounding variables)
- Small but a sufficiently large sample
Simple randomization
- Toss a coin (random)
- Limitation -> not equal number of persons between groups
Block randomization
- Randomize subjects into groups
- Sample size is relatively small
- Not intended for controlling confounding variable
Stratified randomization
- Limit the undesired influence of a known factor
- Based on more than one factor
- Number of strata should be in proportion with the target sample size
Laboratory Experiment
- Control setting
- Participants informed and compensation
- Allows intrusive design
- Reduces chance of confounding variables
- Improves internal validity but reduces external validity
When should we use laboratory experiment?
- Stake of making a mistake is high
- Confounding variables tend to occur
- Setup is not feasible in real-world
- Budget and participants available are limited
Field experiment
- Impact of a treatment in the real-world
- Enabled by digitalization
- Less control over other confounding factors
- Spillover and crossover effects
- Ethical issues
Application in marketing
- Academic -> testing a theory
- Company -> testing an intervention)
- Public policy -> testing an intervention
- Litigator -> testing a hypothesis based on intuition
Designing an experiment
Step 1: Formulate your hypothesis Step 2: Select an appropriate sample Step 3: Set up and run your experiment Step 4: Analyse your data & interpret your findings Step 5: Reformulate your hypothesis
Research Question
A general statement of your observation or assumption
A scientific hypothesis should be
- Testable
- Falsifiable
- Stated as an expected relationship between variables
- Simple and easy to understand
Practical concerns sample size
- Availability participants
- Costs
- Time management
- Data collection method
- Ethical concerns
Power analysis
Assess minimum sample size required given a significance
level, effect size, and statistical power.
Power Analysis Advantages
- Limit p-hacking
- Avoid over (i.e., waste of resources) or under power (i.e., unable to detect
an effect when there is one)
Type II error
Finding an effect when there is no effect to be found
Type I error
Falsely rejecting the null hypothesis
Set up and run your experiment
- Research question
- Resource available
- Access to participants
- Type of experiment
- Design: between/ within subjects design
- Experimental groups and intervention/ treatment
- Pilot test & manipulation check
- Dependent variables
- Threats to classical assumptions
Other considerations
- Demand characteristics
- Motivation
- Language issues
- Gender and physical appearance
- Control variables
- Internal factor
- External factors
- Ethical concerns
Paired sample t-test
Compare the means of the same group across two
time points
Independent sample t-test
Compare the means of two independent groups
Analysis of variance (ANOVA)
Compare the means of more than two groups
ANCOVA
ANOVA with control variables
Chi-square test
Categorical variables
Interpret your findings with caution
- Explore potential confounder (seasonal change)
- Take context into account
- Mine your data (with caution)
- Follow up tests (Robustness check & Replication)
Limitations of experimental method
- High control and internal validity but low ease of use
- Infeasibility of randomized experiments → Ethical concerns
- Difficult to implement
- High financial and opportunity cost
- Availability of participants
