Natural Experiments

Question 1

2 Features of controlled experiments

Answer 1

1. researcher has control over the treatment or ‘intervention’
2. the treatment and control groups are randomly composed → all confounders are on average equal for T and C groups

Question 2

NEs vs CEs

Answer 2

• seek to approximate experimental ideal by relying on ‘naturally’ occurring randomness
• main differences:
• composition of T and C groups often only as if random
• intervention not controlled by researcher
  → NE studies considered ‘observational’ rather than experimental
  → NE studies more discovered than designed

Question 3

sources of (as-if) randomness

Answer 3

• actual lottery (usually government administered)
• jurisdictional borders
• discrete ‘close call’ outcomes (close elections, attempted assassinations, …)
• natural disasters
• date of birth thresholds
• population thresholds (next week)
  General Point:
  Credible sources of (as-if) randomization share that assignment to T and C is due to factors that are beyond the control of the analyzed units.

Question 4

Advantages of NEs

Answer 4

• identification not reliant on statistical modelling (e.g. parametric assumptions)
• estimate of causal effect can then usually be obtained from a comparison of means
• however, often other complementing techniques necessary, such as
• instrumental variable approach (intention to treat)
• fixed effects
• difference-in-differences

Question 5

Check for as-if-randomization

Answer 5

General Remark: For NEs with as-if randomization only, it’s essential to check whether pre-treatment observables are balanced for treated and control groups. They should be, at least conditional on controls.

Question 6

Criticism of Haji study

Answer 6

• concerns over bias from sampling response
• people working a lot less likely to be interviewed
• people living in district other than their travel mates less likely to be interviewed
  → systematic non-response of better situated?
• external validity (Sunnis from one province in Pakistan)

Question 7

disadvantages of NEs?

Answer 7

• in practice, usually other techniques necessary (instrumental variables, fixed effects, difference-in-differences, …)
• relying on NEs affects what is being studied
• research opportunism; natural experiments cannot be designed, just discovered
• danger: ‘focus on variation in search of questions’ versus ‘focus on questions in search of variation’
• NEs rarely useful for contracted policy evalution (NEs not plannable)
• like controlled experiments, not useful to explore effects of causes
• limited use to study mechanisms of effects

Question 8

Quality criteria of NE

Answer 8

Three quality criteria of NE-designs

1. plausibility (of as-if randomization)
2. credibility (of model of data-generating process)
3. relevance (of the substantive treatment)
   - has the treatment the same effect as the real-world cause of interest
   - is the randomized-over population the same as the one of interest

Question 9

research challenge in NE

Answer 9

1. isolation of effect (avoid confounders)
2. capture all variance (model specification)
3. generalization (external validity)