Randomized field experiments Flashcards
Full control versus randomization
- random treatment assignment is different from a fully controlled experiment
- fully controlled experiment actually yields unit-homogeneity (and environment-homogeneity)
- de facto impossible to do, esp in the social sciences
- think of random assignment of di as ‘breaking’ all arrow-relationships around D
- ‘on average’ or ‘in expectation’ xi are unrelated to di , i.e. that X is unreltaed to D)
inference becomes straight-forward
(difference-in-means test between T and C groups, or OLS)
Randomization and STratification in Corruption example
Randomization Level
- audit treatment at subdistrict level
- means: either all or no village in subdistrict gets treated
- reason: to avoid spill-over effect from audits in one village to others
Stratification
- by subdistrict (invites + comments), by district and duration in KDP (audits)
- ensures that share of treated villages is equal in all subdistricts (i + c)
- ensures that share of treated subdistricts with a given time in KDP is equal in all districts
(audits)
- Important: stratification ensures this to be true ex post, not just in expectation as randomization would
Interpreting experimental results
Overall versus ceteris paribus effects
- effect of D contains effect of treatment itself as well as effects of any responses to the treatment
- example of cash transfers to schools ! parents may respond by lowering education-related expenditure
Implications
- ATE from experimental treatment estimates overall effect, not ceteris paribus effect
- mechanism decomposition difficult ex post
I consider nepotism results in controlling corruption example - we only know that audits reduced fund-diversion, and that they enhanced employment of family members
- we do not know if the overall effect is partially due to trusted family members being less corrupt, or it is the positive net effect of audits despite increased nepotism
- disentangling that would have required ex ante theorizing and collection of respective data
Types of field experiments and treatment design
- classic randomization (‘trial’)
– some get treated, some don’t, no questions asked, no fussing around - randomized piloting or phase-in
– some groups are treated earlier, but eventually all groups get the treatment ) often lowers ethical concerns
) beware of anticipation effects - encouragement designs
– useful if availability of treatment is universal but take-up is not
– administered ‘treatment’ is only encouragement to opt for actual treatment
-> difference between those receiving encouragement and the actually treated leads to
complications in estimation (see below) - oversubscription designs
– randomly admit some marginal rejects in addition to those treated anyhow -> minimal ethical concerns or interference with existing measures
Assignment modes with multiple treatments
Joint treamtement
Multiple treatment
cross-cutting treatment
- multiple treatment option allows to assess relative effectiveness of each treatment
- cross-cutting design allows additionally to investigate interactions between treatments
The level of randomization
- is not always smallest unit of observation -> risk of spill-overs e.g. envy leading to non-compliance - individual-level randomization may be unfeasible or uneconomical due to fixed cost
- randomization at a higher level
-> can minimize spill-overs
> but affects statistical power, and thus sample size and budget - in the corruption study earlier
> the audit treatment was randomized at the subdistrict level because of feared spill-overs
> the invites+comments treatment was randomized at the village level
statistical power of an experiment
- is the probability that we reject the H0 (‘no effect’) for a given real effect size and significance level
- alternatively, think of power in terms of the minimum detectable effect size (MDE)
> when designing an experiment, power calculation is crucial to determine required number of subjects and randomization strategy
Design factors that influence power of an experiment
- number of subjects and share of T versus C groups
- group-level treatment
> rule of thumb: increasing number of groups better than increasing number of subjects per group - imperfect compliance
> rule of thumb: rate of non-compliance lowers power by more than number of observations increases it - control variables
> trade-off between reducing variance versus losing degrees of freedom > baseline value of Y is always a good control to have - stratification
> generally more effective than including controls
Two main reasons for imperfect compliance
- it may be that subjects in the control group receive the treatment
> spill-overs (envy or ‘desperation’)
> strategic action (defiance if subjects resent being experimented with) > the treatment might be available elsewhere
- it may also be that some subjects in the treatment group to not receive treatment > they refuse
> the mistakenly miss out > maybe implementation was disturbed > encouragement designs do not even attempt to treat all> encouragement designs only aim at affecting the probability of subjects to receive the
actual treatment
–> there is thus a difference between ‘intention to treat’ and actual treatment )
–>has implications for estimation of causal effects
Intention to Treat
- denote being in treatment group with
Z = T , Z = C otherwise I den
ote actual treatment received with D = 1, D = 0 otherwise
By comparing the means of the treatment versus control groups, we obtain
E[Y|Z =T] - E[Y|Z =C].
This is the intention to treat estimate (ITT), but not the ATE, which is
E[Y|D = 1] - E[Y|D = 0].
The Wald estimator
The shares of actually treated in the T and C groups are
exanteE[D|Z =T] and E[D|Z =C],or expost ⇡T and⇡C.
deltaˆWald={E[Y|Z=T]E[Y|Z=C]}/
{E[D|Z =T] - E[D|Z =C] }
ITT / {⇡T -⇡C}
Three assumption for the Wald estimator
- E[di|zi = T] >= E[di|zi = C] for every individual, or
E[di|zi = T] <= E[di|zi = C] for every individual
The first assumption requires that not all subjects need to be affected by Z , but those who are all need to be affected in the same ‘direction’ (monotonicity)
- any difference (Y|Z =T)- (Y|Z =C)is due to Z
the second assumption requires that Z is (‘as if’) randomly assigned (thus, sometimes also called the independence assumption)
- outcome Y is not directly affected by Z , only through D
the third assumption is the same as the exclusion restriction for IVs
LATE
is the effect of the treatment on those whose treatment status is changed by the instrument (the so called ‘compliers’). Neither does it apply to all treated or untreated, nor to the entire sample (like the ATE does).
Three more problems
- probability of treatment differs by stratum e.g. when a fixed number of treatments is assigned to strata with different numbers of subjects
> implies that treatment is not random overall but random within each stratum
> conditioning and averaging over strata (weighted by treatment probabilities) can solve this (see lecture 1)
> use of OLS with a saturated model works, too - externalities / spill-overs
> means that SUTVA is violated (i’s treatment effect independent of j’s assignment)
> rule of thumb: when spill-overs from T to C group are positive ! underestimation of effects - non-random attrition
> over-time reduction in ability to collect data on certain subjects > problem even with equal attrition rates in T and C groups
Pro’s of field experiments
- experiments are a powerful and arguably the cleanest way to ‘identify’ and estimate effects of causes
- most design problems are by now well-understood; large method toolbox
- social sciences hard to imagine without field experiments
- not only used in development research, also in firms, in public opinion, in campaigning and elections
