Chapter 8 Flashcards
The two impact evaluation designs with the greatest inherent ability to yield unbiased effect
estimates:
Randomized control designs and regression discontinuity designs
Random Control Trial Design
A group design that uses random assignment to intervention or control produces equivalent groups if we randomize enough units
Points on RCT units of analysis:
Measuring larger units requires specialized statistics
Only very large scale evaluations have the capacity to include larger units
How to conduct a successful RCT
Get buy in from all stakeholders
Minimize attrition
Monitor randomization
Plan an adequately powered experiment: recruit the number of units you need plus the amount you expect to lose
How to minimize attrition:
Randomize after pre-test
Incentives
Assertive efforts to find participants
Assumption of independence
Each individual score on variable being measured is only being measured as an individual. Can cause type 1 error.
How to monitor randomization
Monitor program diffusion, see who’s acessing it.
See if subjects receive services elsewhere.
Monitor Resentful demoralization of control group
Monitor Local history
Limitations of Randomized Control Trials
Stakeholders can be extremely resistant to randomization
Evaluation can affect how the intervention is delivered
Time and Cost
Differential Mortality and local history effects
Ethical Threshold for Randomized Field Experiments
Present practice must need improvement
Efficacy of practice must be uncertain under field conditions
There aren’t simpler alternatives for evaluating the intervention
Results are potentially important for social policy
Design must meet ethical standards of both researchers and service providers
Regression Discontinuity Design
Assigns participants to treatment or control groups based on a quantitative assignment variable. A predefined cutoff score on this variable determines program access.
Benefits of RDD
Strongest quasi-experimental design
Don’t have to guess what characteristics differ between groups
Acceptable to program staff
Can assign participants to intervention based on need
More ethical and internally valid
Cutting point
Threshold measure for risk or severity in reference to admittance to experimental group.
Why RDD is the simplest way to analyze data
Statistically control for variable used as “cutting point”
Compare intervention group to control group
If cutting point was a measure of need for program we expect the outcomes of intervention group to be similar to outcomes for control group
Other ways to analyze RDD data (Bands)
Only include the participants just above and just below the cutting point because they are assumed to be equivalent
Disadvantages
Requires very large N
Doesn’t allow us to analyze program effects on participants farther above or below cut point causing us to miss important subgroup differences
Bands
group of closely related participants relative to cut point graph
How to analyze bands
Analyze each band separately until we begin to see a noticeably different program effects
Only include bands with similar program effects in final analysis
Assign bands that are closest to cut point a higher weight so that they have a bigger impact on our program effect estimates
Disadvantages
Doesn’t allow us to analyze program effects on participants at the far ends of our distribution could still cause us to miss subgroup differences
Intent to treat analysis (Textbook def)
a method of estimating program effects by comparing the outcomes of individuals assigned to the program group and the control group based on their original assignments, regardless of whether they complied with that assignment or not.
intent to treat analysis (slide notes)
After intent to treat analysis, then you can do treatment on the treated analysis.
Include every participant in the intervention group even if they didn’t receive the full intervention
Preserves equivalence between control and intervention groups
Provides more “real world” effect sizes
Underestimates the program effect size on those that actually receive the program
Considered main test of program efficacy
Other parts of treatment on treated design analysis
Test for differences between participants in intervention group who received acceptable dose and those that did not
Dose response effects
Moderation analysis
Treatment on the treated analysis
Analyze program effects on participants who actually received acceptable dose of program compared to those who did not
Dose response effects
Was a “higher dose” of the program correlated with better outcomes among participants who received the program?
Moderation analysis
Did the program work better for some subpopulations than others?
Part of treatment on treated analysis
