Exam 2 Chapter 7 Flashcards
The Goal of impact evaluation
to determine what changes in outcomes can be attributed to
the intervention being evaluated
Comparison Group Design
outcomes are obtained for individuals or other units that are naturally exposed to the program without any manipulation of their access or opportunity to participate
Comparison Group Designs vs. Randomized Control Group
Randomized control group designs that have adequate numbers of participants are the best way to determine if a program has caused the change it was designed to change. But they are expensive and require large samples.
A randomized control group design provided evidence that the Perry Preschool caused high-risk preschoolers perform better in school than high-risk preschoolers who did attend the preschool
Biased vs. Unbiased Program Effects
When our evaluation designs work the way they are supposed to work and provide accurate estimates of program effects, we say that the effects are unbiased
When our evaluation designs do not provide accurate estimates of program effects we say that the effects are biased
When is bias present?
when either the measurement of the outcome with program exposure, or the estimate of the counterfactual outcome departs from the corresponding true value
Selection Bias
a systematic preintervention difference between the intervention and comparison group that affects outcomes
Factors that cause selection bias
Unknown differences between people joining program and people in the comparison groups
Attrition
Missing data
Attrition
loss of outcome data for members of intervention or comparison groups that have already been formed
Other sources of Bias: Secular Trends & Interfering Events
Naturally occurring trends that affect one group but not both intervention and comparison groups
Differential secular trends and interfering events are more likely to occur when the intervention group and comparison group come from different communities
Also Regression to the mean and maturation
Quasi Experimental Comparison Group Designs
Naïve effect estimates
Covariate-adjusted regression effect estimates
Matched comparisons (Propensity Score)
Naïve Estimates of Program Effects
(Quasi Experimental Group Comparison Design)
Average outcome for participants compared to non-participants
Measures may come from administrative data or from direct assessments
If the Perry Preschool evaluation had used a naïve estimate of program effects instead of a randomized control trial they might have compared Perry Preschool students to statewide tests of school readiness for first graders
Issues with Naive Estimates
No consideration of bias
Covariate Adjusted Regression Based Estimates of Program Effects (Quasi Experimental Group Comparison Design)
group exposed to a program are compared with those for a comparison group selected on the basis of relevance and convenience. But in contrast to the naïve design, this design uses statistical techniques to adjust for differences between the groups that might bias the effect estimates.
Covariates
baseline variables needed for all the members of the study sample, especially characteristics expected to be related to the outcomes of interest.
Two Types of Covariates
One type has to do with differences on pre-intervention characteristics related to the outcome of interest. The second type is differences between the program and comparison groups in term of their reaction to the program.
Multivariate Regression
model that statistically adjusts the effect estimate for influential
initial differences between the groups
Aspects of Multivariate Regression Techniques
Accounts for pre-intervention characteristics likr
likelihood that participants will receive intervention
Motivation
Ease of access to program
Main Comparison Group Designs
Naive Program Effect Estimates
Covariate-Adjusted, Regression-Based Estimates
Matching Designs (e.g., Propensity Score Matching)
Interrupted Time Series Designs
Matching Group Designs (Comparison Group Design)
Matching Designs (e.g., Propensity Score Matching): This method attempts to create comparable groups based on pre-intervention characteristics to minimize selection bias.
Interrupted Time Series Design (Comparison Group Design)
Interrupted Time Series Designs: interrupted time series designs compare outcomes for a period before program implementation or participation with those observed afterwards. Coinciding events, secular trends, maturation, and regression to the mean, for instance, may bias program effect estimates from time series designs
quasi-experiments
term to describe the impact evaluation designs we have referred to here as comparison group designs
Exact Matching
the objective is to select a “clone” for each member of the program group from the pool of comparison group members
Propensity Score
program participants and the individuals selected as potential matches are first combined in a common dataset and all the covariates of interest are used in a variant of a regression model that attempts to predict who is a program participant
Propensity Score Advantages and Disadvantages
Advantages
Can use a larger set of covariates than is possible in exact matching
Directly addresses selection bias –uses variables that actually predict receiving intervention
Disadvantages
Will still produce biased estimates if important covariates are left out of analysis
Interrupted Time-Series Designs for Estimating Program Effects
Requires multiple measures of outcomes before intervention to establish a trend line and multiple measures after intervention to test if trend changes
Evaluators typically use existing datasets that measure key outcomes in healthcare, education, employment, crime
Very useful in evaluating law or policy changes that affect large geographic areas
Cohort Design(Interrupted Time-Series Designs for Estimating Program Effects)
Cohort designs estimate the program effect by comparing outcomes for the cohort(s) of individuals exposed to a newly initiated or revised program with those for the cohort(s) before that with no such exposure.
For the resulting program effect estimates to be valid, various sources of potential bias would have to be ruled out
Fixed Effects Designs(Interrupted Time-Series Designs for Estimating Program Effects)
Uses outcome data for each unit within a group of units
For example, measuring an outcome several times in a group of individuals before intervention and then measuring an outcome several times in the same group of individuals after the intervention
Example – a school changes its behavior modification protocol – measure each student’s points before change and then after change
Difference-in-differences designs (Interrupted Time-Series Designs for Estimating Program Effects)
Interrupted time series designs that compare pre- and post-intervention outcomes in sites that implemented the intervention to analogous before-after changes in sites in which it was not implemented, thus adding a comparison time series to the intervention one.
Comparative Interrupted Time Series Designs (Interrupted Time-Series Designs for Estimating Program Effects)
Uses same logic as Difference in Differences Designs with one improvement
Include sufficient pre-intervention data to model the trend over time
At least four periods of data are needed prior to the intervention
