Exam 2 Chapter 6 Flashcards
Program Effect/Program Impact
Difference between outcomes that occurred with the implementation compared to the outcomes that would have occurred if the program had not been implemented
In other words, what outcomes and how much of those outcomes did the program cause?
Unintended effects
Affects not intended by program (positive and negative)
Variable program effects
Moderation analysis to determine if the program impacts subpopulations in different ways
Dose-response analysis
It is expected that larger doses of the program will have stronger impacts. If this doesn’t happen the program may not be having any impact.
What is a sufficient dose of the program?
Additional impact questions include
Relationship between fidelity and impacts
Average effect
average difference attributable to the treatment
This is a measure of what impacts: What works and what doesn’t work?
Intended
This is a measure of what impacts : What works for whom under what circumstances and why?
unintended impacts, variable response, dose related response and fidelity
When Is an Impact Evaluation Appropriate?
impact evaluation is appropriate for any program whose mission includes bringing about change in some set of identifiable outcomes
Ways to test a new policy or program:
Efficacy Trial
Effectivness Trail
Proof of Concept
Efficacy trial
Demonstrates that the program or policy can achieve desired outcomes
Program or policy is often carried out under ideal circumstances
Proof of concept
For example – high-quality preschool programs can have life long effects
Doesn’t provide evidence that large scale effects will be achieved if program is implemented with lower fidelity and/or quality
Effectiveness trial
For new program after they have done an efficacy trial
Can the program achieve its outcomes in real world not just ideal conditions?
Can the program be delivered with enough fidelity and quality to achieve outcomes?
When is evaluation is especially appropriate for ongoing programs?
There may be a time when a program is modified and refined to enhance its effectiveness, accommodate revised program goals, or reduce costs.
Counterfactuals – what would have happened without the program?
Outcomes in the absence of the program
What is the best counterfactual?
Randomization
Can be difficult to impossible for many programs
Fundamentals of Causal Inference
The Sample
Internal Validity
External Validity
Fundamentals of Causal Inference: The Sample
Any program we evaluate is only delivered to a subset of the target population the program was designed to serve
Perry Preschool project only served a very small percentage of preschoolers at risk for academic underachievement in the US
Fundamentals of Causal Inference: Internal Validity
Validity or accuracy of an effect estimate for the specific participant study sample used in the impact evaluation.
Internal validity increases with full participation of all program participant and use of reliable and valid measures
Fundamentals of Causal Inference: External Validity
External validity
Refer to the extent to which the program effect estimates we measured for our evaluation sample will generalize to the entire target population
Would the entire population of preschoolers at risk for academic underachievement have the same positive outcomes as our evaluation sample if they attended Perry Preschool?
External validity is high if our sample is similar to the target population on characteristics that affect our outcomes
The fundimental problem with causal inference:
when the outcomes for those exposed to the program are observed, their potential outcomes without program exposure must somehow be inferred in order to determine the program effect
