Impact evaluation Flashcards
5 levels of program evaluation
- Needs Assesment (needs)
- Program Theory Assessment (needs, input, output, outcome, impact, long term)
- Process/optional evaluation (input, output)
- Impact evaluation (outcome, impact, long term)
- Cost-benefit analysis
Theory of change
A statement of how inputs being provided lead to intended outcomes and impacts.
Counterfactual
A controlgroup representing an identical world without the program
Without controlgroup: Under-estimation of the effect of the program
Cannot attribute causal impact of the program
3 Techniques for Impact Evaluation
- Randomisation
- Matching methods
- Difference in Difference
Randomisation, what is random
Random sample, treatment+control group, random sample from both groups
RCT issues
External validity
Hawthorne effects (change behaviour when knowing being observed)
John Henry Effect (control treatments work harder to keep up with treated)
Contaminotion/spillover
Dropout or attrition
Partial Eq (Measure short term effects)
Matching method
When randomization is not possible
Propensity Score Matching: find a control group comparable to participant in a large number of observable characteristics: 1. use survey, select large number of characteristics helping predict program participation. Using treated
and untreated estimate the probability of participation p(X ) as function of characteristics → PSM. 2. Match treated to untreated using p(x ). The closer the p(X ) the better
3. The impact of the program is the average difference in outcomes between treated and untreated.
Assumption: unobservable are similar across treated and untreated areas.
Cons: requires lot of data, must be randomisatoin to have similar treaten/untreated
Differens-in-Difference
An alternative is to observe the change over time of non-beneficiaries (when we don’t have random sample).
Subtracts changes.
Parallel trends assumption: without program, the trends between treated and untreated would have been identical.
