Policy Evaluation Flashcards
how do we measure the impact of a programme?
d = (Y | D=1) - (Y | D=0)
What is the funamental problem of causal interference?
We cannot observe D=1 and D=0 at the same time
Who wrote about the fundamental problem of causal interference and when?
Holland 1986
What is a counterfactual?
Outcome in the world where the intervention did not happen
What assumptions do we need for a good counterfactual?
1) Same average characteristics
2) Same reaction to treatment if both groups were exposed
3) Neither exposed to any other interventions
What is the Before- After comparison?
Using the before treatment as a counterfactual
What are the issues with the before after comparison?
The outcome for the individual may have changed even if they weren’t treated.
What is the key assumption for before after comparisons?
The outcome for the individual would have been the same as it was before the intervention
What is the treated vs non treated comparison?
Comparing those treated with those who weren’t treated by a programme
What is the key assumption for treated vs non treated?
That both trends are the same if they were in the same conditions
What are positive and negative selection biases
Positive selection bias - impact is overestimated
negative selection bias - Impact is underestimated
What is the idea of randomised selection
Using random assignment as an allocation rule
-provides a good counterfactual
How does randomised selection work?
- Random evaluation sample picked from population
- Random treatment group picked from the evaluation sample, comparison group is those left over
What can be said about the characteristics of the two groups in randomised selection?
Average observable and unobservable characteristics will be identical
-random sample in large population - normal distribution
What are the advantages and disadvantages of randomised selection?
Adv: estimated impact is the true impact of that particular population
Disadv: External validity, hard to generalise the result
What are some of the complications associated with random selection?
1) Programme may have already finished
2) Non-excludable policies eg smoking bans
3) Unfeasible to randomise, eg road tunnels
4) Ethical reasons - health reasons
What is regression discontinuity design?
- non-experimental impact evaluation method
- Creates counterfactual from exogenous eligibility rule of program participation
- many policies use index or threshold to decide who can enrol
- compares those nearest the threshold to get average characteristics similar
What are the two conditions/ assumptions for RDD?
1) Continuous eligibility index for ranking individuals
2) Exogenous clearly defined cut off point for participation
Who conducted the merit awards impact on future performance experiment?
Thistlewaite and Campbell
1960
What are the problems with RDD?
- The closer you get to the threshold the better BUT there’s little data available.
- Trade off between statistical power and selection bias (close to threshold No selection bias but little stat power)
- Limited external validity - only applies to those close to the cut off
What are the main advantages and disadvantages of RDD?
Adv)
- Close to threshold impact is very accurate
- Mimics random selection
Disadvantages)
- only a local impact
- Lack of data around cut off limits meaningful estimates
Wha is the difference-in-differences method?
- non experimental
- Explores changes in outcomes over time between treated and non treated.
- Cross between before/After and Treated vs Non-Treated
What is the key assumption for DID method?
- Parallel/common trends assumption
- trends of treated and comparison must be identical before the programme
How do we measure the DID value?
DID= [ Yt(post) - Yt(pre) ] - [ (Yc(post) - Yc(pre) ]
When is a good time to use the difference - in - differences approach to evaluation?
1) When characteristics of two groups can be assumed to be constant over the period of analysis
2) We have pre program data
3) Outcomes of two groups would have respond to program in identical manner
What are the advantages of the DID method?
ADV
- Generally applicable method
Disadvantages
- Several factors can cause differences in trends - DID attributes these changes to the program
- Bias may still appear if trends are parallel before program
What is Ashenfelter’s dip?
Ashenfelter noticed that people who enrolled in training programmes often had a drop in income pre programme
- This cause an upward bias of the DID estimate
What is the matching method?
- non experimental method
- Artificial counterfactual made by finding people with similar characteristics to those of the treated group.
- Differences are then compared to their similar treated counterpart
- Average of differences is found
What are the two ways to match participants?
1) Directly on observable characteristics (exact matching, very rare)
2) On a propensity score, tells us the probability of enrolment based on observable characteristics
What are the advantages of the matching method?
Main advantage
Can be applied in almost any context as long as we have non participants
What are the main disadvantages of the matching method?
- May not have enough data available for matching
- May have no comparable non-participants
- Observed factors must determine program participation, very strong assumption