Week 10- Policy Evaluation Flashcards
What are public policies designed for?
Public policy programmes are usually designed to change a specific outcome, eg education policy to improve learning.
What is the purpose of policy evaluation?
Policy evaluation attempts to determine whether the policy objectives have been achieved.
What is impact evaluation? Give 2 examples of impact evaluation
Impact evaluation is a specific type of policy evaluation which aims to ascertain answers to cause-effect questions:
• Do vocational training programmes cause an increase trainee incomes?
• What is the causal effects of scholarships on school attendance and achievement?
What types of effectiveness can we consider when we do policy evaluation?
- Absolute effectiveness
* Relative effectiveness (effectiveness relative to what would have happened)
What does ROAMEF stand for in the Central Government Guidance On Appraisal And Evaluation?
- Rational
- Objective
- Appraisal
- Monitoring
- Evaluation
- Feedback
At what stage in ROAMEF does implementation of the policy occur?
The implementation of the policy occurs during the monitoring stage.
What is the definition of causality?
X is a cause of Y if …. Y would not have occurred without X.
If we want to establish the effect of a training program on an individuals income, is observing a rise in a trainees income after training sufficient to assume that the training causes a rise in trainees income?
No, this is not sufficient, as income may have risen in spite of training, and we need to establish to what extent the training contributed to the change in income.
- What type of questions are causal questions?
- Give an example
- What can these scenarios be thought of as?
- Causal questions are “what-if” questions
- What if an individual attended private school vs state school?
- These scenarios can be thought of as treatments or interventions
So how do we calculate/write the causal effect of an outcome? Explain this equation
- 𝛿 = (𝑌|𝐷 = 1) − (𝑌|𝐷 = 0)
- 𝛿 if the causal effect of a policy, D.
- Y is the outcome of interest.
- 𝛿 is the difference between outcomes with and without the policy change (e.g. the training programme).
- What is the Fundamental Problem of Causal Inference (Holland, 1968)?
- Give an example of this
- It is impossible to calculate unit level causal effects because we can never observe the same individual in both states of the world at a given point in time
- Eg we cannot observe an individual as being enrolled in a training programme and not being enrolled in a training programme at the same time.
How do we overcome the Fundamental Problem of Causal Inference (Holland, 1968)?
- We establish a counterfactual- the outcome where the intervention didn’t happen
- We find an approximation for (𝑌|𝐷 = 0) - eg the outcome without training
- We find approximation by finding a comparison group similar to those we are “treating”. These serve as an estimate of the counterfactuals
What are the 3 necessary requirements for a comparison group to be valid? What would observed differences then be as a result of?
1) Average characteristics between both treated and control are the same
2) Treatments and Controls must have reacted the same in the presence of a treatment
3) Treatments and control groups not exposed to different interventions
• Observed differences are as a result of the policy intervention
What are 2 potentially problematic methods to finding a counterfactual?
- Before-After Comparisons
* Treated vs Non-Treated
What are Before-After Comparisons?
- Impact of a policy change by looking at changes overtime of participants outcomes of interest
- In this case the counterfactual is outcome for the same individual/unit prior to the intervention
- Assumption: The outcome for the individual would have been the same as it was before the intervention
