Policy eval Flashcards
What is policy evaluation + evaluation problem
-Policy eval invovles determining whether a policies objectives have been achieved.
-A problem with pol evaluation is that causality is difficult to determine therefore requires scrutiny in defiition of causal inferece. Eg) X is a cause of Y if Y wouldnt of occured without x.
-Eg) determinng effect of training program on income. Observing income risk isnt enough, this mightve occured due to other factors
Problem with causaul inference + formula
-Causaul effect of a policy interence is given by difference in outcomes with and without policy intervention.
= (Y|D = 1) - (Y|D = 0).
Problem is that impossible to observe units in both states of the world simultenously, therefore cannot caluclate unit level causal effects.
How do we overcome problem of causal inference
-Need to establish a counterfactual, which is something that could’ve happened but didn’t happen. In this case the counterfactual would be the outcome in the world where intervention didn’t take place.
-Training program example) find a approx of outcome without training, find comparison group similar in characteristics of those who had the policy intervention. Comparison group stands as estimate for counterfactual.
assumptions for valid comparision group to overcome causal effect problem
-Average characteristics between treated and control groups equal.
-Both groups not exposed to further interventions.
-Treatment and control groups mustve reacted the same in the presence of treatment.
Describe two problematic methods of finding a counterfactual (problem with determining counterfacutal)
-Before and after coparisions refers to looking at impact of policy intervention by looking at participants outcomes of interest over time.
-Treated vs non treated refers to comparing treated and not treated therefore counterfactual in this case is outcome for indivudals who dont decide to enrol, which isnt a good counterfactual leading to selection bias. Those who dont enrol but recieve polict may have diff characteristics to those who do enrol.
Describe random section (method to move more towards causality)
-Randomisation provides a good counterfactual, by generating a comparison group of similar characteristics on average.
-Treatment and comparison group will be statistically identical, therefore on average only difference between groups is the policy intervention.
-If we know random assignment took place causal effects can be estimated by funding difference between average outcomes of each group.
Advantages and disadvantages of randomised selction for moving towards causality
AD:
-Provides a good counterfactual, statistically same groups on average.
-Estimated impact is the true impact of policy on a particular group of the population.
-Internal validity
DIS:
-Hard to genralise the result of the policy intervention to other groups of the population as they have different characteristics.
-Randomisation not alwats applicable, program may have already finished.
Describe method of difference in differences in moving towards causality + method
-DID refers to a non-experimental method for evaluation impact of policy by performing a comparison between:
-Before after changes in the outcome of participants ( treated group.)
-Before/After changes in the outcome of non-participates (comparison group.)
-The first difference (treated group) controls for factors which are constant.
-Second difference (comparision group) controls for any factors which vary over time at the same rate for both non and participants
When is DID a good method of comparing effects of policy + disadvantage
-Characteristics of both treatment and comparison group can be assumed to be constant over period of analysis.
-Baseline/benchmark set of data required.
-Effective if the difference in outcomes of both groups would have changed at the same rate over time, without the program. Genrally applicable method
Disadvatange: several factors acn cause the trends in outcomes of both groups to differ leading to bias of estimations.
DiD effect of policy formula
DID = (Ypost/treatment - Ypre/treatment) - (Ypost/compartison - Ypre/comparison)
Describe IV method for reducing endogenity, aiming towards more causality
-IV variable method attempts to account for the correlation between endogenous variables and the error term. We need to find an IV, Z which is correalted to program participation but isnt correnalted to unobserved charactersistic, error term.
-Used to account for endogenity in indnivdual participation and reverse causality.
-Should be used with caution, weak IV lead to more bias OLS.
-IV method works by identifying exogenous variation in treatment group by using a variable which effects only the treatment group but not the unobserevd factors (error term)
Regressions discontinuity designs RDD + for RDD to work effectively
-This method works by creating a counterfactual from an exogenous elegibility rule of a programme participation, only specific people entitled to enrole.
-Needs to be a continous elgecbity index, eg) income, age test score, an index that can be ranked.
-There is an exogenous cut off point where above or below candidates are allowed to participate.
Problems with Regressions discontinuity designs RDD
-Closer to the cut off the better
-Cannot use data on the threshold
-Narrower the data used less available, therefore have to use data further away.
-Trade off between statistical power and bias
Describe matching methods (more causality/better evaluation)
-Uses statisitcal methods to contruct an artifical counterfactual group. Works by for each participant identify a non participant that is similar as possivle in observable characterstiics. The outcome of this group is now the counterfactual.
-Useful when many observable factors.
-Direct exact mathcing or propensity matching which creases statisitcal comparison based on prob of participaing.
-Aim is to constuct comparison group in which charactersitics are average to those in the treatment group
Advantages and disavantages of maatching methods
-ADS
-Can be applied in many contexts as long as non participants exist.
-Normally useful when combined with other appraoches such as DID
DIS:
-Comparision group is dependent on data avaliable, may not be enoigh data.
