Policy Evaluation

Question 1

how do we measure the impact of a programme?

Answer 1

d = (Y | D=1) - (Y | D=0)

Question 2

What is the funamental problem of causal interference?

Answer 2

We cannot observe D=1 and D=0 at the same time

Question 3

Who wrote about the fundamental problem of causal interference and when?

Answer 3

Holland 1986

Question 4

What is a counterfactual?

Answer 4

Outcome in the world where the intervention did not happen

Question 5

What assumptions do we need for a good counterfactual?

Answer 5

1) Same average characteristics
2) Same reaction to treatment if both groups were exposed
3) Neither exposed to any other interventions

Question 6

What is the Before- After comparison?

Answer 6

Using the before treatment as a counterfactual

Question 7

What are the issues with the before after comparison?

Answer 7

The outcome for the individual may have changed even if they weren’t treated.

Question 8

What is the key assumption for before after comparisons?

Answer 8

The outcome for the individual would have been the same as it was before the intervention

Question 9

What is the treated vs non treated comparison?

Answer 9

Comparing those treated with those who weren’t treated by a programme

Question 10

What is the key assumption for treated vs non treated?

Answer 10

That both trends are the same if they were in the same conditions

Question 11

What are positive and negative selection biases

Answer 11

Positive selection bias - impact is overestimated

negative selection bias - Impact is underestimated

Question 12

What is the idea of randomised selection

Answer 12

Using random assignment as an allocation rule

-provides a good counterfactual

Question 13

How does randomised selection work?

Answer 13

• Random evaluation sample picked from population

- Random treatment group picked from the evaluation sample, comparison group is those left over

Question 14

What can be said about the characteristics of the two groups in randomised selection?

Answer 14

Average observable and unobservable characteristics will be identical
-random sample in large population - normal distribution

Question 15

What are the advantages and disadvantages of randomised selection?

Answer 15

Adv: estimated impact is the true impact of that particular population
Disadv: External validity, hard to generalise the result

Question 16

What are some of the complications associated with random selection?

Answer 16

1) Programme may have already finished
2) Non-excludable policies eg smoking bans
3) Unfeasible to randomise, eg road tunnels
4) Ethical reasons - health reasons

Question 17

What is regression discontinuity design?

Answer 17

• non-experimental impact evaluation method
• Creates counterfactual from exogenous eligibility rule of program participation
• many policies use index or threshold to decide who can enrol
• compares those nearest the threshold to get average characteristics similar

Question 18

What are the two conditions/ assumptions for RDD?

Answer 18

1) Continuous eligibility index for ranking individuals

2) Exogenous clearly defined cut off point for participation

Question 19

Who conducted the merit awards impact on future performance experiment?

Answer 19

Thistlewaite and Campbell

1960

Question 20

What are the problems with RDD?

Answer 20

• The closer you get to the threshold the better BUT there’s little data available.
• Trade off between statistical power and selection bias (close to threshold No selection bias but little stat power)
• Limited external validity - only applies to those close to the cut off

Question 21

What are the main advantages and disadvantages of RDD?

Answer 21

Adv)

• Close to threshold impact is very accurate
• Mimics random selection

Disadvantages)

• only a local impact
• Lack of data around cut off limits meaningful estimates

Question 22

Wha is the difference-in-differences method?

Answer 22

• non experimental
• Explores changes in outcomes over time between treated and non treated.
• Cross between before/After and Treated vs Non-Treated

Question 23

What is the key assumption for DID method?

Answer 23

• Parallel/common trends assumption

- trends of treated and comparison must be identical before the programme

Question 24

How do we measure the DID value?

Answer 24

DID= [ Yt(post) - Yt(pre) ] - [ (Yc(post) - Yc(pre) ]

Question 25

When is a good time to use the difference - in - differences approach to evaluation?

Answer 25

1) When characteristics of two groups can be assumed to be constant over the period of analysis
2) We have pre program data
3) Outcomes of two groups would have respond to program in identical manner

Question 26

What are the advantages of the DID method?

Answer 26

ADV
- Generally applicable method

Disadvantages

• Several factors can cause differences in trends - DID attributes these changes to the program
• Bias may still appear if trends are parallel before program

Question 27

What is Ashenfelter’s dip?

Answer 27

Ashenfelter noticed that people who enrolled in training programmes often had a drop in income pre programme
- This cause an upward bias of the DID estimate

Question 28

What is the matching method?

Answer 28

• non experimental method
• Artificial counterfactual made by finding people with similar characteristics to those of the treated group.
• Differences are then compared to their similar treated counterpart
• Average of differences is found

Question 29

What are the two ways to match participants?

Answer 29

1) Directly on observable characteristics (exact matching, very rare)
2) On a propensity score, tells us the probability of enrolment based on observable characteristics

Question 30

What are the advantages of the matching method?

Answer 30

Main advantage

Can be applied in almost any context as long as we have non participants

Question 31

What are the main disadvantages of the matching method?

Answer 31

• May not have enough data available for matching
• May have no comparable non-participants
• Observed factors must determine program participation, very strong assumption