Topic 6 - Impacts Evaluation of Development Policies and Programs

Question 1

What is Impact Evaluation?

Answer 1

To determine if and to what extent the change in something is due to intervention of a policy and if the change from that policy was cost effective.

Trying to establish causality

Question 2

What is the issue in establishing causality between intervention (X) and output (Y)?

Answer 2

The diff types of causalities:
1.CAUSALITY: 𝑋 → 𝑌
2.REVERSE CAUSALITY: 𝑌 → X
3. Simultaneous change 𝑋 → 𝑌 and 𝑌 → X
4. SPURIOUS CORRELATION/OMITTED VARIABLE BIAS:
 Observed X/Y correlation due to some common Z.

Question 3

In finding a method what is crucial?

Answer 3

Random selection crucial
Intervention/ Program (P)
Outcome (Y)

Question 4

What is the ideal impact of an intervention?

Answer 4

(𝛿i) = Yi (1) - Yi (0)

Change in outcome in future greater than outcome without intervention

Question 5

What are some commonly used methods for impact evaluation?

Answer 5

1. Experimental design with randomization (Randomised Control Trial (RCT) approach)
2. Matching methods
3. Difference-in-Differences techniques (Diff-in-diff)
4. Roll-out approach and Panel analysis
5. Regression discontinuity design (RDD)
6. Event analysis approach
7. Instrumental variables approach (IV)

Question 6

What is the RCTs (Randomised Control Tests)

Answer 6

->Randomisation in constructing TREATMENT and CONTROL groups from an eligible population

Two random groups one placebo and one with medicine for instance

Question 7

How do you measure the impact of a RCT?

Answer 7

Avg treatment effect:
ATE = Y bar t - Y bar c

Y bar t = Avg outcome in Treatment group
Y bar c = Avg outcome in contrl group

Question 8

What are the potential behavioural problem with randomised experiments?

Answer 8

1. Hawthorne effect - modify their behavior simply because they are being observed
2. Contamination and Spill-over effects
3. John Henry effect:
   When control group behave different to make up for the disadv of being the control group
4. Dropout / Attrition effects - those not who dropped out
5. Pioneer / Partial equilibrium effect

Question 9

What are type 1 and type 2 errors in research (RCT)

Answer 9

Type I “false positives” – When the tester validates a statistically significant difference even though there isn’t one

Type II - false negative, also known as an error of omission

Question 10

In what situation may RCT not be applicable?

Answer 10

When the event has already occurred

Question 11

What is the Propensity Scoring Matching (PSM)?

Answer 11

Select Control (non-participants) group (C) with many observable essential characteristics as Treated group (T)

Question 12

What is the process of the PSM?

Answer 12

1. For each participant (i )
    Find non-participants (mi ) with a very close propensity score (prob of participation) based on several variables (X)
2. Impact of program is average difference in outcomes between treated i and his/her matched control.
   = Treatment on the Treated (ToT) = 𝑇oT = 1/𝑁𝑇∑𝑖∈𝑇( 𝑌𝑖 − 𝑌𝑚(𝑖))

Question 13

What are the assumptions the Difference-in-difference (DiD) approach?

Answer 13

Make crit assumption about occurence in absense of program

Question 14

What are the stages the DiD approach take to calculate the approach?

Answer 14

1. Change in outcome variable (Y) among participants over some period
   ∆𝑌𝑝 = Program Impact + Potential Role of other factors
2. Compute change in same outcome among non-participants over same period

∆𝑌𝑛𝑃 = Potential Role of other factors
∆𝑌𝑝− ∆𝑌𝑛p = Programme Impact

Question 15

What is the graph in the DiD display?

Answer 15

How the diff stages work

SEE GRAPH IN NOTES

Question 16

Why is Impact Evaluation useful for policymaking?

Answer 16

1. Importance of Internal validity and External validity in Impact Evaluation
2. Measures to avoid ‘data mining’ or ‘cherry picking’
   - Prevalence of ‘false positives’
   - Selective reporting
   - Pre-analyses plan (PAP) requirement: Official independent bodies exist for PAP issues.
3. Importance of Qualitative Methods