Lecture 3 - Potential Threats in Randomized Evaluation

Question 1

What are the 5 most common issues related to data collection?

Answer 1

1. Data available?
2. Survey teams
3. Training
4. Local cooperation
5. Ethics

Question 2

What is meant by “Partial Compliance”?

Answer 2

1. Some participants do not receive the treatment they were assigned too
2. Some non-participants (control) do receive the treatment
3. Role of implementation staff
4. Problem of defiers: react in opposite way

Question 3

What is the “Wald estimator” and what does it measure?

Answer 3

effect on compliers= { (mean_t-mean_c) / (take up in t - take up in c) }

t = treatment
c = control
take up = How many people are left from the group that I started with.

Wald estimator: Measures the treatment by including partial compliance.

Question 4

What is the measure of the “Wald estimator” relevant for?

Answer 4

It’s relevant for encouragement design

Question 5

What is meant by “Attrition”?

Answer 5

Occurs when outcomes cannot be measured for some participants in the study. It is a problem of missing data

due to:

• drop-out
• death
• migration
• Still participate but cannot be found
• Refuse to answer

or they lose a characteristic –> mess up the design –> similarity is in danger of getting lost

Question 6

How can we check if the characteristics, of those who haven’t dropped of the program, are the same as when they started it?

Answer 6

To check the characteristics of those who haven’t dropped of and stayed we can regress their background information against what it was before. If we have this problem in the assignment, do this with a probit-analysis

Question 7

What should we do when we encounter “Attrition”?

Answer 7

1) The more you know about the units who dropped out of the program the more you can analyze the effect of their loss on the treatment results.
2) Analyse attrition with a probit estimation: can you explain dropout based on baseline characteristics? If so, your drop-out is not random
3) Model the potential effect on the drop-outs: what would have happened to those who dropped out? Use baseline data, compare dropouts, impute outcomes.

Question 8

What is meant by “Spillovers”?

Answer 8

This is when a character or group has an effect on other groups.

Question 9

How do we know when we have a Spillover-effect and what should we do when we encounter “Spillovers”?

Answer 9

How do we know if we have a spillover effect? Start to interview the observations or make a survey to find out.

Find out if there are:

• Distinguish between, T, C without Spillover, and control with spillover.
• Treatment effect = treatment of the treated + spillover effect on the control

Question 10

What are “Evaluation-driven effects”?

Answer 10

These are effects caused by our intervention. Our intervention can cause incentives to a certain behaviour for instance.

Question 11

What can we do about “Evaluation-driven effects”?

Answer 11

1. Limit it
2. Different level of randomisation
3. Do not announce
4. Impartial staff
5. Same level of interaction

Question 12

When does the evaluation of the project have to take place, before or after?

Answer 12

BEFORE