Threats and Analysis

Question 1

True treatment effect

Answer 1

True treatment effect is a concept that applies to each individual. So the impact of this program on me is X, the impact on you is Y. Just like the true counterfactual, the true treatment effect cannot be observed, we can only estimate it with a certain degree of confidence.

Question 2

Average Treatment Effect

Answer 2

ATE is also hypothetical, and assumes everyone is treated, but averages across me, you and everyone else in the sample (or population). This too, is impossible to measure if there is any non-compliance.

Question 3

Intention to Treat:

Answer 3

ITT is similar to ATE, however if we assume no spillovers from compliers to non-compliers in the treatment group

we can think of it as the treatment effect on compliers and spillover effect on non-compliers within the treatment group.

Question 4

Local Average Treatment Effect (LATE) or Complier Average Treatment Effect (CACE):

Answer 4

CACE or LATE basically just limits the sample to compliers and can be estimated (using the Wald estimator) if there are no spillovers; or alternatively, CACE could be estimated if we knew the spillover effect.

Question 5

Noncompliance

Answer 5

when some members of the treatment group don’t receive treatment and/or control group receive treatment, etc.

Question 6

Attrition

Answer 6

whether or not you have outcomes for your subjects; unable to find your subjects many years later, etc.
And/or the control group is less likely to feel like they need to engage in outcome measures (taking a test, survey, etc) because they didn’t receive the beneficial treatment

Question 7

Spillovers

Answer 7

to what extent to the treatments assigned to some people leak over to the controls nearby

Question 8

Generalizability

Answer 8

how can you translate your results to policy & behavioral recommendations within the limitations of your results?

Question 9

Unit level treatment effect & average treatment effect (ATE)

Answer 9

How would an experimental subject have responded if treated? How would same subject have responded if untreated? Difference between these two potential outcomes is the unit-level treatment effect;

Average unit-level treatment effect is the ATE (average treatment effect in the subject pool)

Question 10

Why is the average treatment effect important?

Answer 10

Randomization pulls a random sample of the treated and untreated potential outcomes. We can’t know the individual level causal effect, but we can estimate in an unbiased manner, under core assumptions, the average treatment effect

Question 11

Core assumptions of the potential outcomes model

Answer 11

Random assignment of subjects to treatments, non-interference, excludability

Question 12

Core assumptions - Random assignment of subjects to treatments

Answer 12

receiving treatment statistically independent of subjects’ potential outcomes

We must compare only randomly assigned groups & resist the temptation to compare the groups that actually take the treatment to the groups that don’t take the treatment - because they aren’t necessarily randomly assigned

Question 13

Core assumptions: non-interference

Answer 13

subject’s potential outcomes reflect only whether they receive the treatment themselves; they are unaffected by how treatments happened to be allocated/how the randomization pans out

Question 14

Core assumptions: excludability

Answer 14

subject’s potential outcomes respond only to defined treatment, not other extraneous factors that may be correlated with treatment – Importance of defining treatment precisely and maintaining symmetry between treatment and control groups (e.g., through blinding)

Must maintain symmetry in everything that you do in design and analysis - Enumerators have to stay the same for treatment and control, for example (otherwise could be a violation of symmetry)

Question 15

What 3 things are missing from the ‘core assumptions’ of the potential outcomes model?

Answer 15

No assumptions about shape of outcome distribution (e.g., that responses are normally distributed) - We’re not assuming that we can generalize our inference about the ATE in the subject pool

The issue of “external validity” is a separate question that relates to the issue of whether the results obtained from a given experiment apply to other subjects, treatments, contexts, and outcomes.

Random sampling of subjects from a larger population is not a core assumption, but aids generalizability

Question 16

When core assumptions are met…

Answer 16

…an experiment generates unbiased estimates of the average treatment effect (ATE)

Question 17

Sampling distribution

Answer 17

collection of possible ways that an experiment could have come out, under different random assignments

Question 18

estimator

Answer 18

procedure for generating guesses about a quantity of interest (e.g., the average treatment effect)

Question 19

Under simple or complete random assignment, the difference-in-means estimator is…

Answer 19

Under simple or complete random assignment, the difference-in-means estimator is unbiased – Any given estimate may be higher or lower than the true ATE, but on average, this procedure recovers the correct answer

Question 20

3 possible reasons for non-compliance

Answer 20

Sometimes there is a disjunction between the treatment that is assigned and the treatment that is received.

(1) Miscommunication and administrative mishaps - Send radio ads but sometimes they just don’t play them; things just don’t happen as you intend sometimes
(2) Subjects may be unreachable - People don’t answer/aren’t home/they moved/they died, etc.
(3) Encouragements sometimes don’t work - Can’t force people to do anything; Encouragement doesn’t push them to enroll. Or, people who are in control group show up anyways

Question 21

When addressing “noncompliance” - how should we think about the ‘excludability’ assumptions?

Answer 21

Are outcomes affected only by the treatment? Or by
both the assignment and the treatment?

Excludability is going to mean that the only thing that will affect people is the treatment itself, and not something other factor - this is a potentially fallible assumption. To the extent that you can design things in a way that doesn’t tip people off to their assignment is important – otherwise things could get distorted a bit.

Question 22

Can you switch non-compliers between treatment and control (depending on their behavior) and/or remove them from the sample?

Answer 22

No! This leads to bias! You ONLY want to compare the randomly assigned groups, do NOT change what you are comparing!

Question 23

What should estimation strategy be based on - in terms of treatment and control groups? What’s the risk of altering group structure?

Answer 23

Subjects you fail to treat are NOT part of the control group! Do not throw out subjects who fail to comply with their assigned treatment

Base your estimation strategy on the ORIGINAL treatment and control groups, which were randomly assigned and therefore have comparable potential outcomes. Groups/individuals should always be analyzed based on their original assignment.

Certain types of individuals may be more likely to take up the treatment than others; comparing only those who are actually treated (whether in the treatment group or overall) with those who aren’t would run the risk of introducing selection bias.

Question 24

Compliers

Answer 24

people who would receive the treatment if they were assigned to the treatment group & only if they are assigned to the treatment group (aka if you assign them to the control group, they don’t take the treatment)

Question 25

Never-takers

Answer 25

people who would never take the treatment regardless of their group assignment; impossible to learn about those people

Question 26

Your treatment group for analysis is…

Answer 26

individuals assigned to treatment, regardless of whether or not they were treated

Question 27

Steps for addressing one-sided noncompliance satistically

Answer 27

• Define “Compliers” and estimate the average treatment effect within this subgroup
• Model the expected treatment and control group means as weighted averages of sub-groups, “Compliers” and “Never-takers”
• Assume excludability: assignment to treatment only affects outcomes insofar as it affects receipt of the treatment

Question 28

Expected outcome in the control group (E0) is a weighted average of Complier and Never-taker outcomes in the control condition - equation

Answer 28

E0 = α Pc + (1 - α) Pn

• Let Pc = the probability that untreated Compliers express empathy for those ostracized because of abortion in the end-line survey
• Let Pn = the probability that untreated Never-takers
express empathy for those ostracized because of
abortion in the end-line survey
• Let a = the proportion of Compliers in the subject pool

Question 29

Expected outcome in the treatment group (E1) is also a

weighted average of Complier and Never-taker outcomes, this time under treatment - equation

Answer 29

E1 = α (Pc + T) + (1 - α) Pn

• Let Pc = the probability that untreated Compliers express empathy for those ostracized because of abortion in the end-line survey
• Let Pn = the probability that untreated Never-takers
express empathy for those ostracized because of
abortion in the end-line survey
• Let a = the proportion of Compliers in the subject pool
• Let T = the average treatment effect among Compliers

Question 30

“aT” is..

Answer 30

the “intent to treat” effect, or “ITT”

E1-E0 = aT

The intent to treat effect is the effect of assignment. Ignore whether or not people took your treatment -
What is the effect of the assignment?

Question 31

Formula to estimate “T” - the behavioral/treatment effect – complier average casual effect

Answer 31

To estimate T, insert sample values into the formula:
T* = (E1 – E0)/a*
where a* is the proportion of treated people (Compliers) observed in the assigned treatment group, and E1 and E0 are the observed average outcomes in the assigned treatment and control groups, respectively

This model will allow us to identify ATE among compliers, WITHOUT assuming that compliers & never-takers are similar in terms of what they do

Question 32

Complier Average Causal Effect (CACE)

Answer 32

the difference in averages for the treatment and control groups, divided by the difference in take-up proportions in the two groups.

Local average treatment effect (LATE) = CACE

Question 33

among Compliers, the CACE

Answer 33

= ITT

For the subject pool segment who are compliers, the CACE = ITT because they do what they’re told; there’s no difference between their assignment and their actual treatment

Question 34

among Never-takers, the ITT =

Answer 34

ITT = 0

The effect of assignment is 0 because no matter what they will never participate.

Question 35

Why is attrition a threat to an experiment?

Answer 35

Can present a grave threat to any experiment because
missing outcomes effectively “un-randomize” the
assignment of subjects

Question 36

Why is attrition a threat to an experiment?

Answer 36

Can present a grave threat to any experiment because missing outcomes effectively “un-randomize” the
assignment of subjects

Question 37

When looking at attrition, what two questions do you want to consider? What is the KEY factor we want to think about?

Answer 37

You want to consider the SYMMETRY between assigned experimental groups.

Are the rates of attrition the same in the treatment and control groups?

Do covariates predict missingness in the same way in both the treatment and control groups?

Question 38

Double sampling

Answer 38

an intensive effort to gather outcomes from a random sample of the missing

Solution to attrition

Question 39

Worst-case bounds

Answer 39

Solution to attrition
Imputation of “extreme values” on the ATE

So you give the highest possible outcome to the control group, the lowest possible outcome to the treatment group, to see the lowest possible effect and do the reverse to see the highest possible effect. This is problematic because if you have high rates of attrition, the ‘worst-case’ bounds will be too extreme & useless for policy implications

Question 40

“Trimming” bounds & “monotonicity” assumption

Answer 40

Solution to attrition
Goal is to bound the ATE among those who would always provide outcomes regardless of treatment
assignment

if you had differential rates of attrition in treatment and control, you could sort the group that has the extra subjects from highest to lowest, and lop off the extra subjects at the top of the distribution and/or at the bottom of the distribution.

a way to estimate the average treatment effect among a subset of people who would always report regardless of their treatment assignment.

Question 41

Missing outcomes measures vs. missing covariates

Answer 41

Missing outcomes - potential for bias
Missing covariates - since covariates are optional, it is not advisable to drop observations with missing covariates (these are not required)

In order to keep all subjects in the analysis, impute missing values for a covariate or in the context of regression analysis, insert an arbitrary value for the covariate & include a dummy variable that indicates missingness.

Question 42

Missing outcomes measures vs. missing covariates

Answer 42

Missing outcomes - potential for bias
Missing covariates - since covariates are optional, it is not advisable to drop observations with missing covariates (these are not required)

In order to keep all subjects in the analysis, impute missing values for a covariate or in the context of regression analysis, insert an arbitrary value for the covariate & include a dummy variable that indicates missingness.

Question 43

When might attrition be a problem?

Answer 43

Even if attrition rates are similar across treatment and control groups, the composition of attriters may be different in both groups. For instance, more women could drop out of one group and more men in the other? Bias introduced by systematic attrition might lead to a flawed estimate of 0 amongst those who remain in the study, or a flawed estimate of non-zero impact when the true impact is 0.

Question 44

Key questions to ask when considering ‘spillovers’?

Answer 44

• Are subjects’ potential outcomes a reflection ONLY of whether or not they personally receive treatment?
• Or could it be that subjects are affected as well by which other subjects receive treatment?

Question 45

Contagion

Answer 45

Spillovers

The effect of being vaccinated on one’s probability of contracting a disease depends on whether others have been vaccinated.

Question 46

Displacement

Answer 46

Spillovers

Police interventions designed to suppress crime in one
location may displace criminal activity to nearby locations.

Question 47

Communication (spillovers)

Answer 47

Spillovers

Interventions that convey information about commercial products, entertainment, or political causes may spread from individuals who receive the treatment to others who are nominally untreated.

Question 48

Social comparison

Answer 48

Spillovers

An intervention that offers housing assistance to a treatment group may change the way in which those in the control group evaluate their own housing conditions.

Question 49

Persistence and memory

Answer 49

Spillovers

Within-subjects experiments, in which outcomes for a given unit are tracked over time, may involve “carryover” or “anticipation.”

Question 50

How do spillovers complicate statistical analysis?

Answer 50

equal-probability random assignment of units does not imply equal-probability assignment of exposure to spillovers

Unweighted difference-in-means (or unweighted
regression) can give severely biased estimates

Question 51

Some suggestions on how to minimize spillovers?

Answer 51

• Unless you specifically aim to study spillover, displacement, or contagion, design your study to minimize interference between subjects. Segregate your subjects temporally or spatially so that the assignment or treatment of one subject has no effect on another subject’s potential outcomes.
• If you seek to estimate spillover effects, remember that you may need to use inverse probability weights to obtain consistent estimates on whether or not they could be exposed to treatment

Question 52

Four considerations regarding generalizability

Answer 52

Subjects, Treatments, Contexts, Outcomes

Question 53

Four considerations regarding generalizability

Answer 53

Subjects (just college students, e.g.), Treatments (just in a lab or in ‘the wild’), Contexts (unobtrusiveness; knowing vs. not knowing about study), Outcomes (measuring outcome the same day vs. at least one day later…do the effects last at least one day)

Question 54

When individuals assigned to the treatment group do not take up the intervention we should:

Answer 54

Leave them in the treatment group

Remember that in comparing treatment and control individuals, one should always make the comparison based on original treatment assignment to ensure that selection bias is not introduced. If those who take up the intervention systematically differ from those who do not take up the intervention (either in observable or unobservable ways), moving them to a different group, or dropping them from the study, would bias one’s estimate of impact. Individuals should always be compared based on their original treatment group assignment.

Question 55

When there is perfect compliance in both groups, the ITT and CACE…

Answer 55

The Intention to Treat (ITT) estimate and the Complier Average Causal Effect (CACE) estimate will be the same

When there is perfect compliance in both groups, take-up is 100% in the treatment group and 0% in the control group. Recall that the CACE is the ITT estimate divided by the difference in take-up proportions in the treatment and control groups. In this case, the difference is 1-0 = 1, thus the CACE = (ITT estimate)/1 = just the ITT estimate.

Question 56

When researchers include a placebo group as part of the evaluation design, they are typically motivated by which concern

Answer 56

Non-compliance

A placebo group can help one to generate estimates for compliers and for non-compliers separately

Question 57

Those who don’t take up the treatment in the control group can be

Answer 57

Never-takers and compliers

Question 58

Always takers

Answer 58

always takers always take up the treatment regardless of whether they are in the treatment or control group

Question 59

Defiers

Answer 59

Defiers choose to defy their assignment status by only taking up the treatment if they are in the control group