How to Randomize

Question 1

Random sample

Answer 1

Target population is not statistically different than the entire population

Question 2

Random assignment

Answer 2

Randomly assigns a group to treatment and control;

Each individual group remains statistically consistent with the entire population’s characteristics

Question 3

Simple randomization

Answer 3

Fixed probability, typically 50-50 would maximize your power

May end up with slightly more in one group & fewr in the other - but with a large enough sample, it’s not a problem (law of large numbers)

Spot randomization

Question 4

Complete randomization

Answer 4

Fixed proportion
Determine the number of individuals in treatment & control (sample frame) and then pull out of a hat/random number generator

Question 5

Why might you want to have the unit of randomization at the school vs. the student level?

Answer 5

1) Might be impossible to have half a school do something & the other half not
2) Risk of contamination/co-mingling, so those in control could benefit from the treatment

Question 6

Multiple treatment arms

Answer 6

A more cost-effective (usually) way to answer multiple questions in the same study; but the sample size has to be large enough

Question 7

Cross-cutting treatments (factorial design)

Answer 7

Involves interaction of 2 or more treatments; requires at least 2 treatments & four groups (1 group per treatment, 1 interaction arm, and 1 control group)

Question 8

Varying the intensity of treatment can help to measure….

Answer 8

Dosage, sensitivity, elasticity, spillovers

Question 9

Give a reason why it might be unethical to conduct a randomized evaluation

Answer 9

We already know an intervention works - so doing a study would needlessly consume resources while also denying people the intervention

Question 10

Why might service providers having trouble distinguishing between treatment and control be an issue? What is a possible solution?

Answer 10

Creates crossovers, no longer a pure counterfactual

Solution: Assign to different service providers; randomly assign to teams - this way a caseworker is trained to treat (or not), doesn’t have to decide on treatment

Solution: Change the unit of randomization - randomize at the cluster level and have all providers in a cluster provide treatment (or not)

Question 11

Why is it a challenge if (a) the control group finds out about treatment? Or if (b) the control group benefits from treatment? Or if (c) the control group is harmed by treatment? (Or if treatment is harmed) What is a possible solution?

Answer 11

(a) If control finds out about treatment, they might get upset…creating a lot of attrition.
(b) If the treatment group knows the control group - they could help them out & contaminate the control group
(c) If treatment and control are competing (for jobs, e.g.) the control could be harmed and the differential between the groups will be larger.

Solution: Vary the unit of randomization to limit spillovers or create a buffer

Question 12

Why is it a challenge if you have the resources to treat everyone? Possible solution?

Answer 12

If you had no capacity constraints, you could treat everyone but have no control group.

Solution: Phase-in treatment

Question 13

What are some of the concerns with a randomized phase-in design?

Answer 13

with a phase-in design, long-term outcomes can be harder to measure impacts on since the control group receives the program in the long run, doing away with the comparison group. This is of especial concern when the phase-in occurs on too short of a timeline for impacts to materialize and be measured.

Question 14

We may want to choose randomization on the bubble when

Answer 14

the eligibility criteria are well defined, but can be changed

If all applicants are equally eligible, there is no marginal set of applicants amongst whom the randomization can be carried out (i.e. there is no “bubble”.) If there are no clear eligibility criteria, the bubble becomes hard to define. However, even when there are strict eligibility criteria, an inability to change these criteria would render a randomized evaluation infeasible since there would be no additional applicants amongst whom the treatment can be randomized (i.e. all applicants would either be eligible or ineligible according to existing criteria - one may want to consider a regression discontinuity design in this situation.)

Question 15

Why is it a concern when everyone is eligible for a program? What is the solution?

Answer 15

Sometimes we are unable to randomly assign access to a program itself, but we can randomly assign encouragement to take up the program. This approach is useful when we want to evaluate a program that is already open to all of the eligible recipients but only some are currently using it. The program continues to be open to all the eligible people, but only some of them (the treatment group) will receive extra encouragement to take up the program

Here, you measure the effect of encouragement vs. the program

Question 16

Why is it a concern if the sample size is too small? What is the solution?

Answer 16

Problem - insufficient power
Solution - Change the unit of randomization or stratification (In stratified random assignment, we first divide the pool of eligible units into strata and then within each stratum follow the procedure for simple random assignment.)

Question 17

Basic Lottery:

Most useful when…
Advantages…
Disadvantages…

Answer 17

Most useful when the program is oversubscribed & not enough resources.

Advantages: familiar, easy to understand, easy to implement, can be implemented in public

Disadvantages: control group may not cooperate, differential attrition

Question 18

Phase-In:

Most useful when…
Advantages…
Disadvantages….

Answer 18

Most useful when expanding over time & everyone must receive treatment eventually

Advantages: Easy to understand, constraint is easy to explain, control group complies because they expect to benefit later

Disadvantages: Anticipation of treatment might impact short-term behavior, difficult to measure long-term impact

Question 19

Rotation:

Most useful when…
Advantages…
Disadvantages….

Answer 19

Most useful when everyone must receive treatment at some point, but not enough resources to treat everyone at once

Advantages: more data points than in phase-in
Disadvantages: difficult to measure long-term impact

Question 20

Encouragement:

Most useful when…
Advantages…
Disadvantages….

Answer 20

Most useful when program has to be open to all comers, and when take-up is low but can be easily improved with a small incentive

Advantages: can randomize at the individual level even when the program is not administered at that level

Disadvantages: Measures impact of those who respond to the incentive; need large enough inducement to improve take-up; encouragement itself might have direct impact

Question 21

level of randomization

Answer 21

the level of observation (ex. individual, household, school, village) at which treatment and comparison groups are randomly assigned.

Question 22

level of randomization

Answer 22

the level of observation (ex. individual, household, school, village) at which treatment and comparison groups are randomly assigned.

Question 23

The concern with pulling names out of a hat is that

Answer 23

its impossible to replicate the same randomization later on.

Balance can be ensured along certain characteristics by using different hats for each group. For instance, if we wanted to achieve balance on gender, we could use different hats for male and female names. We can perfectly dictate the size of our treatment (and thereby the control) group by pulling names out of a hat. For example, if there were 124 names in the hat, we could pull out exactly 62 names and achieve equal sized treatment groups. One of the virtues of randomizing by pulling names out of a hat is that is it provides a transparent process (since people can see the randomization take place in front of them.) However, the process of pulling names out of a hat can never be exactly replicated at a future date, making the randomization process inherently unable to replicate.

Question 24

If we have several causal hypotheses we’d like to test with different interventions

Answer 24

It is possible to test all within one experiment and if we have sufficient sample size, advisable

A single control group can be used as the counterfactual for all treatment arms within one experiement. However, it is important to pay careful attention to the sample size required and to ensure that one has a sufficiently large sample to test each of the hypotheses that one is interested in. The sample size that’s relevant for any comparison is that of the two groups that are being compared for each hypothesis. For a given, fixed sample, the more treatment arms we have, the smaller the sample size of each treatment arm.

Question 25

Rather than just comparing two treatments, or comparing each treatment to the control in a separate experiment, we may want a cross-cutting treatment (factorial design) because

Answer 25

We want to know whether a combination of treatments is necessary, even if each individual treatment is insufficient by itself

If one were merely interested in which of two unique treatments works, one could compare them using a standard A-B test with one treatment arm each. A fatorial design may or may not require a control group depending on our question. A factorial design allows one to see how a combined treatment compares to either of the two treatments by themselves, or to a control that receives no treatment at all. We can also simultaneously see the impacts of each of the separate treatments as well as that of the combined/interaction treatment.

Question 26

Examples of research questions that could be answered by varying or randomizing the intensity of treatment at the individual level (i.e. how much of a treatment each individual receives) include: (Select all that apply)

What is the demand curve for a product?
What proportion of a population must be treated for there to be herd immunity?
What is the minimum dosage necessary to treat an ailment?
Are there negative spillovers from the program on non-treated people in treatment communities?

Answer 26

All but the “demand curve for a product” question