CHAPTER 11 Randomized Experiments Flashcards
What is the primary benefit of randomizing treatment in experiments?
Randomizing treatment can yield unbiased causal estimates.
What tools work in experimental settings for assessing genuine effects?
All the tools of statistical inference and hypothesis testing.
What complications can arise even in randomized experiments?
Numerous complications can arise and must be planned around.
What should comparisons be based on when experimental subjects fail to comply?
Comparisons should be based on randomized assignment.
What are ‘natural experiments’?
Instances where treatment of interest was randomized for non-research purposes.
What is a key idea about research designs mentioned in the introduction?
Using research designs to learn about causality often involves cleverness and creativity.
Why are randomized experiments considered effective for eliminating bias?
If you can randomize treatment, there are no confounders.
What did the World Health Organization state about breastfeeding?
Breastfeeding is associated with lower blood pressure, cholesterol, and better performance on intelligence tests.
What negative correlation is observed in developing countries regarding breastfeeding?
Breastfeeding is correlated with worse health outcomes for children.
What confounding factor is suggested to influence breastfeeding in developing countries?
Poverty may cause both breastfeeding and malnourishment.
What did a 1997 study in Peru find about breastfeeding and child size?
It found a negative correlation between breastfeeding and size.
What conclusion did the Peruvian study reach about breastfeeding?
Children who are not growing are more likely to be breastfed due to health issues.
What did researchers in Belarus attempt to achieve with their randomized experiment?
To create an apples-to-apples comparison of breastfeeding outcomes.
What was the outcome of the Belarus study regarding breastfeeding benefits?
They found scant evidence for large benefits of breastfeeding.
What was one significant finding from the Belarus study regarding IQ?
Children from hospitals with breastfeeding encouragement performed slightly better on IQ tests.
What does randomization guarantee in experiments?
Randomization guarantees that the bias is zero.
What do noise and random assignment contribute to in experiments?
Noise can come from sampling variability, measurement error, and random assignment.
How does increasing sample size affect noise in experiments?
As the sample size gets big, the noise becomes small.
What combination provides highly reliable causal estimates?
Randomization plus a large sample size.
What is the only way to guarantee unbiased estimates of causal relationships?
Randomization.
Fill in the blank: Randomized experiments can help to eliminate _______ from estimates.
[bias]
True or False: Randomized experiments ensure that confounders do not affect the treatment assignment.
True
What does randomization in experiments guarantee?
Unbiased estimates of causal relationships
Randomization ensures that the treated and untreated groups are comparable, reducing the risk of bias.
Why is a large sample size important in experiments?
It reduces noise, generating precise estimates
A large sample size helps ensure that the treated and untreated groups are similar with respect to other characteristics, which improves the validity of the estimates.
What is blocking or stratification in experimental design?
Grouping subjects based on observable similarities before random assignment
This technique helps reduce noise by ensuring that treated and untreated groups are similar in key characteristics.
What is a matched-pair design?
Identifying pairs of similar individuals and randomly assigning treatment within pairs
This design improves the precision of estimates by controlling for individual differences.
What is the standard error in the context of experimental estimates?
A measure of how far, on average, an estimate is from the true effect due to noise
Standard error provides insight into the reliability of the experimental estimate.
How does the size of the treated and untreated groups affect noise in experimental estimates?
Larger and roughly equal group sizes reduce noise and improve precision
When the groups are unequal, the estimates may be sensitive to the characteristics of a few subjects.
What happens when the outcome variable has little variance?
Estimates will be more precise
Low variance in outcomes means less potential for noise, leading to more reliable estimates.
What is the purpose of hypothesis testing in experiments?
To assess statistical significance of experimental results
Hypothesis testing allows researchers to evaluate whether observed effects are likely due to chance.
What is noncompliance in the context of experiments?
When subjects do not adhere to their assigned treatment
Noncompliance complicates the analysis because it can introduce bias if not properly addressed.
What is the intent-to-treat (ITT) effect?
The effect of being assigned to the treated group, regardless of compliance
ITT provides an unbiased estimate of the impact of the treatment assignment itself.
What can analysts do to deal with noncompliance?
Estimate the intent-to-treat effect
This approach helps in understanding the impact of treatment assignment while accounting for noncompliance.
What is one risk associated with hypothesis testing?
Over-comparing and under-reporting results
Analysts must be cautious to avoid drawing conclusions from multiple comparisons that could lead to false positives.
What is the relationship between sample size and standard error?
Larger sample size leads to smaller standard error
A larger number of subjects reduces the impact of random variation on the estimates.
What is the significance of R. A. Fisher’s work in the 1920s?
He emphasized the value of randomization in experiments
Fisher’s contributions laid the foundation for modern experimental design and analysis.
Fill in the blank: Randomization eliminates _______ in experiments.
bias
True or False: Randomization guarantees that treated and untreated groups will be identical in every characteristic.
False
Randomization ensures comparability in expectation but not in every specific instance.
What should analysts do before conducting experiments to avoid issues?
State questions of interest and pre-specify hypotheses
This helps prevent fishing for significant results and increases transparency.
What is the relevant quantity for cost-benefit analysis regarding Body Vibes?
The intent-to-treat effect
This is the average effect of being provided Body Vibes, regardless of individual usage.
What are the four types of people in the Body Vibes experiment?
- Compliers
- Always-takers
- Never-takers
- Defiers
Compliers wear Body Vibes if assigned to treatment, always-takers wear them regardless, never-takers do not wear them at all, and defiers wear them only when not assigned to treatment.
True or False: Compliers are subjects who will not wear Body Vibes if they are not assigned to treatment.
True
Compliers will only wear Body Vibes if assigned to the treatment group.
What is the goal of an experiment in terms of subject assignment?
To randomly assign treatment
The idea is to have compliers who are willing to adhere to the treatment assignment.
What does the term ‘intent-to-treat effect’ refer to?
The average outcome difference between the treated and untreated groups
It estimates the effect of being assigned to treatment, not necessarily the effect of actually taking the treatment.
Fill in the blank: The _______ effect is the effect of treatment for those who actually take the treatment.
[complier average treatment effect (CATE)]
What happens if we ignore treatment assignment when comparing Body Vibes wearers and non-wearers?
The comparison is not apples-to-apples
This is because the groups may contain different proportions of compliers and never-takers.
What is the relationship between the intent-to-treat effect and the proportion of compliers?
The intent-to-treat effect equals the average effect for compliers multiplied by the proportion of compliers
This allows us to estimate the average effect of treatment for compliers.
How can we estimate the proportion of compliers in our sample?
By calculating the proportion of the treated group that takes up the treatment
This proportion is an unbiased estimate for the whole sample due to random assignment.
What is the Wald Estimator?
It is the ratio of the intent-to-treat effect to the proportion of compliers
It is used to estimate the complier average treatment effect.
What are the four key conditions for instrumental variables (IV) analysis to work?
- Exogeneity
- Exclusion restriction
- Monotonicity
- Relevance
These conditions ensure that the instrument used provides unbiased estimates.
True or False: The exclusion restriction means that the instrument can influence the outcome through pathways other than the treatment.
False
The exclusion restriction states that all effects must occur through the treatment.
What is the first key condition for IV analysis to work?
Exogeneity: The instrument must be randomly assigned or be ‘as if’ randomly assigned.
What does the exclusion restriction state in IV analysis?
All of the reduced-form effect must occur through the treatment; there is no other pathway for the instrument to influence the outcome.
What does the term ‘compliers’ refer to in the context of IV analysis?
There must be some compliers who actually use the treatment they were assigned to.
What is the problem with defiers in IV analysis?
If there are defiers, the estimate will give a weighted average of the treatment effects for compliers and defiers, potentially leading to bias.
How did the Body Vibes experiment satisfy the condition of exogeneity?
The treatment was randomized.
What is a chance imbalance in experiments?
A situation where the treated and untreated groups differ due to random chance.
What are three potential responses to a chance imbalance in an experiment?
- Throw out the ‘broken’ experiment
- Proceed as normal
- Control for unbalanced variables
What does it mean to proceed as normal in the case of chance imbalance?
The experimental estimate is still unbiased, and the imbalance can be reported for transparency.
What is the trade-off when controlling for pre-treatment variables?
It can improve precision but may produce a biased estimate.
What is one way to avoid chance imbalance in experiments?
Use a blocked or stratified experimental design.
What does lack of statistical power mean in an experiment?
An experiment yields inconclusive results due to a large standard error.
What can be done if an experiment is underpowered?
Consider controlling for variables or accept that the experiment may not provide convincing answers.
What is attrition in the context of experiments?
When participants drop out of an experiment after treatment assignment.
How does random attrition affect the estimation of treatment effects?
It allows for an unbiased estimate, although statistical power is reduced.
What should an analyst do if there is nonrandom attrition?
Test if the treatment influenced the rate of attrition and assess differences in covariates.
What is one method to bound the extent of bias due to attrition?
Consider best-case and worst-case scenarios for the treatment’s effect based on attrition rates.
Fill in the blank: The exclusion restriction in IV analysis requires that all of the reduced-form effect must occur through the _______.
[treatment]
True or False: Underpowered experiments can still contribute valuable information.
True
What is the effect of Body Vibes on skin health in the treated group?
52.5 percent of subjects in the treated group would have good skin health compared to 47.5 percent in the untreated group.
This implies a positive effect of 5 percentage points.
What does interference refer to in the context of experiments?
Interference occurs when the treatment status of one unit affects the outcome of another unit.
This can bias the results of an experiment.
What was the purpose of Blattman’s randomized experiment in Liberia?
To evaluate the impact of cash grants and cognitive behavioral therapy on young men at high risk for engaging in crime or violence.
The study aimed to compare levels of crime and homelessness among treated and untreated groups.
What problem arose in Blattman’s pilot study regarding cash grants?
Participants formed an insurance agreement, sharing money among themselves, which led to interference.
This undermined the integrity of the untreated group.
What is the goal of a randomized experiment?
To randomly assign a treatment of interest across different units of observation.
This ensures that the treated and untreated groups have, in expectation, the same potential outcomes.
How do analysts deal with interference in studies?
Sometimes the interference itself becomes the object of investigation.
This can include studying spillover effects.
What is a natural experiment?
A situation where something is randomized not for research purposes, but analysts can still utilize the randomization to answer causal questions.
An example includes charter school admissions lotteries.
What is the intent-to-treat (ITT) effect?
The average effect on the outcome of being assigned to the treated rather than the untreated group.
ITT does not account for noncompliance.
What is the exclusion restriction in the context of instrumental variables?
An instrument satisfies the exclusion restriction if it affects the outcome only through its effect on the treatment.
It should not affect the outcome through any other channel.
What was the natural experiment related to military service and future earnings?
During the Vietnam War, draft-eligible men were randomly assigned draft numbers, affecting their military service.
This created random variation in military service that researchers could study.
What are always-takers in the context of noncompliance?
Units that are always treated, regardless of their assigned treatment status.
This contrasts with never-takers, who are never treated.
What is statistical power in a study?
The probability of rejecting the null hypothesis of no effect if the true effect is of a certain non-zero magnitude.
Low statistical power means it is unlikely to produce a statistically significant result even with a large effect.
What is attrition in experimental studies?
The situation where experimental subjects drop out of the experiment, leading to unobserved outcomes for those subjects.
Attrition differs from noncompliance.
What is the definition of noncompliance?
When an experimental subject chooses a treatment status other than the one to which it was assigned.
This can affect the validity of experimental results.
What is the first-stage effect?
The average effect of being assigned to the treated group on take-up of the treatment.
This corresponds to the fraction of compliers.
What are defiers in the context of treatment assignment?
Units that take up the opposite of the treatment status they are assigned.
This can complicate the analysis of treatment effects.
What is blocked/stratified random assignment?
Dividing experimental subjects into groups and then randomizing treatment within each group.
This can improve the precision of estimates.
What is the impact of chance imbalance in experiments?
The treated and untreated groups may differ in important ways despite random assignment due to noise.
This can lead to biased estimates of treatment effects.
What is attrition in the context of experiments?
The loss of subjects from a study that affects the outcomes observed.
What is the definition of noncompliance?
When subjects do not adhere to the assigned treatment in an experiment.
Define interference in experimental design.
The situation where the treatment status of one unit affects the outcome of another unit.
What is a natural experiment?
When something was randomized not for research purposes, but analysts use this randomization to answer a causal question.
In the behavioral priming experiment, what was the treated group exposed to?
Words related to aging and old age.
What is a potential problem if the treated group over-represents older and less athletic individuals?
It may introduce bias in the results due to differing characteristics.
What issue arises from subjects not paying close attention to the word association activity?
It can lead to meaningless answers affecting the validity of the experiment.
What impact does a subject stopping to talk or check their phone have on the experiment?
It can skew the timing results and introduce variability unrelated to the treatment.
What happens if subjects never cross the hallway?
Their data is missing, which can lead to attrition bias.
Why is it a concern that research assistants knew the hypothesis?
They may unintentionally influence the outcomes based on their expectations.
What effect might subjects discussing the task have on the experiment?
It could lead to contamination of the treatment effect among subjects.
What is the goal of the GOTV experiment?
To estimate the effects of get-out-the-vote interventions on voter turnout.
What complicates the analysis of the GOTV experiment?
Variability in treatment assignment probability, non-receipt of treatment, and missing turnout data.
What should you consider when calculating the mean turnout for treated vs. untreated groups?
Potential biases arising from treatment assignment differences.
How do urban and non-urban treatment assignment probabilities affect results?
They can lead to biased estimates of treatment effects if not accounted for.
What is intent-to-treat effect?
The effect of assigning treatment regardless of whether the treatment was received.
What is the compliance rate in experimental design?
The proportion of subjects who adhere to their assigned treatment.
What assumption is made when dropping subjects for whom turnout is not observed?
It assumes that their outcomes would not differ from those observed.
What is the worst-case scenario estimate for the effectiveness of GOTV?
Assuming none of the attrited subjects would have voted.
What is a recommended resource for conducting experiments?
Field Experiments: Design, Analysis, and Interpretation by Alan S. Gerber and Donald P. Green.
What does the study by Marquis et al. (1997) illustrate?
The association between breastfeeding and stunting in Peruvian toddlers.
What is the focus of the Kramer et al. (2002) study?
The relationship between breastfeeding and infant growth.
What does the Angrist (1990) study analyze?
Lifetime earnings and the Vietnam Era Draft Lottery.
What did Angrist and Chen (2011) investigate?
The effects of the Vietnam-era GI Bill on schooling.
True or False: The influence of behavioral priming on walking speed is unaffected by who conducts the timing.
False.
Fill in the blank: Interference occurs when the treatment status of one unit affects the _______ of another unit.
[outcome]
