CHAPTER 13 Difference-in-Differences Designs Flashcards
What is a difference-in-differences design?
A research design that estimates causal effects by comparing changes in outcomes over time between units that experience a treatment and those that do not
What does a difference-in-differences design control for?
It controls for all confounders that don’t vary over time, even if they can’t be observed or measured
What is the basic idea behind difference-in-differences?
To compare the change in outcomes for units that changed their treatment status to those that did not
What is the parallel trends requirement?
The assumption that the trends in outcomes would have been the same across units in the absence of any treatment change
What do we compare in a difference-in-differences design?
The difference in outcomes over time for treated units versus untreated units
What is the average treatment effect on the treated (ATT)?
The average effect of a treatment on those units that actually received the treatment
True or False: Difference-in-differences estimates the average treatment effect (ATE).
False. It estimates the ATT.
What is the main conclusion of the Card and Krueger study?
A higher minimum wage appears to increase employment by 2.75 FTE per restaurant
What does the noise in difference-in-differences accounts for?
Random variation in the data that can affect the estimates
What is the first step in conducting a difference-in-differences analysis?
Identify the treatment and control groups and the time periods before and after the treatment
Fill in the blank: The difference-in-differences method can be used as a ______ to evaluate causal claims.
[gut check]
What is the significance of the January 1992 and November 1992 data in the Card and Krueger study?
They provide the employment data before and after the minimum wage increase in New Jersey
What are the two groups compared in a difference-in-differences design?
- Treated group (changed treatment status)
- Untreated group (remained untreated)
What is the key factor that validates the difference-in-differences approach?
The parallel trends condition must hold
What does the difference-in-differences estimate account for?
Systematic differences between states and over-time trends
True or False: The difference-in-differences design can estimate causal effects without randomization.
True
What is the potential bias in comparing employment levels between New Jersey and Pennsylvania after the minimum wage increase?
Differences between the states other than the minimum wage could confound the results
Fill in the blank: The difference-in-differences approach uses data from ______ to eliminate bias.
[untreated units]
What does the term ‘estimand’ refer to in the context of difference-in-differences?
The actual causal effect we are trying to estimate
How does the difference-in-differences method help in observational studies?
It helps to control for unobserved confounders that do not change over time
What was the employment change observed in New Jersey from January to November 1992?
An increase of 0.59 employees per restaurant
What is the difference-in-differences approach?
A method to account for systematic differences between states over time trends without directly observing those differences.
What is the parallel trends condition in difference-in-differences?
The assumption that the over-time trend in outcomes would have been the same across units but for the change in treatment status.
True or False: The difference-in-differences approach assumes that all confounders are controlled.
False.
What does the difference-in-differences method account for?
- Differences between units that don’t vary over time
- Time-specific factors affecting outcomes
What is a limitation of the difference-in-differences approach?
It does not account for time-varying differences between units that correspond with the treatment.
In the context of difference-in-differences, what does ‘N Units and Two Periods’ refer to?
Analyzing changes for multiple units over two time periods where some units receive treatment while others do not.
What are the three options for calculating difference-in-differences with N units and two periods?
- By hand
- First differences
- Fixed effects regression
What does fixed effects regression include?
Dummy variables for each unit and time period to control for average differences.
What was the main research question of Gentzkow and Shapiro?
How watching television as a pre-schooler affects future academic performance.
What major study did Gentzkow and Shapiro use for their analysis?
The Coleman Study.
Fill in the blank: The towns in group A first got TV in ______.
1953
What is a confounder in the context of comparing test scores between towns?
Any systematic difference between towns that affects academic performance.
What does the term ‘noise’ refer to in the context of test score comparisons?
Random variation that can affect the results of the comparisons.
What is the significance of subtracting the difference between 6th graders in two town types?
It estimates cross-town differences without the effect of TV exposure.
How does the difference-in-differences calculation help in estimating the effect of TV?
It accounts for systematic differences between towns and over time.
What happens when you have more than two periods with changing treatment?
You can use first differences or fixed effects, but they may yield different results.
What is a key consideration when using first differences?
You are looking for immediate effects after the treatment status changes.
What is an advantage of using fixed effects regression?
It allows including additional time-varying control variables.
True or False: The timing of the effect is irrelevant in difference-in-differences analysis.
False.
What key intuition does the difference-in-differences design provide?
It allows checking whether changes in treatment correlate with changes in outcomes.
What is the main purpose of difference-in-differences designs?
To check whether changes in treatment are correlated with changes in the outcome.
What does a significant correlation between treatment and outcome suggest?
It might indicate a causal relationship.
What is a confounding variable?
A factor that may distort the perceived relationship between treatment and outcome.
Who conducted a key study on the impact of oral contraceptives on women’s lives?
Claudia Goldin and Lawrence Katz.
What created a natural experiment regarding oral contraceptives in the U.S.?
State policies that varied the legal availability of oral contraceptives.
What was the significance of the timing of state policies on contraceptive access?
Different states removed restrictions at different times, aiding causal inference.
What did Martha Bailey’s study estimate?
The effect of early access to oral contraceptives on women’s childbearing and labor force participation.
What was the estimated reduction in likelihood of becoming a mother before age 22 due to early access to contraceptives?
14 to 18 percent.
How much did early access to oral contraceptives increase women’s participation in the paid labor force by their late twenties?
8 percent.
What is essential for difference-in-differences to yield an unbiased estimate?
Parallel trends.
What are pre-trends in the context of difference-in-differences?
Trends in outcomes before any units changed treatment status.
What is the implication if the coefficient on the lead treatment variable is zero?
It supports the assumption of parallel trends.
What study did Jonathan Ladd and Gabriel Lenz conduct?
They studied the effect of newspaper endorsements on voting decisions.
What event allowed Ladd and Lenz to use a difference-in-differences design?
Newspapers unexpectedly switched endorsements during the 1997 UK general election.
What did Christakis and Fowler’s research claim about obesity?
Obesity is contagious.
What percentage increase in the chance of becoming obese is associated with having a friend who becomes obese?
57 percent.
What is homophily in the context of social networks?
The tendency for similar individuals to associate with one another.
What is a critical question when analyzing network effects?
Are the groups being compared similar in characteristics other than their friends’ behaviors?
What does the difference-in-differences design require regarding treatment changes?
That changes in treatment are not correlated with other factors affecting the outcome.
What is the role of diagnostic tests in difference-in-differences analysis?
To probe the plausibility of parallel trends.
What does it mean if the trends in outcomes are not parallel?
The estimates may be biased.
What is one way to check for parallel trends?
Compare trends in outcomes for units that do and do not change treatment status.
What is necessary for a valid comparison in social network studies?
Comparing individuals from similar backgrounds, excluding the influence of their friends’ behaviors
This means that members of networks being compared must have similar characteristics except for their smoking status.
What is homophily in the context of social networks?
The tendency for individuals to associate with similar others
In this case, smokers may be more likely to be friends with other smokers.
How might being a smoker influence one’s social network?
Smokers may form friendships in smoker-friendly environments like bars or designated smoking areas
This suggests that social networks may be influenced by pre-existing smoking habits rather than smoking being caused by friendships.
What does the difference-in-differences design account for?
Fixed characteristics of units, such as the likelihood of obesity among friends
This design helps to isolate the effect of variables by comparing changes over time within the same units.
What is a potential concern with the difference-in-differences design?
Violations of parallel trends, which could bias causal estimates
Parallel trends refer to the assumption that the treatment and control groups would have followed the same trajectory over time in the absence of treatment.
What did Cohen-Cole and Fletcher’s study on height and acne suggest?
The apparent social contagion of height and acne is likely due to homophily rather than direct social influence
This study indicates that individuals with similar characteristics tend to associate with each other.
What is the main idea behind the democratic peace theory?
Democracies are less likely to go to war with each other compared to autocracies
This theory suggests that shared norms and domestic political constraints play a role in preventing conflict between democratic nations.
What is a dyad in the context of the democratic peace literature?
A pair of countries being studied for their democratic and war interactions
The relationship between democracy and war is often analyzed using dyad-year observations.
What is the Polity score?
A standard measure of the level of democracy in a country
Higher Polity scores indicate more democratic governance.
What does the minimal level of democracy in a dyad measure?
The lower of the two Polity scores within a dyad
This variable helps assess the democratic nature of both countries in a dyad.
What was the outcome of the regression analysis on democratic dyads and war?
A statistically significant negative correlation between democratic dyads and the occurrence of war
This suggests that as the level of democracy increases in a dyad, the likelihood of war decreases.
What happens to the correlation between democracy and war when controlling for additional variables?
The correlation remains negative and statistically significant, but its strength diminishes
This indicates that controlling for confounding variables does not fully explain the relationship.
What is a critique of the use of difference-in-differences in international relations studies?
It may ignore most variation in the treatment variable, making it harder to detect relationships
Critics argue that this limitation reduces the robustness of findings in causal analysis.
What did the difference-in-differences analysis reveal about the relationship between democracy and war?
The correlation disappeared, suggesting a lack of evidence for a causal effect
This finding indicates that the previously observed correlation may be influenced by unobserved confounders.
What can be inferred if the relationship between treatment and outcome disappears in a difference-in-differences analysis?
It supports skepticism about the causal interpretation of the original correlation
This suggests that the observed correlation may be due to differences other than the treatment.
What is the significance of using fixed effects in difference-in-differences designs?
It helps control for unobserved heterogeneity across units and over time
This approach isolates the effect of the treatment variable by accounting for constant differences.
What is difference-in-differences?
A research design for estimating causal effects when some units change treatment status over time but others do not.
This method helps to control for unobserved factors that could affect the treatment and outcome.
What does the parallel trends condition state?
Average potential outcomes without treatment follow the same trend in units that do and do not change treatment status.
If this condition does not hold, difference-in-differences does not provide an unbiased estimate of the Average Treatment effect on the Treated (ATT).
What are first differences?
A statistical procedure for implementing difference-in-differences that involves regressing the change in outcome for each unit on the change in treatment for each unit.
This method helps isolate the effect of the treatment by looking at changes over time.
What is wide format in data structuring?
A way to structure a data set where each unit is observed multiple times, with each row corresponding to a unique unit.
This format is useful for certain types of analyses but can complicate others.
What is long format in data structuring?
A way to structure a data set where each unit is observed multiple times, with a row for each unit in each time period.
This format is often preferred for time series analyses.
What is fixed effects regression?
A statistical procedure for implementing difference-in-differences that regresses the outcome on the treatment while including dummy variables for each time period and each unit.
This method controls for unobserved heterogeneity that is constant over time.
What are pre-trends?
The trend in average outcomes before any unit changes treatment status.
If pre-trends are not parallel, it complicates the justification for the parallel trends condition.
What is a lead treatment variable?
A dummy variable indicating that treatment status in a unit will change in the next time period.
This can be useful for anticipating effects before they occur.
True or False: Imprecision accounts for the disparate results obtained by difference-in-differences and other approaches.
False.
The estimates from difference-in-differences are statistically significantly different from those obtained by other methods.
What is the significance of the Illinois State Aptitude Test (ISAT) example?
It illustrates how to analyze the average effect of high-stakes testing on student test scores using a difference-in-differences design.
The example compares cohorts before and after the change in stakes.
What concerns arise when estimating the effect of the Nike Vaporfly shoe?
Variability in marathon times due to day-to-day and course-to-course differences, as well as individual runner abilities.
These factors must be controlled for to isolate the effect of the shoe technology.
What is the purpose of controlling for confounders when estimating policy divergence?
To obtain a more accurate estimate of how differently Democratic and Republican candidates represent the same constituents.
This helps to remove bias from the analysis.
What is the challenge of attrition in studies like marathon running?
Not everyone who starts a marathon finishes, which can bias the results of the study.
Strategies must be implemented to address this potential problem.
What is the causal effect of blind auditions in orchestras according to Goldin and Rouse?
They study how making auditions blind impacts the hiring of female musicians.
The implementation timing varies across orchestras, allowing for causal analysis.
What is the concern with pooling data for regression analysis in the context of orchestra auditions?
It may not provide a causal interpretation due to confounding factors.
The timing of adopting blind auditions can affect the outcomes.
What assumptions must hold true for a difference-in-differences design to give an unbiased estimate?
The parallel trends condition must be met, indicating that units would have followed the same trend without treatment changes.
Other external factors must not influence the outcomes during the treatment period.
What types of analyses can be used to estimate policy divergence?
- Regression of roll-call voting on an indicator for being a Republican
- Regression discontinuity design
- Difference-in-differences design
Each method has its own assumptions and strengths.
What is the goal of estimating causal relationships in research?
To understand not only whether a treatment has an effect but also why that effect occurs.
This often involves complex analyses and clear thinking.
