DiD, Quasi Flashcards
Threats to Internal Validity of Experiments
1) FAILURE TO RANDOMIZE
- treatment not randomly assigned, based on part of characteristics or preferences
- ethnic difference last name
- if you use vouchers
* Can test for if control variables coefficients W are 0 or not. If Random, X will be uncorrelated with W.
2) Failure to follow treatment protocol / partial complience
3) Attrition
4) Experimental effects / Hawthorne
5) Small Sample Sizes
- small sample does not necessary bias estimator of causal effect
- raises threat to validity of conf intervalls and hypothesis test
What are the threats to Internal Validity for Idealized Experiments?
- Failure to Randomize
- Failure to Follow the Treatment protocol / Partial Compliance
- People in the experiment does not do what they are told. Called partial compliance with the treatment protocol. Some controls get treatment, some “treated” get controls. This failure leads to bias in the OLS estimator.
- 10% of students switched groups because of behavior problems - Attrition
- Some subjects drop out. If the reason for attribution is related to the treatment itself, then the attrition can result in bias in the OLS
- Students move out of district
- Students leave for other schools
- Experimental effects
Subjects behaviour might be affected by being in a experiment. (Hawthorne Effect) - Small Sample Sizes
Experiments on humans might be expensive. This can result in a small sample. A small sample means that the causal effect is estimated imprecisely and raises threats to the validility of confidence intervals and hypothesis tests.
What are the threats to External Validity for Idealized Experiments?
- Nonrepresentative Sample
- The population studied and the population of interest might differ
- sample only includes people with one type of characteristics - Nonrepresentative program or policy
- policy or program must be similar to program studied to give generalizing results
- exp might be small-scale, might differ from real world - General Equilibrium effects
- Turning a small, temporary exp intro a widespread, permanent program
- sometimes only works with small groups
- Ac training Zimbabwe, 10 villages 40% increase wages. Nationwide: Different effect, become skilled, decrease in wage gains
What is Quasi-Experiments / Natural Experiments?
Two types of Quasi-Experiments:
- Whether an individual (entity) receives treatment is “as if” randomly assigned, possible conditional on certain characteristics
“Treatment (d) “as if” randomly assigned
• For example a new policy measure that is implemented in one but not in another are, whereby the implementation is “as if” randomly assigned.
- Does immigration reduce wages? Eco theory suggest that if the supply of labor increases, wages will fall. However immigrants tend to go to cities with high labor demand, so the OLS estimator of the effect on wages of immigration will be biased. Was done a Quasi on Cubans that moved to Miami. Estimated the causal effect on wages of an increase in immigration by comparing the change in wages of low-skilled workers in Miami to the change in wages of similar workers in comparable U.S cities. He found no effect.
2. Whether an individual receives treatment partially determined by another variable that is “as if” randomly assigned
“A variable (z) that influences treatment (d) is “as if” randomly assigned: use IV regressions
• The variable that is “as if” randomly assigned can then be used as an instrument variable in a 2SLS regression analysis.
What is the Difference-in-Difference estimator?
Change in Y = Y (after) – Y (before)
Change Y = a + B1*G + u
Treatment group G turns 1 for treatment group and 0 for control group. B1 is the DID-estimator.
PANEL DATA FORMULATION:
y = a + B1DG + B2D + B3G
D – Turns 1 after treatment and 0 before
G – Turns 1 for treatment group and 0 for control group
D * G – interaction term. Effect of being in the treatment group after treatment was received
B1 – DID-estimator
What is the parallel trend assumption?
Parallel trend assumption requires that in the absence of treatment, the difference between the treatment and control group is the same.
We cannot test for this, but if the treatment and control firms seem similar before the treatment, this is more likely to be the case.
Threats to internal validity during ideal randomized experiments
Failure to randomize Failure to follow the treatment protocol Attrition Hawthorne effect Small samples
Conditional Mean Independence
Conditional Mean Independence:
E[ui | Xi , Wi] = E[ui | Wi] != 0
This is unproblematic as long as we are only interested in the causal effect of Xi and not in the causal effect of Wi:
Under Conditional Mean Independence, OLS will give an unbiased estimate of the causal effect of Xi
The treatment might not be assigned randomly but instead is based on characteristics or preferences of the subjects
If this is due to the fact that the experimenter assigned the treatment randomly conditional on observed characteristics…
The treatment might not be assigned randomly but instead is based on characteristics or preferences of the subjects
If this is due to the fact that the experimenter assigned the treatment randomly conditional on observed characteristics…
…we can estimate the causal effect by including these observed characteristics in the regression (conditional mean independence)
If the treatment is randomly assigned conditional on unobserved characteristics or preferences…
If the treatment is randomly assigned conditional on unobserved characteristics or preferences…
…the estimated treatment “effect” will reflect both the effect of the treatment and the effect of these unobserved characteristics.
Double-blind experiment
Subjects and experimenters know that they are in an experiment, but neither know which subjects are in the treatment group and which in the control group.
How does true experimental research methods differ from correlational studies?
Experimental methods test for the presence of cause and effect.
Other methods generally seek to reveal relationships between variables.
After an experiment we can conclude more than a relationship, one variable may directly affect another
What is the key difference between Quasi Experiments and True Experiments?
Random assignment of participants is not possible in quasi-experimental methods, and useful when full experimental procedures are not possible for ethical practical or logical reasons.
If random assignment cannot be undertaken and there IS a control group - what type of design is this?
Quasi-experimental
Quasi-experiments
A study similar to an experiment except that the researchers do not have full experimental control (e.g., they may not be able to randomly assign participants to IV conditions).
