Lecture 4 - Experimental vs Observational research Flashcards
Causal Inference
Researchers are often interested in explanatory (causal) questions.
Does social media exposure increase affective polarization?
Does economic wealth shape people’s policy preferences, e.g., on wealth or inheritance tax?
Does democracy spur/hinder economic development?
Does X cause/explain Y
A high level of education increases the probability that an individual participates in elections
Probalisitic or Deterministic
Prob
Highly educated individuals always participate in elections
Probabilistic or Deterministic
Determ
What is the problem with causal inference
- Tricky
- Cannot directly observe causal effects
- Cannot observe counterfactuals
Causal inference = inferring something we do not know (causal effects) from something we do know (data)
Counfounder
- Correlation does not equal causation
- There may be a correlation between X and Y, However how do we know that Z hasnt caused both of these.
Three requirements for establishing causality
- Association between X and Y
- All confounders ruled out
- Reverse causality ruled out
Does economic development cause democracy OR does democracy cause economic development?
Reverse Causality
A research design in which the researcher both controls and randomly assigns values of the independent variable to participants
Randomized Experiments
- Researcher assigns differents treatments to participants, they are randomaly assigned.
- Gold standard for causal relationships
A/B experiment
Involves a treatment group which receives the treatment we want to investigate and a control group which does not receive the treatment (or a placebo).
Randomized control trial
Randomized Experiments: Why Do They Work So Well?
Random assignment ensures that the treatment and the control groups are comparable based on pre-treatment characteristics
This includes any known confounder… and even unknown confounders!
The treatment precedes the outcome, thus also ruling out reverse causality
Internal Validity
The degree to which we can be confident that a study identifies the causal effect of the independent on the dependent variable
External Validity
The degree to which findings can be generalized to other contexts
Ecological validity
Behaviour observed in artificial experimental settings may not generalize to the real world
Population Validity
Experiments often involve unrepresentative subject pools (e.g., UG students) and it can therefore be questionable whether experimental findings generalize from the study sample to the population of interest
Reactivity
People may change their behaviour when they know they are being observed
Labortary Experiments
- High level of control on what subjects are exposed to
- Concerns on population, ecological and reactivity validity
- High internal validity
Field Experiments
- Higher ecological validity, lower reactivity.
- Higher population validity
- Researches have less control over application.
Survey experiments
- Highest population vailidity, diverse or representative samples
Why not always experiments
- Sometimes it is hard for political scientists to manipluate variables.
- External validity concerns
A research design in which the researcher does not have control over values of the independent variable
Observational Research Design
- Good for description, questions regarding distributions, questions regarding charaterisitcs and meaning.
- Used for explanation
The values of the independent variable arise naturally in such a way that we can speak of true or, more realistically, “as if” random assignment
Natural experiments
- For causal questions only
- High internal and external validity
A good example is Joshua Angrist’s study of the effects of military conscription on earnings. Angrist leveraged the fact that there was a lottery in 1970-1972 to draft soldiers to the Vietnam War. Angrist compared earnings of those who were drafted with those who were not. Also illustrates that natural experiments are rarely perfect. Among thorny issues range that there were ways to evade the lottery, such as getting a medical prescription. Esp. young men from rich backgrounds (e.g., Donald Trump) were successful in this, which leads to bias.
A good example of what?
Natural Experiments
Different data structures - Cross sectional studies
Examine a cross-section of social reality, focusing on variation between individual units, such as citizens, countries, etc.
Typical example is an election study. Or a snapshot of democracy levels across the world at a given point in time. An explanatory study would leverage cross-sectional variation in the independent variable between units.
Different data structures - Time series
Examine evolutions of a single unit over time. A typical example would be studies looking at trends in economic performance in a single country.
