stats Flashcards
structured plan or strategy for conducting a
study, outlining how data will be collected, analyzed, and
interpreted to address specific research questions or hypotheses
Research design
method used to investigate
causal relationships between variables by manipulating one or
more independent variables and observing the effect on one or
more dependent variables
Experimental research design
Experimental research design aims to
establish cause-and-effect relationships by controlling
for extraneous variables and minimizing bias
systematically managing or holding constant variables other than the independent variable to ensure that
observed effects are attributed specifically to the independent
variable rather than to these other factors
controlling
controlling for confounding variables helps to
isolate the real relationship between the independent variable (X) and the dependent variable (Y)
Experimental vs. observational research design
Experimental research designs involve actively manipulating
independent variables and using control groups to establish
causal relationships between variables (causes) vs. observational research designs involve monitoring
and measuring variables as they naturally occur, focusing on
identifying correlations without manipulating any variables (associated w)
experimental studies where participants are randomly assigned to either a treatment group or a control group to evaluate the effectiveness of an intervention while minimizing biases
Randomized controlled trials (RCTs)
group of participants in a
study that does NOT receive the experimental treatment or
intervention
Control group:
control group serves as
a baseline or a counterfactual to
compare the effects of the treatment on the experimental group.
a group of participants in a study that receives
the treatment or intervention being tested, allowing researchers to
assess its effects compared to a control group
Treatment group:
the process of randomly allocating
participants to control and treatment groups in a study to ensure that each group is comparable and to eliminate selection bias
Random assignment:
when the sample of participants in a study is not
representative of the population being studied, leading to distorted
or unrepresentative results.
Selection bias
random assignment assures
comparability between the control
group and the treatment group.
random assignment v random sampling
Random assignment means participants in an experiment are
randomly designated to receive a treatment or intervention vs. Random sampling means that members of the population under
study are randomly selected to participate in a survey
Randomized controlled trials are considered the gold standard for
causal research because they can cross the
four causal hurdles
the four causal hurdles
- Is there a credible causal mechanism that connects X to Y?
- Can we eliminate the possibility that Y causes X?
- Is there covariation between X and Y?
- Have we controlled for all confounding variables Z that might
make the association between X and Y spurious?
Because randomized controlled trials pass these 4 causal hurdles,
they are described as having
high levels of internal validity
the extent to which a study accurately measures
the causal relationship between the independent and dependent
variables, free from the influence of confounding factors.
Internal validity:
Drawbacks of randomized controlled trials
-not every x can be experimentally manipulated (ex:gender)
-experiments can exhibit low levels of external validity
-some experiments can’t be perfored because they create ethical issues
-Just because an experiment finds that X causes Y does not mean
that X is the most important cause of Y
the degree to which one can be confident that
the results of an analysis apply to the broader population
external validity
RCTs can be administered in many ways, the
most common way is
in-person at a laboratory.
experiments that leverage naturally occurring
random variations or events to investigate causal effects, without
direct manipulation of the independent variable by the researcher. (examples magnet school lotteries,
* military draft lotteries,
* random IRS audits,
* immigration lottery visas)
Natural experiments
Natural experiments exhibit
high levels of internal validity.
Natural experiments also exhibit –
because they study –
high levels of external validity, real-world scenarios and natural conditions
a research method where participants are
exposed to different experimental conditions within a single survey to examine how variations in the survey’s content or format influence their responses
Survey experiments
If the treatment is randomly assigned (as it usually is), a survey
experiment exhibits
high levels of internal validity
because survey experiments are executed on the
computer – rather than in the real world - they exhibit
low levels of
external validity.
research conducted in a natural setting where participants are usually randomly assigned to treatment and control groups to assess the causal impact of an intervention in real-world conditions
Field experiments
If the treatment is randomly assigned (as it usually is), a field
experiment exhibits
high levels of internal validity
Because survey experiments are executed in the real world – they
also exhibit
high levels of external validity!
studies that compare the effects of an
intervention or treatment between pre-selected groups that are not
randomly assigned, aiming to assess causal relationships while
controlling for confounding variables
Controlled experiments
we do controlled experiments instead of RCTS because
Practical constraints, such as logistical challenges or limited
access to participants
* Ethical concerns
* The groups are pre-existing so random assignment is not
possible
* Cost and time
research designs that aim to evaluate
interventions or treatments without full randomization, often using
pre-existing groups or natural conditions to infer causal
relationships.
Quasi-experiments
Types of quasi-experiments
-Regression discontinuity design (RDD)
-Nonequivalent groups design
-Pretest-posttest design
-Interrupted time-series design
- Matching design
-Difference-in-differences (DiD)
-Instrumental variable research design
participants are assigned
to treatment or control groups based on a cutoff score or threshold
on a pretest measure, comparing outcomes just above and below
the cutoff (ex: Example: evaluating the impact of a voter turnout program by
comparing election outcomes for districts with voter turnout
rates just above and just below a pre-set threshold that triggers
eligibility for the program)
Regression discontinuity design (RDD)
compares outcomes between
groups that are not randomly assigned, often using pre-existing
groups.
(Example: evaluating the impact of a new voter ID law by
comparing election turnout rates between states that enacted
the law and neighboring states that did not)
Nonequivalent groups design
measures the same group of participants
before and after an intervention to assess changes over time.
(Example: assessing the impact of a new civic education
curriculum by measuring students’ political knowledge and
engagement levels before and after the curriculum is
implemented in schools)
Pretest-posttest design
compares data collected at multiple
time points before and after an intervention to identify changes
attributable to the intervention.
(Example: analyzing the effect of a new campaign finance
reform on political contributions by examining trends in donation
amounts and frequency before and after the reform was
implemented)
Interrupted time-series design
researchers pair participants or groups based on
similar characteristics to control for confounding variables and
compare outcomes between the matched groups.
(Example: comparing the effectiveness of two different voter
outreach strategies by matching neighborhoods with similar
demographic and political characteristics, and then comparing
voter turnout rates between those exposed to each strategy)
Matching design
estimates the effect of a naturally
occurring intervention by comparing the changes in outcomes over
time between a treatment group and a control group.
(Example: Analyzing the impact of a new voter ID law by
comparing changes in voter turnout before and after the law’s
implementation between states that adopted the law and similar
states that did not)
Difference-in-differences (DiD)
estimates causal
relationships when randomization is not possible, by using a
random, naturally occurring external variable (the instrument) that
affects the treatment but is not directly related to the outcome
except through its effect on the treatment.
(Example: examining whether economic slowdowns lead to
increased likelihood of conflict by using rainfall as an instrument,
assuming that rainfall affects agricultural productivity and
economic conditions but does not directly influence conflict)
Instrumental variable research design
Because quasi-experiments do not include a fully random
assignment of treatment, they exhibit
low levels internal validity.
Because quasi-experiments rely on observational data from
real-world settings, they exhibit
high levels of external validity.
research designs in which the
researcher does not have control over values of the independent
variable because the independent variable occurs naturally
Observational research
In cross-sectional data, the data vary by
geography, individual,
institution, or other units
— does not vary in cross-sectional data
time
Can observational research using cross-sectional data cross the
four causal hurdles?
No, but we can get close
Researchers using observational methods do their best to
approximate causality, but
correlations between variables that are established by observational research should not be confused with
causation.
