Foundations of Design Flashcards
Non-experimental design
Descriptive
Correlational
Researcher gathers data without making any kind of intervention
Experimental designs
Non-randomized (quasi experimental)
Randomised
Aim to examine associations in order to make predictions or explore causal linkages
Non-experimental: descriptive
Used to assess prevalence, incidence rates
Non-experimental correlational
Examine the relationship between two or more variables to see whether they covary, correlation or are associated with each other
Two types of correlational design
Cross-sectional (all observations made only once at a single time)
Longitudinal (measurements made at two or more time points)
Inferring causality from correlational research
- Covariation (variables must occur together)
- Precedence (the hypothesized causal variable must reliably precede the outcome variable)
- Exclusion of alternative explanations
- Logical mechanism (there must be a plausible account/THEORY for they hypothesized causation)
Problems with causation
- Bidirectionality (two-way causality; X –> Y OR Y –> X)
- Spurious association (no relationship between X and Y, but by a third variable)
- Mediation or Moderation
Non - experimental: Quasi/non-randomized experimental design
One group posttest only design (X O; intervene and then observe)
One group pretest posttest design (O X O)
RCT
Gold standard
Pretest-posttest design with randomised groups
No-treatment control
Control group gets zero treatment
Wait-list controls
Delay before treatment
Placebo control
No real treatment given
Comparative treatment groups
Alternative treatment used which is also effective (‘treatment as usual’)
Dismantling studies
Break apart treatment into components, and use each component in isolation
Good experimental feature designs
- Patient homogeneity
- Randomised assignment
- Specific intervention (manualisation)
- Control
- Low attrition
- Groups treated equivalently (except intervention)
- Double/triple blind
- Independent replication
Three types of experimental validity
- External validity
- Internal validity
- Statistical conclusion validity
External validity
To what extent can the study results be generalized to other samples with different characteristics than the study sample?
Internal validity
Degree to which causality can be inferred from a study
To what extent the intervention/manipulation and not chance can account for the study results
Spontaneous remission (threat to IV)
Recovery from a disorder without reason or intervention
Interfering events (threat to IV)
Significant events that occur between pre and posttest measurements (natural disaster, finds a partner, wins lotto etc.)
Secular drift (threat to IV)
Long term social trends taking place over time (e.g. smoking, premarital sex, etc.)
Maturational trends (threat to IV)
Growth or maturation of the person that may account for some of the findings
Regression to the mean
Extreme scores revert towards the mean of a distribution when a measurement is redistributed
Extreme scores can only get better, even if the treatment isn’t working
Attrition (threat to IV)
Loss of participants over time
Only a problem if its different between conditions OR there are different reasons
Diffusion (threat to IV)
Participants in the control group may receive aspects of the intervention (drug sharing, talking amongst participants)
e.g. workplace studies
Special treatment/reaction of controls
Controls may be aware that they are not receiving treatment and may be motivated to out perform the experimental group
May feel demoralized or lose interest
Poor adherence to treatment (threat to IV)
Doesn’t attend all treatment sessions or take medication (50% of schizophrenics)
Confounded manipulation (threat to construct validity)
More than one thing can be accidentally manipulated (e.g. therapy and therapist)
Expectancy effects (threat to construct validity)
Sometimes people get better simply by thinking the experiment will work (placebo)
Hawthorne (threat to construct validity)
Sometimes people appear to get better simply because they’re being observed (i.e., like the attention)
Statistical conclusion validity
The extent to which the analyses performed enables one to draw correct/true inferences about the phenomena of interest
Threats to statistical conclusion validity
Lower power (small n/effect size)
Improper data analysis (reporting only the “best” results)
Confounding variables
Poor reliability/validity of measures
The dead salmon study
Brain activity “found” in a dead fish
If you do a lot of statistical tests some will give you result just by chance
Systemic pressure on statistics
Best journals favour novel and statistically significant results, thus researchers are motivated to keep running statistics until they find something that fits these criteria
