Exam 2 Flashcards
What is a Quasi-Experimental Appraoch
must have an IV and DV
cannot randomly assign participants to levels of the IV
Loss of control over the experiment
Types of Quasi-Experiments
- Independent groups (non-equivalent control-groups design)
- Repeated measures (interrupted time-series design)
- Combined: non-equivalent control-groups interrupted time-series design
What are Independent groups quasi experiments
non-equivalent control group design
have a comparison group but no random assignment to condition
What are repeated-measures quasi experiments
interrupted time-series design
Participants/groups are measured multiple times before, during, and after an “interruption” (the event of interest)
what are non-equivalent control-groups interrupted time-series design
The interrupted time-series design with nonequivalent groups involves taking a set of measurements at intervals over a period of time both before and after an intervention of interest in two or more nonequivalent groups.
How are quasi-experimental designs in terms of statistical validity
pretty good, same as experiments
How are quasi-experimental designs in terms of construct validity
can be really good, same as or even better than experiments
How are quasi-experimental designs in terms of external validity
can be really good, same as or even better than experiments
How are quasi-experimental designs in terms of internal validity
not good!
how to controls for selection effects in quasi-experiments
“wait list control”
Likert-style items
Rate agreement of a statmetn on a scale of strongly disagree to strong agree
I am creative
Forced choice items
I am creative
I am uncreative
Semantic differential items
uses opposite adjectives
rate your creativity from
1 = uncreative
7 = creative
Threats to constructs validity of surveys
Problems with survey itself:
* Leading questions
* Limited range of response options
* Double-barreled questions
* Negatively-worded (confusingly-worded!) questions
Problems with how participants may respond:
* Response sets
* Fence-sitting (not an issue for every construct)
* Acquiescence bias (“yea-saying”)
* Social desirability
Doubled-barreled questions
Touches on more than 1 issue, but you can only give 1 answer
Likert Items are for
continuous quantitive variables (interval)
Forced-choice items
Categorical / nominal variables
An ANOVA or a regression is typically prefered to a chi square because
if we can analyze more variance (1 to 7 vs. yes/no), we have a better chance of finding an effect
___ data have less power and precision than ____ data
continuous
If it’s an important construct…
include more than one item
what is an important consideration in survey construction
pay attention to reliability and validity
Bivariate Correlation
How 2 variables (usually scale/continuous but not necessarily) are linearly related; on a standardized scale
coefficient r in a bivariate correlation
quantifies the relationship
-1 to +1
Sign indicates direction
Absolute value indicates strength of association
Correlations are both descriptive and inferential
You can have weak but
significant correlations (mostly a
test of if your N is large enough)
Internal Validity for correlational design
correlational studies have limitations that make it hard to make a causal claim
Construct and statistical validity for correlational designs
both are pretty similar between correlational designs and experimental designs
What three things can influence a correlation
outlier, restriction of range, and nonlinearity
What is face validity?
Does it look like a good measure? Ask experts!
Criterion Validity
Measure predicts some real-world outcome (like an important test/clinical diagnosis etc.)
Convergent Validity
- Measure is more associated with similar measures
- Correlates with things it ought to correlate with
Discriminant (divergent) validity
- Measure is not associated with dissimilar measures (not negatively associated
with; r = 0) - Doesn’t correlate with things it ought not correlate with
Patterns and Parsimony
When a variety of correlational findings establish a single cause-effect relationship then that is a case of pattern and parsimony
Longitudinal Designs
Measure people over multiple time points
Cross sectional correlations
2 variables measured at same time are correlated
Autocorrelations
the correlation of each variable with itself over time; might be a problem
Cross-lagged correlations
earlier measure of one variable correlates with a later measure of a different variable — how people change over time; can help establish ??? .
What can help with the directionality problem
Longitudinal studies
What is the third-variable problem
A type of confounding in which a third variable leads to a mistaken causal relationship between two others.
Multivariate Designs
- More than 1 “IV”
- Often we begin with a simple bivariate exploration: variable A & B
- Then we often use multivariate analyses to dig deeper
- To look at the impact of multiple predictors on a single outcome variable
- Especially when trying to establish causality, but can’t do an experiment
Multiple regression is just an expansion of
correlation
Multiple Regression is a
correlation between several predictor variables (IVs) and a single outcome variable (DV)
Each regression coefficient represents the relationship between
IV-1 and DV, controlling for the other variables
Mediation Grades, Self-Esteem, and Happiness example
Grades predict Self-Esteem, and Self-esteem predicts Happiness. Self-esteem is the “mechanism”. Grades themselves do not
predict happiness.
Moderation Grades, Self-Esteem, and Happiness example
Grades predict Happiness but
only for certain levels/groups/
aspects of Self-esteem
Mediators and Third Variables Similarities
- Both involve multivariate research designs
- Both can be detected using multiple regression
Mediators vs Third Variables Differences
- Third variables are external to the correlation (problematic)
- Mediators are internal to the causal variable (not problematic; explain a
relationship)
Four Types of Reliability
Test-retest
Inter-rater
Internal Consistency
Parallel forms
Test-retest:
Relevant for most measurements
Inter rater:
Relevant only if 2 raters produce the measurement
Internal Consistency
Relevant only if measurement includes survey/
questions; do not get it confused with “internal validity”
Mediation
A mediator variable transmits the effect
from an independent variable to a dependent
variable
Moderation
A term interchangeable with
“interaction.” Eg. IV predicts DV only for certain
levels.
Same methods same data
reproducibility
Same methods different data
Generalizability
different methods and same data
sensitivity
different methods and different data
full generalizability / conceptual replicability / other reason
why would you do reproducibility
to ensure that there wasn’t a type one error
When alpha is set at .05, this means that approximately __ in ___ significant effects is a false finding (if there is truly no effect)
1 in 20
What is the point of conceptual replication
generalizability, to test the truth of the underlying hypothesis, and discover boundary conditions
What did diederik Stapel do?
Had influential work and fabricated data for over 50 peer reviewed journal articles
What research did not replicate well
Social priming research
What percent of the studies replicated
35%
P-hacking
Collecting data or analyzing your data in different ways until non-significant results become significant.
HARKing
Hypothesizing after results are known
You analyze data and find a significant result (might be unexpected), and post-hoc come up with a hypothesis. Importantly, you then report the findings as though this has been the case all along
Cherry Picking
Select/report only data/findings that support your hypothesis. If your data/findings do not support it, you hide it away in the “file drawer”
Fishing/data dredging
No hypothesis in mind
just look through data until you find something significant
Some methods of catching/correcting errors in science
solid training in methodology and statistics
peer review
open discourse
retractions
replications
fraud detection
open framework
adversarial collaborations
Institutional Review Board
Committee that interprets ethical principles and ensures ethical procedure for a human research
Tuskegee Study
Did not give informed consent or tell people they had syphilis or give them proper treatment
Besides the IRB who else decides what is ethical and how
the HHS (Department of health and human services) and OHRP (office for human research protections)
Beneficence
Maximize benefits and minimize risks
Respect for persons
an autonomy principle involving informed consent
Justice
Ensure that equity is not violated when selecting participants
Decisions to include or exclude must be made on scientific grounds
3 Belmont Principles
Beneficence, Respect for Persons, and Justice
5 APA Ethics Principles
Beneficence and Nonmaleficence
Fidelity and responsibility
Integrity
Justice
Respect for People’s Rights and Dignity
Fidelity and Responsibility
Be responsible and professional in interactions with people
Integrity
Don’t lie, cheat, steal, commit fraud, etc
What are the problems with Facebooks study?
No Informed Consent
Did not know they were in a study
No debriefing
Violates beneficence and nonmaleficence (benefit does not outweigh the risk)
