Experimental Exam #2 Flashcards
Quasi Experiments
- Have an IV and a DV
- But cannot randomly assign participants to levels of the IV
- Loss of control over the experiment
Independent groups: non-equivalent control-groups design
Have a comparison group but no random assignment to a condition
Ex: Olympic medal level (gold, silver, bronze) and mental health
Repeated measures: interrupted time-series design
Participants/groups are measured multiple times before, during, and after an “interruption” (the event of interest)
Ex: COVID mask mandates
Combined: non-equivalent control-groups interrupted time-series design
combo of the two others
Validity and Quasi-experiments
1) Internal validity: you can not control for confounds → suffer a lot
2) Statistical validity: pretty good, same as experiments
3) Construct validity: can be really good; same as or even better than experiments
4) External validity: can be really good; same as or even better than experiments
Quasi experiments and causal claims
**It is very hard to make a causal claim with quasi-experiments: if there are several quasi-experiments that all have the same results, you can be more confident in making a casual claim
Survey research
Self-report data:
- Asking the person: the world’s best expert on the many aspects of life is probably you
- The most common type of data collected → may be overused
- May not be true for young children or older adults
Question Formats for self-report
1) Likert
2) Open-ended/free response
3) Forced choice items
4) Semantic differential items
Likert
Ex: rate your agreement with the following statement “I am creative”
- 1 = strongly disagree, 7 = strongly agree
Open-ended/free response
Ex: “do you think that you are creative”
Hard to do math on these kinds of questions → often have to go in and code words into numbers (how often did you use positive vs. negative words etc.)
Qualitative research → takes effort to make it quantitative
Forced choice items
Ex: have to pick one option (force into yes/no)
Pros: avoids fence sitting → have to make a decision
Cons: loses nuance (range of responses)
Semantic differential item
use opposite adjectives
Ex: rate your creativity:
1 = uncreative, 7 = creative
problems with questions
1) Leading questions
2) Limited range of response options
3) Double-barreled questions
4) Negatively-worded/confusingly worded questions
Leading questions
wording leads the participants to give the results you want
Limited range of response options
forcing people to say yes, no or I don’t know → there is a range of support and feelings within these (no room for nuance)
Double-barreled questions
touches on more than one issue but you can only give one answer
Response sets
1) Fence sitting (middle for every construct)
2) Acquiescence bias (yes saying)
3) Social desirability
Statistical analysis and response types
ANOVA: requires categorial IVs and continuous DVs
- Can NOT use a forced choice question as a DV and do an ANOVA
- Can use Likert item as a DV and do an ANOVA
Experiment and prioritized validity
Internal validity
frequency claim and prioritized validity
external validity
Association claim and prioritized validity
construct validity
bivariate correlation
how 2 variables (usually scale/continuous but not necessarily) are linearly related
- Has a standardized scale
- Coefifient “r”
Descriptive correlations
Descriptive: heres how two things trend together
**Effect size: quantifying the strength of the correlation
Inferential correlations
Inferential: can make a statement about a population
- To do this ask: is this correlation coefficient statistically different from a correlation of 0
- You can have weak but significant correlations (it mostly tells you if your N is large enough)
- If you have a huge N, everything is significant
Correlations and manipulations
Nothing is manipulated → no causal inference drawn
Just seeing how things are related
What can influence correlations
1) outliers
2) restriction of range
3) nonlinearity
subjective validities
1) Face validity: does it look like a good measure? Ask experts to give their perception → think smell test
2) Content validity: does it include all the important components of the construct
Empirical validities
1) Criterion validity: measure predicts some real-world outcome
2) Convergent validity: Measure is more associated with similar measures
3) Divergent (discriminant) validity: measure is NOT associated with dissimilar measures
Reliability and validity
Reliability is necessary but not sufficient for validity
correlations and causal claims
For correlations: we have covariance NOT temporal precedence → so not causal claims
Directionality problems
- also internal validity: we do not look at other variables
Longitudinal designs
Measure people over multiple points
Cross-lag longitudinal study:
1) Auto correlations
2) Cross-lagged correlations
3) Cross-sectional correlations
Auto correlations
the correlation of each variable with itself over time
- Tell you about stability (interindividual stability)
Cross-lagged correlations
earlier measure of one variable correlated with later measure of a different variable
- How people change over time → can help establish temporal presedence
cross-sectional correlations
2 variables measured at the same time are correlated
Pattern and parsimony
Patterns are plausible, coherent, consistent, strong
What is most likely and simple is probably true
Multiple regression
- An expansion of the correlation
- Correlation between several predictor variables (IVs) and a single criterion variable (DV)
- Not r but beta → can be interpreted as a correlation coefficient
Moderation
when, for whom, or under what conditions are two variables related
A is related to b for one type of c but not for the other type
Ex: work frequency and reading time are related only for younger adults and not for older adults
Mediation
why are two variables related
A is related to b because a leads to c and c leads to be
Third variable problem
two variables are correlated but only because they are both linked to a third variable
Mediators and third variables similarities
1) Both involve multivariate research designs
2) Both can be detected using multiple regression
Mediators and third variables differences
1) Third variables are external to the correlation (problematic)
2) Mediators are internal to the causal variable (not problematic; explains relationship)
IRB
Institutional Review Board
- Required at every institution that received federal funds
Three ethical principles of the belmont report
1) Beneficence
2) Autonomy
3) Justice
Beneficence
maximize benefits and minimize risk
Autonomy
respect for persons
- Informed consent
- Need to provide compensation
- Describe any foreseeable discomforts or risks and how we will address that
- Need to describe what happens if you need to drop out of the study
- simple language
Justice
Ensure that equity is not violated when selecting participants
- Decisions to include or exclude must be made on scientific grounds
APA ethics code
Principle A: beneficence and nonmaleficence:
- Maximize benefits and minimize risks
Principle B: Fidelity and Responsibility
- Be responsible and professional in interaction with people
Principle C: Integrity
- Don’t lie, cheat, steal, commit fraud, etc.
Principle D: Justice
- Fairness and equity
Principle E: respect for people’s rights and dignity
- Respect for persons (informed consent)
Replication
attempt to repeat the result of an experiment by repeating an original study
- same data, same methods
- Ensures that the initial findings are not a case of “discovering” an effect that is not real (type I error) → false positive
Generalizability
fundamental results from as study are produced across a variety of situations
Why: generalizability across contexts, text the truth of the underlying hypothesis, discover boundary conditions
P-hacking
Collecting data or analyzing your data in different ways until non-significant results become significant
- Increases probability of type I error (false positive)
HARKing
Hypothesizing after results are known
- You analyze data and find a significant result (might be unexpected) and post-hoc come up with a hypothesis
- Increases probability of type I error (false positive)
Cherry picking
select/report only data/findings that support your hypothesis and hide other data
Only reporting significant effects
- Increase false positive
Fishing/data dredging
Look at a ton of different combinations of variables to find something significant
- Look at a ton of different combinations of variables to find something significant
- increase type I error rate
Peer review cons
1) Not a paid position → takes so much time
2) You don’t know who your reviewer is
3) If your reviewer doesn’t have good training, you are left with weird/wrong comments
Retractions
when you pull a paper from a journal (remove from scientific literature)
- Usually not because it’s wrong: because it is unethical
Replications
- Within and across labs
- Did not start this until 2011
- Expensive
Fraud detection
people explore published data sets and see if things look fishy
Open science framework
- Publish all of your data and statistics code
- Encourages transparency in all aspects of scientific conduct
- Provides a platform for this to be possible
Adversarial collaborations
Work with people who have opposing views to conduct a study that will help you figure out the correct
Solutions to bad data practices
1) Open science framework
2) Preregistration: forces scientists to publically outline tier plans prior to starting work
- Introduction and methods before collecting any data
- You can pivot but have to justify reasons for doing that
5) Change p-value standard
6) Publish your analyses so everyone knows what you did
