Ch 14 Flashcards
Replicable
the same results have actually been reproduced
- Not that the study could be replicated, but that the result of the study has been repeated
types of replication studies
- Direct replication
- Conceptual replication
- Replication-plus-extension
Direct replication (Exact replication) + risks
original study is repeated as closely as possible to determine if the original effect is found in the new data
- Risk of having the same threats to internal validity or construct validity in the original study - other types address this
Conceptual replication:
same research question, different procedures- conceptual variables are the same, but the variables are operationalized differently
Replication-plus-extension
Replication-plus-extension: researchers replicate their original experiment and add variables to test additional questions or to better understand the scope of the original effect
- Could add an additional level to an IV
- or add an additional IV
Replication Crisis
Journals prefer to publish new research, no incentive to do direct replication studies
- Open Science Collaboration decided to attempt replication on larger scale, (39% was lowest rate)
Why might replication studies fail?
- Contextually sensitive effects:
- Number of replication attempts:
- problems with original study
(why might fail) Contextually sensitive effects:
Contextually sensitive effects:
- Measures and manipulations used in replications might not have same meanings as in original study- too sensitive, if replication context is different might fail
(why might fail) Number of replication attempts:
- One replication- any single study has the potential to miss a true finding
- Many Labs Project- did up to 36 replications of each study + combined them (replication rate rose to 85%)
(why might fail) problems with original study
Sample size-
- too small, extreme participants could’ve had disproportionate influence on means and pattern
Harking
P-hacking
Harking
Hypothesizing after the results are known - usually due to a surprising result.
more likely due to chance, can’t be replicated (Type I errors more likely)
P-hacking
- Researcher peaks at study’s results, if not significant may run a few more individuals, remove outliers, or run different type of analysis
- Called p-hacking because trying to get statistically significant reading- p of under .05
improvements to scientific practice
- Larger sample sizes: Journals now require much larger samples for both original and replication studies
- Report all analyses and variables
- Open Science
- preregistration
open science
Sharing ones data and materials freely so others can collaborate, use, and verify results
Open data- provides full data set so researchers can conduct new analyses on it, or reproduce results
Open materials- provides full set of measures and manipulations (ALL variables) so others can replicate study
preregistration
Preregister study’s method, hypotheses, or statistical analyses online, before collecting data
- Discourages p-hacking
meta-analysis
statistical analysis that gives a quantitative summary of a scientific literature
- Averages the results of all the different studies
Scientific literature
Series of related studies conducted by different researchers who have tested similar variables
- A review article collects all the literature on a topic and reviews them together
File drawer problem
since journals don’t publish null effects or opposite effects as often, a meta-analysis of published studies may overstate the size of an overall effect
- To counteract, researchers conducting meta-analyses should ask their colleagues for published and unpublished data
probablility sample vs. convenience sample
A probability sample is intended to generalize to the population it was drawn from, but convenience sample may not generalize to the intended population
external validity- what’s important?
External validity comes from how not how many- how obtained is more important
Just because a sample comes from a population doesn’t mean it generalizes to that population
how can we see if a study generalizes to other settings?
Conceptual replication can show if a study generalizes to other settings
Ecological validity(mundane realism)
aspect of external validity that asks if a study conducted in a lab generalizes to real-world settings
Theory-testing mode:
typically used to test association or causal claims, to investigate support for a particular theory
- In theory testing mode, internal validity is more important than external validity
Generalization mode:
where researchers want to generalize the findings of a previous study
- Important to use probability samples w/ diversity
- Applied research tends to be done in generalization mode
what mode are frequency claims done in?
generalization mode
what mode are association and causal claims done in?
Usually in theory testing mode, sometimes generalization
- Might first try it in theory testing mode, then shift into generalization mode
Cultural Psychology:
type of psychology that focuses on cultural contexts and how they shape thoughts, emotions, and behavior
- Work in generalization mode, challenge researchers who don’t consider context and only work in theory-testing mode
WEIRD
Western, Educated, Industrialized, Rich, and Democratic
WEIRD samples aren’t representative
field setting
When a study takes place in the real world, it occurs in a field setting and has high external validity
ecological validity
how similar a study’s manipulations or tasks are to situations participants might encounter in their lives
- field settings have high ecological validity
- important to consider in terms of generalizing to non-lab settings
Experimental realism:
when lab experiments create situations in which people experience actual emotions, motivations, and behaviors
Laboratory research can be just as realistic as studies in the real world
