the replication crisis Flashcards
what is the replication crisis
the failure to replicate published findings
what is the timeline of the replication crsis
1) published ESP (extra sensory perception) article that said people could predict the future
- uproar when this came out, the way you present research can appear a certain way
- author of this was famous for writing about how to get article published, but did this in immoral ways
2) confessions to fabricating data
- 30 publications deleted
3) failure to replicate famous studies
- could not replicate basic social psych studies (elderly words priming slower walking)
4) fewer than 50% of studies replicated
what is the crisis in the field surrounding replication crsis
are a lot of phenomenon we found not real?
-why some things do not replicate, but they do actually exist?
and why somethings do not exist but get published anyways
what are the null effect possibilities
1) too small sample size
2) inadequate manipulation
3) wrong hypothesis
what are meaningful failures to replicate
when the OG study results fail to replicate but they actually do mean something
why dont studies replicate
1) direct v. conceptual replication
2) questionable research practice
what are the two kinds of replications
direct and conceptual
what are direct replications
seeks to validate a particular finding with same method (exact same operational definitions and procedure)
- lots of little things could affect the results of replications (ex. different computer monitors)
- what appears to be identical might not be
what are conceptual replications
seeks to validate the underlying theory or phenomenon with different methods (also occurs during “extension” studies that identify moderators etc)
- replication with different methods is stronger evidence a phenomenon exists
- moderators: effect exists for some people in some conditions but not all
explain the example with naturalistic bias
people prefer success in others with natural talent rather than those with work ethic
- could not replicate it in Arcadia (different measures-so went to original measures)
- original research read news articles about pro musician who either worked hard than had natural talent (it worked)
how can a direct manipulation attempt deviate too much
1) inadequate manipulations
2) moderators
explain inadequate manipulations example in facial feedback
facial feedback hypothesis
-mouth in smile = more likely to find funny but with video cameras on face
** this did not replicate, participants were video recorded which affected the results**
self awareness moderates facial feedback
explain moderators
the official effect may have been real only for that sample
explain Brown’s example of questionable research practice
Brown could not replicate own findings and she used direct methods, it was the same and not fishy
-concluded there was a moderator (study about politics) depended on the political climate at the time (everything occurs in a broader context)
explain common and questionable research practices
- failing to report all DVs
- collecting data after preliminary checking results
- only reporting studies that “worked”
- failing to report all conditions ran
- claiming to have predicted an unexpected finding
explain researcher degrees of freedom and an example
researchers used to do poor practices, it was the accepted field standard bc they had this freedom but it has changed
ex. participants computer freezes, do you scratch the data
explain low incentives for conducting replications
- published studies shown once without any replication attempts
- science as a public dilemma (whats good for the individual might not always be good for the group)
- its good to get new ideas published for individuals (do not replicate others to improve their own brands)
- its good to replicate studies for the field as a whole
what are some strategies for change in the replication crisis
1) reward replications
2) pre-registering studies
3) making data publicly available
4) large sample sizes (reduces type one error)
explain rewarding replications
motivate people to do this
-but if we reward them too much “someone can build a career on incompetence”, get credit for repeating someone else’s work, if something does not replicate it could be because of poor manipulations (build a career on trashing others work)
explain pre-registering studies
- make methods and analyses ready in advance to public
- forces people to be honest
what are the costs of the changes in the replication crisis
- the push for huge sample size hurts researchers at small colleges, researchers without grants, young researchers (big colleges, established people and the wealthy benefit)
- exploratory work will suffer (brand new ideas will likely fail)
- unrealistic high standards
