W12 open science Flashcards
Replication crisis
- started in 2010
- researchers were unable to replicate key studies in psychology and other fields
- independent researchers following the same study protocols as original studies could not find similar results
- led to the ongoing questioning of established phenomenon and recommendations for scrutinising science in psychology
What triggered the replication crisis - Bem (2011)
- Bem (2011) claimed to have precognition - an ability to “feel into the future”
- claimed recall of information (words) was better when never initially exposed to the information
- this is contrary to existing knowledge about remembering information (rehearsal)
- three replication attempts were made by other researchers, nobody could replicate the results
What triggered the replication crisis - Simmons, Nelson, Simonsohn (2011)
- demonstrated you could find a significant effect in almost any dataset
- coined the term “Questionable Research Practices” (QRP’s)
- > engaging in research practices that will lead to finding a significant effect
- > they found that listening to a Beatles song made people younger which is obviously impossible
- used simulations to demonstrate that various decisions about data analyses made the results significant
What triggered the replication crisis - Stapel (2011)
- admitted to fabricating data, 50 publications retracted
- more researchers have been identified for doing similar things, one researcher has over 200 retracted papers
What is the solution - the reproducibility project
Reproducibility:
reproducing results when re-analysing the original data
Replication:
replicating results from new data sets
The reproducibility project
- started in Nov 2011 by developing a protocol for selection and conduct of replications from 3 major Psychology journals
- 100 replications of 100 experiments, 270 contributing authors
- converted the results to correlations
- only 36% of replications were successful (produced sig p values)
- average effect size of replications was around half that or the original studies
- social psych findings were less than half as likely to replicate as cognitive psych findings
Questionable research practices
- p-hacking
- > removing participants
- > not reporting all experimental conditions
- HARKING
- > Hypotheses After the Results are Known
Publication bias
- journals not publishing null results
- publications are major factor in academic promotion
- people are concerned about sharing their data/analyses
Small sample (original studies)
- small N leads to less reliable statistics
- original studies often lacked measures of effect size (relied on p-values)
Changes in the field in reaction the replication crisis
Making science transparent
- pre-registration
- open access to data
- open access to materials
- open access to results
Pre-registration
Preregistration (unreviewed preregistration)
Registered Reports (reviewed preregistration)
Registered Replication Reports (RRR)
Unreviewed preregistration
- researcher creates a detailed description of their plans for their study and records this is a time-stamped, unedited archive
- the plan can be shared with others
Registered reports
- researcher submits a detailed proposal of their planned study to a journal for a review of the planned methods and analysis
- similar to preregistration but addresses publication bias (results are published regardless of the outcome)
- some journals require this
Registered replication reports
- similar to registered reports but focus on direct replication of original findings
- numerous labs follow the same preregistered plan and the results are reported collectively
- > e.g. ManyLab project
Open science backlash
- a lot of backlash from established researchers (who typically benefit from flashy novel findings) as to this move to Open Science and the replication initiatives
- > tone
- > shameless little bullies
- > methodological terrorists
