Replication crisis

Question 1

Why is replication important in psychology?

Answer 1

Replication helps to verify that a presence of a behaviour at one point in time isn’t due to chance

Question 2

When was the term replication crisis first used?

Answer 2

Brian Nosek’s lab
replicated 100 studies from 3 psychology journals
found that they were unable to produce the large majority of findings

Question 3

What is the replication crisis?

Answer 3

Significant amount of scientific studies have difficulty replicating the findings,
Issues with validity and reliability

Question 4

What is the difference between reproducibility and replication?

Answer 4

Replication: consistent results when a study is repeated
Reproducibility: the ability to achieve the same results when reanalysing the same dataset

Question 5

What is the impact on science and society as a result of replication crisis?

Answer 5

Credibility of research
Policy and decision making: decisions based on non-replicable research could implicate sectors such as health, education and public policy
Scientific progress: inability to build on previous research

Question 6

What did Ioannidis find regarding the replication crisis?

Answer 6

A large majority of published research can’t be replicated to find similar results
Lower false positive rates (error type 1)

Question 7

What are some examples of questionable research practises (e.g. HARKing, publication bias and voodoo correlations)?

Answer 7

HARKing: hypothesizing after results are known
Publication bias: studies with statistically significant findings are more likely to be published
voodoo correlations: unrealistically high correlations are reported

Question 8

What is the Diederik stapel scandal?

Answer 8

Fabricated data in many studies
Fraudulent research

Question 9

What is the scandal with Bem’s ESP study?

Answer 9

Controversial study
Claimed evidence for precognition
we now have rigor peer review processes

Question 10

What percent of replicated studies produced the same results?

Answer 10

36%
Rest of the studies couldn’t produce the same results

Question 11

What is the power pose study? What did the replicated studies show?

Answer 11

Higher power pose: made riskier bets, higher testosterone, lower cortisol
Compared to lower power pose

Replicated studies found: no risk tolerance, no effect on hormones

Question 12

What is the social priming study? What have replicated studies found?

Answer 12

Ps had to unscramble a jumbled list of words to form coherent sentences
When these words were pertained to ageing or the elderly, ps walked out the lab slower than the list that did not contain such words

This has been a tricky study to replicate

Question 13

How do small sample sizes impact replication?

Answer 13

Lower statistical power
Increase likelihood for type 1 error

Question 14

How does misusing statistical significance impact replication?

Answer 14

overestimating p value

Question 15

How does p hacking impact replication?

Answer 15

manipulating data analysis to achieve statistical significance

Question 16

How does publication bias/selective reporting impact replication?

Answer 16

Journals favour positive/ publishing significant findings over null
Leads to distorted scientific records

Question 17

How does the lack of data sharing impact replicability?

Answer 17

impossible to perform re-analysis

Question 18

What is the scandal in research into the drug Paxil?

Answer 18

SSRI antidepressant
Controversy over safety in children and adolescents
Later criticized for downplaying risks and ghost writing

Question 19

How can we mitigate the replication crisis?

Answer 19

Promotion of replication studies: encourage researchers to replicate studies, even if they fail to produce the same results as the original study

Open Science practises: open access to data and methodologies, to increase transparency

Preregistration of studies: researchers should register their study design and analysis plan before collecting data to prevent selective reporting

Question 20

What is the evidence for how well mitigation strategies for the replication crisis work?

Answer 20

4 independent research groups
replicated each others findings
used open data sharing, standardized methodologies
found a very high replication rate
research protocols increase reproducibility