The Replication Crisis (Social Psych)

Question 1

What should Science be?

Answer 1

• Should provide reliable results.
• Should be transparent
• Should be reproducible
• Should be self-correcting
• Should be trusted
• Should be replicable

Question 2

Define what is meant by replication.

Answer 2

• a fundamental principle of the scientific method.
• Replicability: obtaining consistent results across studies aimed at answering the same question, each of which has obtained its own data.

Question 3

Define what is meant by reproducibility.

Answer 3

• obtaining consistent results when the same data is analysed using the same techniques.

Question 3

Identify the two types of replication.

Answer 3

Exact/Direct Replication.
Conceptual Replication.

Question 4

What is Conceptual Replication?

Answer 4

• where scientists try to confirm a finding using a different set of methods and measures that test the same hypotheses.

Question 4

What is Exact/Direct Replication?

Answer 4

• where a study uses the exact same measures, conditions as an original study to reproduce the results.

Question 4

Describe the Replication Crisis.

Answer 4

• Recently, psychology has been criticised because many classic research findings do not replicate.
• an ongoingmethodologicalcrisis insciencewhereby researchers find that the results of many scientific studies are difficult or impossible toreplicate or reproduceon subsequent investigation.

Question 5

Describe a case relating to the replication crisis (Bem, D.J. (2011)

Answer 5

• Title: Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect.Journal of personality and social psychology,100(3), 407.
• Used established protocols such as affective priming and recall facilitation – methodological rigour.
• 9 experiments with significant results – seemingly robust phenomenon but surprising (Wagenmakers et al., 2011)
• Underwent peer review and published in the prestigious JPSP(which rejects 90% of submissions).

Question 6

Describe problems with (Bem, D.J. (2011).

Answer 6

• “It was both methodologically sound and logically insane” (Engber, 2017)
• ESP is are outside current scientific explanations of human behaviour because they contradict fundamental principles of our current understanding of reality.
• Ritchie, S. J., Wiseman, R., & French, C. C. (2012). study indicated that the study by Bem, D.J., was not replicable as there were 3 unsuccessful attempts to replicate it.
• “If one had to choose a single moment that set off the “replication crisis” in psychology – an event that nudged the discipline into its present andanarchicstate, where eventextbook findingshave beencast in doubt – this might be it: The publication, in early 2011, of Daryl Bem’s experiments…”

Engber (2017)

Question 7

Describe the Open Science Collaboration Project (2015) and Many Labs.

Answer 7

• A team of international researchers called “Many Labs” aimed to replicate 100 studies in 3 top Psychology journals.
• Many Labs 1: Examples of studies included:
  Does people’s belief that human behaviour is predetermined encourage cheating?

Do children blindly follow eye gaze to find hidden objects?

Is there a motion ‘after-effect’ from still photographs depicting motion.

A large percentage of studies couldn’t be replicated.

Question 8

What Questions did people have about the replication failures?

Answer 8

• What if the replication study wasn’t identical to the original?
• Would the findings replicate in the same culture?
• What if the experimenters aren’t competent enough and lack the know-how to pull off the original experiment?
• What if the replication sample sizes were too small?
• Are there certain conditions in which the study would replicate? (these are known as ‘moderators’)

Question 9

Describe the Many Labs 2 project (Klein et al. (2018)).

Answer 9

• Many Labs 2 project was specifically designed to address these criticisms.
• 15,305 participants = 60x more than the original
• Researchers worked with the scientists behind the original studies to check every detail
• Repeated experiments many times, with volunteers from 36 different countries, to see if the studies would replicate in some cultures and contexts but not others.
• Only a 54% success rate.

Question 10

What is the failure rate for replicating studies relating to preclinical cancer research?

Answer 10

• 90% failure rate of replicating studies relating to preclinical cancer research.

Question 11

Identify what influences a Replication Crisis?

Answer 11

• The incentive structure of academia, which influences.
• Questionable Research Practices (QRPs), like P-hacking, HARKing.

Question 12

Describe the Incentive Structure of Academia.

Answer 12

• papers were required to get JOB > GRANTS > PDRs PRESTIGE > PROMOTION.
• The slow pace of science helps ensure that research is done correctly.
• But it can come into conflict with the incentive structure of academic progress, as publications—the key marker of productivity in many disciplines—depend on research findings.
• This led to small sample sizes, lots of papers, unreliable estimate and type 1 errors
• “More speed, more haste, more stress, more waste” (Frith (2019)).
• There was a drive for novel ‘sexy’ findings.
• Selected papers should present novel and broadly important data.
• The journal publishes cutting-edge research articles.
• novel research” methodologies

Question 13

Describe problems associated with the incentive structure of academia.

Answer 13

• leads to PUBLICATION BIAS
• this is the phenomenon that significant results have a better chance of being published, are published earlier, and are published in journals with greater prestige.

Question 14

Explain what are QRPS (Questionable Research Practices).

Answer 14

• in psychology, p-values are typically used to show that a finding is significant (p<.05)
• an incentive structure that can also lead to psychologists chasing positive findings (p<.05).

-These ‘significant/positive’ findings were traditionally sought after by journals (remember “sexy findings”).

• Influences researchers to engage in QRPs to find positive findings to enhance chances of publication/rewards.
• QRPs are not fraud but exploit a ‘grey area’.

Question 15

Describe what is meant by Academic Misconduct/Fraud.

Answer 15

• the explicit effort of a researcher to falsify or misinterpret data.
• e.g.,
  Research star: 130 papers, 24 book chapters.
  Developed studies and pretended he had conducted them.
  Was dismissed, returned his PhD.
  Criminal prosecution.
  54 papers retracted so far.

Question 16

Describe what P-hacking is (QRP)

Answer 16

• it occurs when someone excessively influences the data collection process or statistical analyses performed in order to produce a statistically significant result.

Question 17

What is HARKing?

Answer 17

• Hypothesising Afters Results are Known
• e.g., ., running multiple analyses…
• Silberzahn et al. (2018): Twenty-nine teams involving 61 analysts used the same data set to address the same research question: whether soccer referees are more likely to give red cards to dark-skin-toned players than to light-skin-toned players.
• Analytic approaches varied widely across the teams. Twenty teams (69%) found a statistically significant positive effect, and 9 teams (31%) did not observe a significant relationship.

Question 18

Explain the process of HARKing.

Answer 18

• Hypothesising AFTER the Results are Known (Kerr, 1988).
• process of looking at the data the developing ‘hypotheses’.
• Write the paper as a ‘story’
• Increases chances of false positives (Type 1 error)
• May mask ‘no effect’ leading to wasteful resources when people try to repeat the experiment.

Question 19

What is the Open Science Movement?

Answer 19

• movement developed in response to a culture secrecy and skepticism that has been pervasive throughout scientific research”.
• We need to know about conceptual, methodological and analytical choices […] so that we can make more informed assessments of credibility.

(Kathawalla et al. 2020; Vazire, 2017)

Question 20

Identify the features of Open Science.

Answer 20

• Open Data
• Open Materials
• Registered Reports
• Equality, diversity & inclusion
• Open Access
• Collaboration
• Open Source
• Open Peer Review
• Preregistration
• Educational Resources
• Open science can be seen as open “access”.

Question 21

What is Study Preregistration?

Answer 21

• Researchers commit to their research predictions and methodsbefore starting their experiments.
• Decisions about a scientific experiment that are supposed to be made before data analysis areactually made before data analysis.

Question 22

How does Open Data and Materials lead to Psychological Science?

Answer 22

• Open Data leads to Open Materials which leads to Preregistered which leads to Psychological Science.

Question 23

Describe the benefits of Open Science.

Answer 23

• (Protzko et al. 2023)
• Assessed the replicability of 16 novel experimental findings with optimal practices:

Optimal sample sizes

Study preregistration

Methodological and analytic transparency

• In contrast to past replication efforts (e.g., Many Labs), 86% of studies replicated, 97% had similar effect sizes.

Question 24

Describe Protzko et al. (2023) (paper)

Answer 24

• Reports an investigation by 4 coordinated laboratories of the prospective replicability of 16 novel experimental findings.
• was done using current optimal practices like high statistical power, preregistration, and complete methodological transparency.
• In contrast to past systematic replication efforts that reported replication rates averaging 50%, replication attempts here produced the expected effects with significance testing.

Question 25

Describe the tradition publishing process of a report.

Answer 25

• researcher plans a study > collects data > writes report > submits to journal and then peer-reviewers evaluate the paper and either accept or reject it.
• They know the design, they know the results, and the researcher can’t change anything about how they’ve run the study.

Question 26

Describe the process of publishing a registered report.

Answer 26

• The researcher writes up the introduction and methods section of a paper and includes detail of the hypotheses, sample and methods.
• Then undergoes peer-review where the reviewer can suggest changes to the study to make it stronger.
• The paper then gets either a ‘in principle acceptance’ or a rejection in the journal.
• If the paper is accepted, the data collection THEN takes place (after the first review process) and the researcher re-submits the full article with the results and discussion section now in place.
• If the researcher has STUCK to their plans, then the paper is published.
• This means that the reviewing process is ‘results blind’ – the paper is accepted regardless of a positive or null result! How much better is that!