Guest Lecture: Replication Crisis

Question 1

reproducibility project in psychology

Answer 1

Massive, multi-lab project that randomly selected articles that were published in 2008 in 3 top psychology journals

Question 2

when did the reproducibility project occur

Answer 2

november 2011-2015

Question 3

findings of the reproducibility

Answer 3

97% of original studies had significant results (P< 0.05), but only 36% of replications had significant results. 39% of effects were subjectively rated to have replicated

Question 4

what famous studies failed to replicate?

Answer 4

Ego depletion (willpower is a limited resource), power posing (expect self-reported mood), and precognition (psi)

Question 5

what causes failures to replicate

Answer 5

fraud, file-drawering, innocent errors, p-hacking

Question 6

Diederik Stapel

Answer 6

fabricated 58 papers going back to at least 2004

Question 7

p-hacking

Answer 7

Run at least one unplanned analysis that you might report as though its p-value is valid

Question 8

false positivity

Answer 8

When we falsely reject the null

Question 9

what is the false positivity rate in psychology

Answer 9

5%

Question 10

common ways of p hacking

Answer 10

• Stop data collection if and only if p < .05
• Analyze many measures; report only those that were p < .05.
• Analyze many conditions; report only those that differed at p < .05.
• Use different (combinations of) covariates to try to get p < .05.
• Exclude participants or trials to try to get p < .05.
• Analyze different subgroups to get p < .05.
  Transform the data to try to get p < .05

Question 11

Choose between two dependent variables correlated at .5 results in

Answer 11

9% false-positive rate

Question 12

Collect 20 observations per cell. If not significant, run 10 more per cell results in

Answer 12

14% false-positive rate

Question 13

Analyze the data using gender and (gender x independent variable) covariates results in

Answer 13

31% false-positive rate

Question 14

Collect 3 conditions. Allow yourself to drop a condition if not significant results in

Answer 14

60% false-positive rate

Question 15

Run two studies. Drop one if it isn’t significant results in

Answer 15

84% false-positive rate

Question 16

three possible solutions to p-hacking

Answer 16

disclosure, pre-registration, and exact replication

Question 17

disclosure

Answer 17

Authors must report all measures, manipulations, data exclusions, and how they arrived at their sample size
Helpful but not a complete solution; it hinges on reviewers’ educated guesses about what analyses were attempted.

Question 18

pre-registation

Answer 18

• An analysis plan you make in advance for conducting the study and upload to a public repository
• A more complete solution
• Slightly higher cost to researchers

Question 19

exact replication

Answer 19

• The best way to establish that a finding is replicable
• most high effort and high cost solution

Question 20

open science network

Answer 20

Online platform for sharing data, materials, and pre registrations