Week Six - Replication Crisis

Question 1

2 types of Replication?

Answer 1

Exact and Conceptual

Question 2

What is Exact Replication?

Answer 2

Direct Replication - Attempt to exactly copy the scientific method used in an earlier study to determine whether results are consistent.
Same/similar result indicates that the findings are accurate

Question 3

What is Conceptual Replication?

Answer 3

Attempt to copy the scientific hypothesis used in an earlier study to determine whether the results will generalise to different samples, times or situations.
Same/similar results indicate that the findings are generalisable/

Question 4

The history of the Replication Crisis?

Answer 4

2011 publication from Daryl Bem claimed to have found evidence that future events could predict the past. Replication attempts failed to reproduce the effect. Stats were criticised eg familywise error rate

Question 5

2015 Open Science Collaboration

Answer 5

98 studies, only 36% replication

Question 6

Effect size and Replication?

Answer 6

Studies that do replicate have smaller effect size (half) compared with original.

Indicates original study estimates are on the high end (and prob why they end up being published)

True effect will emerge after multiple replications.

Question 7

Implications of False Positive Publications

Answer 7

1. Erodes the credibility of psychological science
   - – risks discrediting the value of evidence and feeding antiscientific agendas
   - – can make people question the scientific process.
2. Can be dangerous
   — health research: claim that a treatment is effective when
   actually it isn’t, or even has negative side effects
3. Wastes resources
   - – Researchers waste time, effort, and money conducting research on effects that don’t actually exist

Question 8

It’s vital that non-scientists understand what?

Answer 8

That science is messy (even when it’s working as it should), so that they don’t flat-out reject it when some of that mess starts to show.

Question 9

Explanations for non-replication?

Answer 9

1. Bad Science
   - original study results might have been falsified
   - unskilled scientists in replication
   - use of wrong stats
2. Questionable Research Practices
   - p-hacking: analysis results in diff ways and reporting the best results
   - HARKing: hypothesising after results
   - not enough samples
   - deleting outliers where not applicable
   - cherry picking
   - stopping data collection when significant
3. Publication Issues
   - bias toward accepting novel and sig findings ‘file-drawer effect’
   - problems with peer reviewers
   - inconsistent research standards with diff journals
4. Diversity in study populations
   - no two samples are exactly the same
   - participants change over time
   - everyone thinks differently

Question 10

Novel findings and replication?

Answer 10

Papers that failed to replicate sound too good to be true

The irony here is that novel findings are in fact less likely to be true precisely because they are so strange—so they should perhaps warrant more scrutiny and further analysis by journals and peer reviewers.

Boring but accurate studies may never get published (put in the filedrawer).

Question 11

Bad Science and non-replication

Answer 11

1. Bad Science
   - original study results might have been falsified
   - unskilled scientists in replication
   - use of wrong stats

Question 12

Questionable research practices and non-replication

Answer 12

2. Questionable Research Practices
   - p-hacking: analysis results in diff ways and reporting the best results
   - HARKing: hypothesising after results
   - not enough samples
   - deleting outliers where not applicable
   - cherry picking
   - stopping data collection when significant

Question 13

Publication Issues and non-replication

Answer 13

3. Publication Issues
   - bias toward accepting novel and sig findings ‘file-drawer effect’
   - problems with peer reviewers
   - inconsistent research standards with diff journals

Question 14

Diversity and non-replication

Answer 14

4. Diversity in study populations
   - no two samples are exactly the same
   - participants change over time
   - everyone thinks differently

Question 15

Examples of notable findings breaking under replication?

Answer 15

Ego depletion theory: When the energy for mental activity is low, self-control is impaired

Smiling makes us feel happier (facial-feedback hypoth)

People who grow up with more siblings are more altruistic

Power posing boosts testosterone, decreases cortisol

Question 16

Priming and replication?

Answer 16

VIDEO: found an effect

LARGER STUDY:
When you’re primed with “old people” words, you walk more slowly. “old”, “lonely”, “retired”, “bingo”

In a replication using infrared sensors (as opposed to students with stopwatches) to time walks, no evidence was found of this priming effect.

You only get the effects if you tell the experimenter that the person is supposed to walk slowly

Question 17

Replication issues in other sciences?

Answer 17

Medical research: only 59% of highly cited clinical research studies replicated.

Genetics: only 44% of associations replicated in genetic studies.

Economics: only 25-63% of studies replicated in economics (some using the same data!)

Question 18

Scientific Values (norms vs counternorms) 5

Answer 18

Communality vs Secrecy

Universalism (evaluate research on its own merit) vs Particularism (evaluate based on reputation)

Disinterestedness (motivated by discovery) vs Self-interestedness (views as competition)

Organised Skepticism (consider old vs new) vs Organised Dogmatism (promoting ones own work)

Quality vs Quantity

Question 19

What is science more reflective of these days?

Answer 19

A mismatch between Scientific Values and the Practice of Research. Science more reflective of counternorms eg doing it for career progression and not knowledge/discovery

Question 20

What cultural shifts are taking place?

Answer 20

Open science
Institutional change
Journal policies

Question 21

Open science and the cultural shift

Answer 21

Pre-registration (prevents cherry-picking), Publication of

Registered Reports, Open Data and Open Resources (aids replication).

Question 22

Institutional Change and cultural shift

Answer 22

The reward structure in academia has served to discourage replication. Promotions, pay raises, and prestige—through the quantity of their research.

Replications are typically discouraged because they do not represent original thinking. Instead, academics are rewarded for flashy studies that are highly cited and given prominence in media reports.

Question 23

Journal Policies and cultural shift

Answer 23

Some journals do not publish straight replications – although
this is changing, open data required, larger participant numbers, effect size required, and supplementary material for detailed methods.

Question 24

What does a failed replication tell us?

Answer 24

A single failed replication tells us almost nothing about which study is correct. Instead, it is the accumulation of evidence — paying close attention to the specific conditions that might be relevant to produce
the finding.

Question 25

Dissemination of replication studies

Answer 25

The fact that replications, including failed replication attempts, now have outlets where they can be communicated to other researchers is encouraging.