1- Reproducibility, Replicability and Robustness

Question 1

Why are these three notions important?

Answer 1

In assessing credibility of study findings

Question 2

What is replicability?

Answer 2

We replicate a study and expect to get the same findings

Question 3

What is Nosek et al’s definition of replicability?

Answer 3

Testing reliability of previous findings with different data

Question 4

What do the credibility of scientific findings depend partly on?

Answer 4

Replicability of supporting evidence

Question 5

Why is replication not always straightforward?

Answer 5

Difficult to determine what counts as same study/same outcome

Question 6

What is Nosek et al’s definition of reproducibility?

Answer 6

Testing reliability of previous findings using same data and same analysis strategy

Question 7

Why should the same results theoretically occur when findings are reproduced?

Answer 7

Someone is applying same analysis to same data

Question 8

What 2 reasons why reproducibility tests may fail?

Answer 8

1. Original analysis may not be repeated
2. Data/necessary software tools may not be available

Question 9

What is the most common problem of reproducibility?

Answer 9

Data availibility

Question 10

What is Nosek et al’s definition of robustness?

Answer 10

Testing reliability of previous findings using same data and different analysis strategy

Question 11

Why is fragility a risk factor for robustness?

Answer 11

For replicability and generalisability

Question 12

What are the three types of replication?

Answer 12

Direct replication, systematic replication, conceptual replication

Question 13

What is direct replication?

Answer 13

New study using same procedure, measures, study population as original features

Question 14

What is systematic replication?

Answer 14

Secondary features have been changed (eg. different stimuli order)

Question 15

What is conceptual replication?

Answer 15

Intentionally different from direct replication, examines validity and generalisability of original findings, similar but not the same

Question 16

What happened in the 1960s-70s?

Answer 16

Social psychologists started to doubt the validity of research

Question 17

When is an effect declared statistically significant?

Answer 17

Null hypothesis test of p<0.05

Question 18

What is the problem with the statistic significance criterion?

Answer 18

Encourages publication and analytic bias favouring significant results

Question 19

What was Strack et al’s (1988) study?

Answer 19

Study on facial feedback hypothesis, 92 US students, hold pen with non-dominant hand/teeth/lips, rate funniness of cartoons

Question 20

What did Strack et al find?

Answer 20

Cartoons funnier when holding pen between teeth (smile) than when holding with lips (pout)- confirms facial feedback hypothesis

Question 21

What was Wagenmakers et al’s (2016) study?

Answer 21

17 replications of same study as Strack et al- study pre-registered and N= 1,894

Question 22

What did Wagenmakers et al find?

Answer 22

Strack et al’s findings not replicated

Question 23

What 3 reasons are there why Strack et al’s findings may not have been replicated?

Answer 23

• Social norms
• Task was not very valid
• Instruction may be differently interpreted

Question 24

What 3 ways does Giner-Sorolla propose to address the replication crisis?

Answer 24

1. Publish both null and significant findings
2. Rigorous and transparent methodology
3. Pre-register hypotheses and studies

Question 25

What is the purpose of statistical power?

Answer 25

Tells you whether a sample is large enough to find an effect

Question 26

What does using a statistically incorrect sample size lead to?

Answer 26

Inadequate results

Question 27

Why do we need an effective sample size?

Answer 27

Enables efficient, high significance studies

Question 28

What is sensitivity?

Answer 28

How likely a study is to distinguish between actual effect and chance effect

Question 29

Why is statistical power generally set at 80%?

Answer 29

There is an 80% chance that the effect exists

Question 30

Why should we pay attention to the sample size?

Answer 30

Effect is unreliable if sample size is too low