Replication + peer review + multiple comparisons

Question 1

Why replication could be important

Answer 1

• Publication bias
  ○ Journals only publish significant results?
  ○ This is problematic, but unlikely to be the whole story
  ○ 20 experiments for 1 payoff seems very inefficient (1/20 times you’ll incorrectly reject the null)
• Multiple comparisons
  ○ Type I error rate – the probability of rejecting H0 when true.
  ○ Decision-wise error rate is 5% – Experiment-wise error rate depends on the number of tests
  ○ Two hypothesis tests
  § P(reject at least one H0|both H0 true)

Question 2

Researcher degrees of freedom (hidden multiple comparisons)

Answer 2

• Refer to the flexibility researchers have in designing, conducting, and analysing studies, which can lead to a multiple comparisons problem, even when researchers only perform a single analysis

○ Might trim the outliers
○ Increased false positive rates: Researchers might find statistically significant results even when there is no real effect in the population.
○ Overoptimistic results: The findings might appear more impactful or definitive than they should be, given the potential for bias introduced by the analytical choices.

allows researchers to to present exploratory analyses as confirmatory

Question 3

Why are confidence intervals better than p-values for guiding replication in a single study?

Answer 3

Confidence intervals show the range of plausible effect sizes and their precision, making it easier to assess consistency in replication.

Question 4

Confirmatory experiment

Answer 4

• Analysis is planned before data is collected
• Only tests that are directly related to a prespecified test are performed
• Can pre-register these analyses, to protect against researcher degrees of freedom
• Type I error rate is 5%

Question 5

Exploratory experiment

Answer 5

• Unplanned analyses performed on data set
• Usually many tests are performed
• Researcher degrees of freedom used
• The Type I error rate is not maintained at 5%

Question 6

What is the journal review process

Answer 6

• Action editors select reviewers based on expertise.
• Action editor reads paper and reviewer comments to decide: accept, accept with minor revision, reject but invite major revision, or reject.
• Reviewers and editors are unpaid; publishers hold copyright and sell journals

Question 7

Determining journal quality:

Answer 7

• Quality of editors and reviewers.
• Acceptance rate.
• Impact factor (citation rate).
• Reputation.
  Australian Research Council classification

Question 8

What are multiple comparisons in statistical designs?

Answer 8

Multiple comparisons occur when more than one comparison between means is tested in a study, increasing the chance of error

Question 9

What risk is associated with multiple comparisons?

Answer 9

They can inflate the probability of Type 1 errors (false positives).

Question 10

How does the experiment-wise error rate relate to the decision-wise error rate?

Answer 10

If the decision-wise error rate is set at α, the experiment-wise error rate will be greater than α when there are multiple comparisons.

Question 11

Why is controlling the experiment-wise error rate considered more conservative?

Answer 11

It reduces the overall chance of making any Type 1 error across all tests, offering stronger protection against false positives

Question 12

When is it appropriate to control the decision-wise error rate?

Answer 12

Only when all comparisons are independent of each other

Question 13

What makes statistical tests independent?

Answer 13

Tests are independent if they do not share means

Question 14

Are tests of main effects and interactions independent

Answer 14

Yes, main effects and interaction tests in factorial ANOVA designs are always independent.