Simmons et al. (2011) false positive psychology

Question 1

false positives, and why they are a problem

Answer 1

the incorrect rejection of a null hypothesis.

1. once they appear in the literature, false positives are particularly persistent. researchers have little incentive to do replications because journals do not like to publish null-results. failed replications are not conclusive.
2. waste resources by leading researchers astray, leads to ineffective policies, undermines credibility of field of research

Question 2

researcher’s degrees of freedom

Answer 2

decisions a researcher can make. For example: Should more data be collected? Should some observations be excluded? Which conditions should be combined and which ones compared? Which control variables should be considered? Should specific measures be combined or transformed or both?

Question 3

Problem with researcher degree of freedom

Answer 3

it is accepted to make these decisions during analysis, after obtaining the results, which increase the chance of type I errors.

Question 4

Reasons for exploratory behaviors

Answer 4

1. ambiguity on how to best make methodological decisions. Confirmation bias and hindsight bias.
2. the researcher’s desire to find statistical results.

-> Confirmation bias and hindsight bias.

Question 5

researcher degrees of freedom examined in simulation study (4)

Answer 5

flexibility in (a) choosing among dependent variables, (b) choosing sample size, (c) using covariates, and (d) reporting subsets of experimental conditions

Question 6

Results of simulation study

Answer 6

every single researcher degree of freedom could be used to increase the rate of significant results. Combining them led to even more significant results, see table in article or lecture slides.

Question 7

effect of optional stopping

Answer 7

in general, we have the intuition that an effect that is insignificant at a low sample size will not turn significant when adding observations. It turns out this is not true. Optional stopping can be effectively used to generate false positives.

Question 8

the six requirements for authors of good research

Answer 8

Requirements for authors

1. Authors must decide the rule for terminating data collection before data collection begins and report this rule in the article.
2. Authors must collect at least 20 observations per cell or else provide a compelling cost-of-data-collection justification.
3. Authors must list all variables collected in a study.
4. Authors must report all experimental conditions, including failed manipulations.
5. If observations are eliminated, authors must also report what the statistical results are if those observations are included.
6. If an analysis includes a covariate, authors must report the statistical results of the analysis without the covariate.

Question 9

Why should authors determine a rule on when to terminate data collection

Answer 9

To prevent type I errors through convenient stopping once a significant result is reached. The justification of the rule is secondary, most important is that this rule actually exists.

Question 10

Why should there be at least 20 observations per cell?

Answer 10

To have a minimum required amount of power. It is a general assumption.

Question 11

Why must authors list all variables and all experimental conditions?

Answer 11

To prevent them from selecting a convenient subset. The reader can get an idea of the researcher’s degrees of freedom.

Question 12

Why must the author report the results without excluding outliers? Why does he have to do the same with covariates?

Answer 12

to pressure him to carefully exclude cases and to make things transparent for the reader. Same reasoning goes for the covariates.

Question 13

What are the guidelines for reviewers?

Answer 13

1. Reviewers should ensure that authors follow the requirements.
2. Reviewers should be more tolerant of imperfections in results.
3. Reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions.
4. If justifications of data collection or analysis are not compelling, reviewers should require the authors to conduct an exact replication.

Question 14

Problems not addressed by the requirements of the article

Answer 14

file-drawer problem: not reporting insignificant experiments.
Enforcement of the disclosure requirements: researchers are not incentivized to disclose everything. Article argues that the reviewers are responsible for enforcement.

Question 15

Why correcting alpha levels is not a solution to the researchers degree of freedom problem

Answer 15

We cannot know how many degrees of freedom exist in a given situation. Creates ambiguities.

Question 16

Bayesian statistics as a nonsolution

Answer 16

adds additional degrees of freedom by adding more possible statistical techniques. Furthermore, requires decisions on assumptions which leads to more ambiguity.

Question 17

Conceptual replications as a nonsolution

Answer 17

are misleading because conceptual replications,

in contrast to exact replications, do not bind researchers to make the same analytic decisions across studies

Question 18

Posting materials and data as a nonsolution

Answer 18

is a good idea in general, but does not help solve the problem because it would be to effortful for readers to control the researcher just based on the raw data.