Trustworthy Science Flashcards
What are the informational and economic definitions of open science and the difference between them?
Informational: full disclosure and improved evidence
Economic: available widely and for free
Difference: Info focuses on studies, while Eco focused access to it, paid or not.
Be able to describe the 6 reforms (DOPRPR) and how do each guard against selective reporting, HARKing, and/or p-hacking.
Disclosure statements
Stating that you disclosed all measures, manipulations, and exclusions in your study and the method of sample size determination. Protects against selective reporting as you state everything you will analyse beforehand, however it relies on honour.
Open data and materials
Making all materials in the study available online, allowing for other researchers to replicate. Protects against selective reporting an p-hacking.
Publishing null results
Protects against HARKing, p-hacking, and selective reporting, as you show you received non-significant results.
Registered reports
These work to promote null result publication. Protects against HARKing and p-hacking as you are publishing negative results
Preregistration
State what you are going to do and how before you do it. Protects against selective reporting, HARKing, and p-hacking.
Replication
Methodology is specified ahead of time as being exactly the same as the target studies
Know what equivalence testing does and why a non-significant results is not enough to say that no effect probably exists
Equivalence testing allows for control of error rates and to design studies based on desired levels of statistical power
You need multiple studies to say there is no effect
What kind of study characteristics predict whether an independent lab is likely to to be able replicate results?
Large sample sizes
P-values <.01
For the green open access and preprint-plus-post-publication models, know their characteristics
Green: authors work on their manuscripts and publish it without the publishers formatting, free to access for everyone
Preprint: early work of the authors paper made available online, before it has been peer reviewed.
What are the limitations of pre-prints?
Who will do the reviewing and ensure coverage of all pre-prints?
How much review is needed before a preprint can become accepted?
With PPPR (post publication review), how do comments become integrated with the paper? Who decides if a comment invalidates a paper?
What are some of the reasons data may not be shared?
Some data is proprietary
Confidentiality of participants
What are constraints on generality statements, and how do they relate to the conditions for replication, and to the problems of sample representativeness in psychology?
Constraints: original effect is not reliable, potentially a type 1 error. Could also be that the effect is reliable but only under conditions in one study but not another.
Relates to replication as researchers should disclose their research constraints, so other researchers can better attempt to replicate
Important in psychology representativeness as Psych relies heavily on WEIRD
How can preregistration improve the defensibility of practices such as one-tailed significance testing and sequential analysis?
Preregistration a one-tailed test increases statistical power and planned sequential trial analysis may provide an opportunity to stop data collection early.
