Open And Reproducible Science Flashcards
Average (and weighted) replication rate in psychology
Avg = 52 %
Weighted Avg = 45 %
What started the replication crisis in psychology?
Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect by Daryl J. Bem
What is Bem‘s problem?
Extraordinary claims require extraoridnary evidence
What is Bem‘s Hypothesis?
That there exists anomalous retroactive influences of some future event on an individual’s current responses, whether those responses are conscious or nonconscious, cognitive or affective
what was so puzzling about Bem’s experiment?
Other than the topic, Bem’s experiments were completely standard, if not better, in the field (at the time).
what is p-hacking?
with enough choices, there will always be one path which leads to a significant effect even in the absence of a true effect
what is type 1 error extremely sensitive to?
questionable research practices (QRP)
what is Generous Grounding?
talking about a significance when there is basically none
what is HARKing?
hypothesizing After the Results are Known
how is excluding data bad?
Very bad = not saying anything
Bad = Random outlier criteria
Okay = Outlier Criteria based on past studies
Proper Science = Pre-register outlier criteria
what is Stopping Data Collection Early?
as the name says
what is Data Peeking?
checking the results of an ongoing A/B test with the intent to stop it and make a decision or inference based on the observed outcome
what is the file drawer problem?
selective reporting
what does open science include (6)?
open data, open source, open methodology, open peer review, open access, open educational resources
what are the 3 labels to show open science?
preregistered, open data, open materials
what is pre-registration?
the act of registering a study’s methods (e.g., IVs, DVs, Statistical Analysis Plan, Sample Size, etc.) before running your experiment
what is open data?
The act of making collected data freely available (usually under a CC license).
what is open materials?
The act of making experimental materials (e.g., stimuli, programs, tools, etc.) freely available.
