Statistical Methods Exam Flashcards
Four key factors leading to poor reproducibility
Publication bias
Low power
P-hacking
HARKing
Are effects in published literature usually generalizable?
Tiers to journals
The preference has distorted the scientific literature
The studies like the bottom one is still in someone’s file drawer while the first one is in a prestigious journal
why does low power occur in many scientific articles?
Often sample sizes are small because resources are thin
are effect sizes usually .2 in psychology? why or why not?
yes overall, however, for individual studies, If we assume the population effect sizes are .2 , you need a lot of participants in order to have high power
- usually not possible for many studies
p-hacking
when I run a bunch of tests on data, then find one is significant, and report as if that was the only test I ran
- this inflates the type 1 error rate
- papers tha p-hack, do not account for this type 1 error inflation . You should report this so ppl can know that the type error is inflated
HARking
hypothesizing after the results are known
ppl actually sometimes advise to do this. but very unethical
- as if you have no prior predictions but upon seeing the results you write a paper as if you planned to look at the effect of what turned out to be significant
solution with p hacking and HARking
transparency
- ex: pre-registration
What is pre-registration for studies? what does it do
Changes order of classic publishing
Registered report: plan study, write up entire plan and submit that with hypothesis to journal, then reviewers advise you on changes, then you get an in principle acceptance, if you do everything in accordance you should get acceptance
Prevents bias against null, harking, p-hacking
The part that is preregistered is the main hypothesis (confirmatory)
Can still have extra exploratory work that is not pre-registered
