questionable research practices Flashcards
Research cycle
Hypothesis
Design study
Conduct and collect
Analyse and test hypothesis
Interpret results
Publish
Questionable research practices
HARKING
P-hacking
publication bias
poor quality control
Low statistical power
Failure to control for bias
p-hacking
- Same as: data fishing, data dredging, data snooping
- Going more into the data than you should
- Doing multiple statistical tests and not adjusting for the fact you are doing that - not reporting that e.g. changing method…
Optional stopping
- What if you did analyses as you recruit participants and stop data collection if the results look good?
- Optional stopping - do data collection until it’s significant
Related term: Data peeking
Acceptable practices
- Assessing data quality—making a decision on data that’s independent to your research question
Remove flawed data of low data quality
Remove flawed data of low data quality
Uncorrected multiple testing
What if you use multiple measures and report the significant results, but don’t report all the measures/tests you used?
- Doing more than you’re disclosing and not correcting for that
- If you have a alpha of 0.05, you’re accepting that there’s a 1/20 chance that maybe you’ll get a significant result that is not a real true result
Harking
- HARKing: Hypothesizing After the Results are Known
- “HARKing is defined as presenting a post hoc hypothesis (i.e., one based on or informed by one’s results) in one’s research report as if it were, in fact, an a priori hypotheses.”
(Kerr, 1998, Pers Soc Psychol Rev) - Looking at the results and then make an hypotheses that would make sense
- “HARKing is defined as presenting a post hoc hypothesis (i.e., one based on or informed by one’s results) in one’s research report as if it were, in fact, an a priori hypotheses.”
Replication crisis
f you closely followed the methods of the published paper, how likely are you find an effect of a comparable effect size?
- How may this outcome relate to:
p, geographic, subfield of psychology, others factors?
- Would they be able to replicate your study and get the same results again
File drawer problem
- Positive (i.e., significant) results have, historically, been more likely to be published than negative results
what is the QRP associated with designing a study?
low statistical power
what is the QRP associated with generating and specifying hypotheses?
failure to control for bias
what is the QRP associated with conducting and collecting data?
poor quality control - not proper randomisation
what is the QRP associated with analysing data and testing hypothesis, and interpreting results?
p hacking
what is the QRP associated with publishing and conducting next experiment?
publication bias
why is a registered report not feasible?
slower process
