Confirmatory Research Flashcards
Wagenmakers et al. (2012)
Factors that make psychological research challenging
Empirical data are noisy, formal theory is scarce, variables of interest are latent
Confirmation bias
Confirmation bias operates in at least three ways. First, ambiguous information is readily interpreted to be consistent with one’s prior beliefs;
second, people tend to search for information that confirms rather than disconfirms their preferred hypothesis;
third, people more easily remember information
that supports their position
hindsight bias
the tendency to judge an event as more predictable
after it has occurred
biases and pressures leading to bad research
confirmation bias (seeking to confirm ones main hypothesis), hindsight bias, pressures to publish
replicability rates
only 50-11% of research in biomedical science is replicable
examples of bad methods (4)
cherry picking significant variables, cherry picking significant experiments, transformations of data, rely on one-sided p-values, post-hoc hypothesis
The core problem with research that does not distinguish between confirmatory and exploratory
the statistical law that, for the purpose of hypothesis testing, the data may be used only once. The stats will go wonky (type I and II error). A data set cannot be used to test the hypothesis that it helped to suggest.
Double dipping and its consequences
Using a data set to come up with an hypothesis and to test it. Type I error rates will be inflated and p values can no longer be trusted.
Confirmatory nature of research in real life
All research is on a continuum from exploratory to confirmatory. Depends on initial outcome of experiment, clarity of research question, amount of data collected, a priori beliefs of researcher, etc.
exploratory research vs. bad research
exploratory research is not bead research, as long as it is clearly labeled exploratory
Two stages in good science
first an exploratory stage where the research can do as she pleases, then a confirmatory stage, where all manipulations and tests must have been preregistered before collecting the data.
drawbacks of p-value, advantages of bayes factor
p-value cannot quantify evidence in favor of null-hypothesis, is sensitive to optional stopping, tends to overestimate the support in favour of the alternative hypothesis. Bayes factor tells us that the null-hypothesis is x-times more likely than the alternative hypothesis.
