CHAPTER TWO Bayes in Science Flashcards
What is the replication crisis in science?
A situation where many scientific studies are unable to be reproduced, raising questions about their validity.
Who was Diederik Stapel?
A social psychologist known for committing fraud by fabricating data in his studies.
What was the significance of Daryl Bem’s study published in 2011?
It claimed to provide evidence for psychic powers through a priming experiment that defied traditional expectations.
Define ‘priming’ in social psychology.
A technique where exposure to one stimulus influences a response to a subsequent stimulus.
What was the major twist in Daryl Bem’s priming experiment?
Subjects were primed after they had already made a choice, yet the priming still affected their responses.
What does a p-value of 0.01 indicate?
There is a 1% probability that the observed results would occur if the null hypothesis were true.
True or False: Daryl Bem believes his study results indicate that clairvoyance is real.
True.
What was the purpose of the ‘False Positive Psychology’ paper?
To demonstrate that standard statistical practices in science are flawed and can yield impossible results.
What was the unexpected finding in the ‘False Positive Psychology’ paper regarding age?
Listening to ‘When I’m Sixty-Four’ made participants appear nearly eighteen months younger.
Who is John Ioannidis?
A scientist who wrote about the prevalence of false findings in published research.
What is the misconception about p-values according to Dennis Lindley?
That a p-value of 0.05 indicates a 5% probability that the null hypothesis is true.
Fill in the blank: The practice of analyzing data in multiple ways to find significant results is known as _______.
p-hacking.
What is HARKing?
Hypothesizing After the Results are Known; formulating hypotheses based on observed data.
What is the competitive reaction time task (CRTT) used to measure?
Aggression, particularly in research into the psychological effects of video games.
True or False: The opponent in the CRTT is a real person.
False.
What drives the publish-or-perish culture in academia?
The need for researchers to publish regularly to secure tenure and career advancement.
What do most science journals prefer to publish?
Results that are interesting and novel, rather than all submitted studies.
What is the implication of publishing primarily ‘interesting and novel’ results in scientific journals?
It may lead to a bias where studies finding nothing significant are less likely to be published, skewing the literature towards positive results
This bias can create a false perception of the prevalence of certain phenomena, like psychic powers.
What threshold do many journals, especially in social sciences, use to determine if results are ‘interesting’?
p < 0.05
Results with p-values below this threshold are often published, while those above may be rejected.
What is the potential issue with the publication of studies reporting on psychic powers?
If many studies find significant results (p < 0.05) while most do not, it can lead to a misleading conclusion that psychic powers are real
This situation can result in a majority of published studies supporting a claim that lacks sufficient evidence.
What is p-hacking?
The practice of manipulating data to achieve statistically significant results
This can involve changing hypotheses or data analysis methods post hoc to obtain a desired p-value.
Who is Brian Wansink and what was he known for?
A food scientist at Cornell University known for publishing numerous studies on eating behavior, some of which were later retracted due to scientific misconduct
His work included studies that were criticized for questionable research practices.
What was the outcome of Wansink’s research practices?
Eighteen papers were retracted, seven received expressions of concern, and fifteen were corrected
Wansink resigned from Cornell in 2019 due to findings of scientific misconduct.
What did Brian Nosek’s Reproducibility Project investigate?
The ability to replicate results from one hundred psychology studies
The project highlighted the replication crisis in psychology, revealing that only 36 out of 100 studies could replicate their original findings.
What was the average effect size found in Nosek’s replication study compared to the original studies?
The effect sizes of the replications were, on average, half the size of the originals
This discrepancy raised concerns about the reliability of published psychological research.
What is a common misunderstanding about p-values among students and professionals?
Many believe that a p-value indicates the probability that the null hypothesis is true given the data
In reality, a p-value measures how likely the observed data would occur under a specific hypothesis.
What is the fundamental difference between frequentist and Bayesian interpretations of statistical significance?
Frequentists assess evidence based on long-run probabilities, while Bayesians incorporate prior probabilities to assess the likelihood of a hypothesis being true
This distinction affects how researchers interpret p-values and statistical findings.
Fill in the blank: A p-value of 0.02 indicates that there is a ______ chance of observing the data if the null hypothesis is true.
2%
However, this does not imply a 98% chance that the hypothesis is true.
True or False: A p-value of 0.05 means there is a one-in-twenty chance that the hypothesis is false.
False
The p-value does not provide the probability of the hypothesis being false; it indicates the likelihood of observing the data under the null hypothesis.
What is a significant flaw in frequentist models according to the text?
They don’t take into account prior probabilities.
What incentive do scientists have under frequentism when conducting experiments?
To come up with novel, surprising theories to get more buzz and notoriety.
What does Bayesian statistics provide to scientists?
A vehicle for skepticism and a way to express disbelief in a theory.
What does Aubrey Clayton suggest about hypotheses with low prior probabilities?
New data doesn’t significantly change skepticism towards them.
What was the extraordinary finding at CERN in 2011?
Neutrons arrived at a detector 60 billionths of a second sooner than expected.
What is the statistical significance of the CERN result?
A p-value of about 0.000000002.
What fundamental axiom of relativity would be violated if the CERN finding were true?
Nothing can break the speed of light.
What was the actual cause of the erroneous CERN result?
A fiber-optic cable in the clock system was not properly connected.
What is Lakens’ position on publishing startling but low-prior results?
Such results shouldn’t be hidden just because of their low prior probabilities.
What did Karl Popper argue about scientific hypotheses?
You can never prove a scientific hypothesis; you can only disprove it.
What does Lakens label the belief that one can determine the probability of a theory being true?
The statistician’s fallacy.
What does Lakens prefer over using prior probabilities when evaluating data?
Letting the p-values stand for themselves.
What should scientists focus on to improve scientific findings according to Lakens?
Obtaining better data and raising the standard for statistical significance.
What philosophical issue did David Hume raise?
The problem of induction.
What did Hume conclude about our expectation that the future will resemble the past?
It is based on custom and is ultimately unprovable.
How did Karl Popper respond to the problem of induction?
He argued that science relies on falsification rather than confirmation.
What example did Popper use to illustrate his point about scientific hypotheses?
The hypothesis ‘All swans are white.’
According to Popper, what happens if you see a swan that isn’t white?
The statement ‘all swans are white’ cannot be true.
What should scientists do when faced with a startling result?
Publish it despite low prior beliefs.
What does Lakens think about the Bayesian approach in choosing research topics?
He uses it implicitly by selecting hypotheses he believes are likely to be true.
Fill in the blank: Popper’s philosophy states that you can only _____ a scientific hypothesis.
disprove
True or False: Lakens believes that knowing which theories are true is achievable.
False.
What is Aristotelian logic’s stance on inferring universal laws from individual examples?
You can’t infer a universal law from individual examples.
What is a syllogism example demonstrating a logical fallacy regarding swans?
This is a swan, this swan is white, ergo all swans are white.
What does the statement ‘all swans are white’ imply if a non-white swan is observed?
The statement cannot be true.
How does Popper believe science advances?
By falsifying false hypotheses, not by confirming true ones.
What method does Popper suggest for discovering regularities?
Trial and error, conjecture and refutation, or learning from mistakes.
What term does Popper use to describe a theory that has stood up to severe testing?
Corroborated.
What is the difference Popper suggests between two hypotheses?
One has been severely tested, and one has not.
What is the critique of Popper’s model regarding the testing of theories?
It suggests that severely tested theories are not necessarily more likely to be true.
What do Bayesians like Wagenmakers argue against Popper’s falsification model?
Most scientific hypotheses are not straightforwardly falsifiable by a single counterexample.
In the context of acetaminophen, what do we claim about its efficacy?
Statistically, you are more likely to recover quickly from a headache if you take acetaminophen.
What is the significance of a p-value below 0.05 according to Fisher?
You tentatively behave as though a hypothesis is true.
What do frequentists reject that Bayesians encode as numbers?
Prior knowledge.
What fundamental question does Wagenmakers argue researchers are interested in?
How likely is this hypothesis to be true, given the data we’ve seen?
What did the authors of papers in Nature Human Behaviour say about their claims after seeing the data?
The data had made the claim more plausible than it was before.
What is a basic advantage that Bayesians have over frequentists?
They don’t have to leave data on the table.
What happens to data from previous studies in frequentist methods?
It gets forgotten with each new study.
What is the expected outcome of running an experiment twenty times with a p-value of 0.05?
You’d expect to see a significant result once.
What can happen if a researcher checks their data after each new entry in an experiment?
They can easily find statistically significant results even when there is no real effect.
How do Bayesians handle new data in relation to prior beliefs?
Each new data point moves their opinion less.
What is the implication of optional stopping for Bayesian analysis?
It can yield the same probabilities as waiting for all results.
What is a limitation of frequentist statistics regarding hypotheses?
They only reject or accept the null hypothesis without giving degrees of belief.
What is a common assumption about the null hypothesis in studies involving human populations?
The null hypothesis is always ultimately false.
What is a null hypothesis?
A hypothesis that assumes no effect or no difference exists in the population being studied.
Is the null hypothesis always true when studying human populations?
No, the null hypothesis is always ultimately false in human populations.
What happens when a large enough sample size is used in hypothesis testing?
You will likely be able to reject the null hypothesis.
What did David Bakan find in his 1968 study involving 60,000 subjects?
Every test he conducted came out significant, showing small but significant differences based on arbitrary criteria.
What was Paul Meehl’s observation regarding his survey of Minnesota high school students?
92 percent of the possible combinations of responses showed statistically significant correlations.
What is a key distinction between frequentist and Bayesian statistics?
Frequentist statistics requires deciding to reject or not reject the null hypothesis, while Bayesian statistics estimates the size of the effect and provides a probability distribution.
What does a probability distribution represent?
A graph of the possible outcomes of an event.
What is a prior distribution in Bayesian statistics?
The best estimate of the size of an effect before seeing the data.
What is a likelihood curve in Bayesian analysis?
The distribution of new data around an average after observing the data.
What is a posterior distribution?
The curve that combines the prior distribution and the likelihood to represent updated beliefs after seeing new data.
What is the implication of a tall and narrow prior curve in Bayesian analysis?
It indicates strong confidence in the prior estimate.
What does the term ‘Lindley’s paradox’ refer to?
The idea that a p-value of 0.05 can be evidence against a hypothesis under certain conditions.
What does a p-value of 0.05 signify?
It indicates a one-in-twenty chance of seeing results as extreme as those observed under the null hypothesis.
How can p-values be misleading in hypothesis testing?
A statistically significant p-value does not provide evidence for the null hypothesis, only that the data is surprising under that hypothesis.
What happens to the distribution of p-values if there is a real effect?
The p-values cluster closely around zero.
What does Kevin McConway suggest about p-values?
They are useful for indicating surprise under the null hypothesis but do not indicate the likelihood of the null hypothesis given the data.
Fill in the blank: The Bayesian approach allows you to update your _____ with new information.
probability distribution
True or False: Frequentist analysis can solve all problems in modern science.
False
What does a p-value of 0.05 indicate?
It indicates that the data is surprising given the null hypothesis.
It does not provide information about the likelihood of the null hypothesis being true given the data.
What is a common misconception among researchers regarding p-values?
Many researchers assume that a statistically significant result confirms their hypothesis.
This is not the correct interpretation of what a p-value represents.
Why is a p-value threshold of 0.05 considered weak evidence?
Because it can lead to false confidence in results that may not be statistically significant in practical terms.
For example, rolling two sixes on loaded dice can yield a p-value of 0.028.
What did a group of scientists propose regarding the standard level of statistical significance in 2017?
They proposed to lower the threshold for statistical significance to 0.005.
This would reduce the number of situations where Lindley’s paradox is relevant.
What is Lindley’s paradox?
It refers to the situation where a statistical hypothesis test can provide contradictory conclusions depending on the p-value threshold used.
What is Bayesianism concerned with in statistics?
It focuses on updating beliefs and assessing probabilities based on prior knowledge and new evidence.
This contrasts with frequentist approaches that rely solely on the data at hand.
What are the two types of uncertainty discussed?
- Aleatory uncertainty
- Epistemic uncertainty
What is aleatory uncertainty?
It is the uncertainty in an unknowable future, such as the outcome of a coin flip or a plane landing.
The term comes from the Latin word ‘alea’, meaning ‘a die’.
What is epistemic uncertainty?
It is the uncertainty regarding knowledge of a past event or fact, such as whether a coin landed heads or tails after being flipped.
How is probability viewed in a Bayesian framework?
Probability is seen as a measure of subjective belief based on available information.
What is a uniform prior?
It is a prior probability distribution that assigns equal probability to all possible outcomes, indicating total ignorance.
Frequentists often implicitly assume a uniform prior.
What problem is associated with uniform priors?
They can lead to paradoxes where total ignorance in one sense results in strong prior beliefs in another sense.
What did Harold Jeffreys suggest regarding prior distributions?
He suggested a U-shaped prior distribution, concentrating probability mass heavily at the extremes.
What is the significance of incorporating expert opinion into priors?
It allows for more informed decision-making rather than relying solely on uniform priors or data.
Experts can provide valuable insights that should be integrated into the prior probabilities.
How can the robustness of conclusions be checked in Bayesian analysis?
By testing different prior distributions to see if conclusions remain consistent.
What is meta-analysis in the context of Bayesian statistics?
It is the process of combining data from multiple studies to create a consensus, which is integral to Bayesian analysis.
True or False: Frequentists believe in using prior distributions based on previous studies.
False.
Frequentists typically do not use prior distributions in their analysis.
What is a thought experiment to determine if one is a Bayesian or a frequentist?
Flipping a coin and assessing the probability of it being heads without looking at the result.
What does it mean if someone answers ‘50 percent’ regarding the probability of the flipped coin being heads?
They are thinking in a Bayesian way, considering their subjective beliefs and available information.
What is the implication of believing that the probability of a past event is either 0 or 100 percent?
It indicates a frequentist perspective, focusing on the actual outcome rather than subjective probability.
What is a meta-analysis?
A method where scientists pool all studies on a subject to create a consensus by combining data like p-values and effect sizes.
What is a meta-analytic prior?
A current standard way of creating a prior distribution from existing data, which incorporates meta-analysis.
Why is using a proper, informative prior important in Bayesian statistics?
Not using existing data leads to less certain conclusions and inefficient use of new data.
What is a potential problem when using priors in Bayesian analysis?
Strange priors can skew results, especially if they misrepresent control group expectations.
How can one avoid skewing results with priors?
By using a mix of distributions for the prior, which down-weights historical information if there’s a significant difference.
What does the term ‘objective’ refer to in Bayesianism?
It describes a school of thought within Bayesian statistics that aims for neutrality in prior selection.
What analogy does Sophie Carr use to explain Bayesian and frequentist statistics?
She compares them to rugby league and rugby union, highlighting that both have pros and cons.
What is one criticism of frequentist methods mentioned in the text?
The p = 0.05 threshold is considered weak and could be improved.
What is a Registered Report?
A model where journals publish papers based on the strength of their methods before data collection.
What is the Octopus program?
A free repository for hypotheses, data, code, and methods, designed to change the incentive structure in research.
What is one reason why the debate between Bayesian and frequentist statistics persists?
Bayesian methods are becoming more common, but frequentist techniques remain the standard in scientific inquiry.
What is the Lindley paradox?
Under frequentist analysis, a statistically significant result can actually be evidence against the hypothesis.
How does Bayesian analysis address the issues of the replication crisis?
It forces comparison of likelihoods between competing hypotheses, making it harder to misinterpret results.
What advantage does Bayesian analysis have in vaccine studies?
It allows for the incorporation of prior data to reach significance thresholds more quickly.
What does Eric-Jan Wagenmakers say about the aesthetic of Bayesianism?
Bayesianism is aesthetically more pleasing due to its coherence and lack of internal inconsistencies.
What is utility in the context of decision theory?
It refers to the satisfaction or benefit derived from choosing one option over another.
Fill in the blank: Using a proper, informative prior in Bayesian analysis prevents _______.
leaving money on the table.
True or False: The frequentist method is the only standard for scientific inquiry.
False.
What is a downside of the academic journal model according to Marcus Munafo?
It is outdated and does not reflect the complexity of modern research.
What is one of the main problems with p-values in frequentist statistics?
They can lead to p-hacking and do not account for prior probabilities.
What is the underlying system for all decision-making according to the text?
Bayes
Bayes refers to Bayesian probability, a method of statistical inference.
What is the title of the recommended Coursera course?
Improving Your Statistical Inferences
The course is by Lakens and is available for free.
True or False: Frequentists cannot estimate effect size within their framework.
False
Frequentists can estimate effect size and perform equivalence testing.
What does equivalence testing allow you to determine?
Whether an apparent effect is big enough to pay attention to
Equivalence testing is a method used to assess if the effect size is practically significant.
Fill in the blank: Rejecting or accepting the null hypothesis isn’t _______.
final
This indicates that statistical conclusions can be revisited.
What is a criticism mentioned regarding the terminology used by statisticians?
They use terms that have different technical meanings from everyday language
This creates confusion, similar to the term ‘significance’.
What is the implication of the statement about frequentists not being idiots?
Frequentists have considered alternatives and developed methods within their framework
This acknowledges the validity of frequentist approaches.
What is a key difference between Bayesian and frequentist approaches as suggested in the text?
Bayesian methods incorporate effect size estimation into the system, while frequentist methods do so ad hoc
This highlights a structural difference in how each approach handles statistical inference.
