23-Reproducibility Flashcards
Why most studies are false?
- Large studies are more likely to yield true results yet we publish studies with small sample sizes
- Willingness to publish small effect sizes; With a smaller effect size, findings are less likely to be true.
- Flexibility in designs, definitions, outcomes & analysis reduce chances that results will be true.
- Fishing expeditions to generate new hypothesis or explore unlikely correlations
- Financial & other interests & prejudices reduce the likelihood that results will be true.
- Competition in research to produce positive findings, especially in hot fields
What does a small p value represent?
Typically a small p-value is used to justify rejection of the null (implies a real effect)
Explain the Amgen study
- Past ~15 yrs, efforts to characterize genetic alterations in human cancers = increased understanding of molecular mechanisms
- Hope this would translate to more effective drugs yet research –> clinical success very low
- Why??? Inherently complex nature of disease, sub-optimal models (cells, mice models), etc.
- Also, quality of pre-clinical data which helps ID new biological targets though we assume these to be ”truth”
What is the Amgen reproducibility effort?
• Between 2002 and 2012, Amgen was not able to reproduce 47 out of 53 seminal publications
- Spectrum of irreproducibility
- Data not reproduced by original researchers in own lab
- Specific data reproduced but not overall findings
- Single, non-representative experiment reports
- Impact?
- Wasted effort –time, $$$, opportunity cost, etc;
- Findings spur new field with 100s of secondary papers
- Clinical studies launched
What is the effect of reproducibility crisis?
Affects all scientific disciplines
•Thus affects society
•Distorts public policy and public expenditures (e.g., public health, climate science)
•Financial consequences ($28B/yrin US alone on irreproducible preclinical research into new drugs)•Distorts public trust in science
What are the 13/40 Ideas of the reproducibility crises?
- Statistical standards
- Data Handling
- Research practices
- Pedagogy
- Universities
- Professional Associations & Journals
- Scientific Industry
- Private philanthropy
- Government funding
- Government regulation
- Federal legislation
- State / Provincial Legislation
- Gov’t Staff & Judiciary
Explain the 1. statistical standards
- Avoid making decisions solely on the p-value
- Be more rigorous and use p<0.01 vs. p<0.05
- Present confidence intervals to better convey the range in which a variable most likely falls
- The p-value was never intended to be a substitute for scientific reasoning
Explain the 2. data handling
- Researchers should make their data available
* Researchers should use born-open data (i.e., open access repository that time-stamps data when created and updated)
Explain the 3. research practices
- Researchers should pre-register their protocols, filing in advance with an appropriate organization (e.g., journal, scientific society, government agency)
- Adopt standardized schemes to outline methods and materials
Explain the 4. pedagogy
- Fields that rely too heavily on statistics (i.e., like ours) to draw conclusions should better educate on the mis-use and mis-understanding
- Teach more holistic approaches as well as reasoning approaches to analyze data
- Integrate more into high school and college math and science classes, especially limits to certainty that statistics can provide
Explain the 5. universities
- When Professors go up for tenure, they should be required to adhere to best-practices for research methods
- Statistics 101 (survey level) into core curricula
Explain the 6. professional associations and journals
- Establish regular evaluation of disciplinary norms
- Journals should make peer review process even more transparent and rigorous
- Journals should only publish pre-registered studies
- All disciplines should establish a journal devoted to publishing negative findings
What is the WEF Code of Ethics?
- engage with the public
- pursue the truth
- minimize harm
- engage with decision making
- support diversity
- be a mentor
- be accountable
Explain the 7. scientific industry
- Industry should advocate practices that minimize irreproducible research
- Work with academic to formulate standard research practices and protocols to promote reproducible research
Explain the 8. private philantropy
- Fund scientists’ effort to replicate earlier findings
- Fund researchers who strive to develop better methods
- Funding university chairs in “reproducibility studies”
- Establish a prize for most significant negative result in various disciplines
- Improve journalism that continues to uncover the reproducibility crises
