23-Reproducibility Flashcards
Why most studies are false?
- Large studies are more likely to yield true results yet we publish studies with small sample sizes
- Willingness to publish small effect sizes; With a smaller effect size, findings are less likely to be true.
- Flexibility in designs, definitions, outcomes & analysis reduce chances that results will be true.
- Fishing expeditions to generate new hypothesis or explore unlikely correlations
- Financial & other interests & prejudices reduce the likelihood that results will be true.
- Competition in research to produce positive findings, especially in hot fields
What does a small p value represent?
Typically a small p-value is used to justify rejection of the null (implies a real effect)
Explain the Amgen study
- Past ~15 yrs, efforts to characterize genetic alterations in human cancers = increased understanding of molecular mechanisms
- Hope this would translate to more effective drugs yet research –> clinical success very low
- Why??? Inherently complex nature of disease, sub-optimal models (cells, mice models), etc.
- Also, quality of pre-clinical data which helps ID new biological targets though we assume these to be ”truth”
What is the Amgen reproducibility effort?
• Between 2002 and 2012, Amgen was not able to reproduce 47 out of 53 seminal publications
- Spectrum of irreproducibility
- Data not reproduced by original researchers in own lab
- Specific data reproduced but not overall findings
- Single, non-representative experiment reports
- Impact?
- Wasted effort –time, $$$, opportunity cost, etc;
- Findings spur new field with 100s of secondary papers
- Clinical studies launched
What is the effect of reproducibility crisis?
Affects all scientific disciplines
•Thus affects society
•Distorts public policy and public expenditures (e.g., public health, climate science)
•Financial consequences ($28B/yrin US alone on irreproducible preclinical research into new drugs)•Distorts public trust in science
What are the 13/40 Ideas of the reproducibility crises?
- Statistical standards
- Data Handling
- Research practices
- Pedagogy
- Universities
- Professional Associations & Journals
- Scientific Industry
- Private philanthropy
- Government funding
- Government regulation
- Federal legislation
- State / Provincial Legislation
- Gov’t Staff & Judiciary
Explain the 1. statistical standards
- Avoid making decisions solely on the p-value
- Be more rigorous and use p<0.01 vs. p<0.05
- Present confidence intervals to better convey the range in which a variable most likely falls
- The p-value was never intended to be a substitute for scientific reasoning
Explain the 2. data handling
- Researchers should make their data available
* Researchers should use born-open data (i.e., open access repository that time-stamps data when created and updated)
Explain the 3. research practices
- Researchers should pre-register their protocols, filing in advance with an appropriate organization (e.g., journal, scientific society, government agency)
- Adopt standardized schemes to outline methods and materials
Explain the 4. pedagogy
- Fields that rely too heavily on statistics (i.e., like ours) to draw conclusions should better educate on the mis-use and mis-understanding
- Teach more holistic approaches as well as reasoning approaches to analyze data
- Integrate more into high school and college math and science classes, especially limits to certainty that statistics can provide
Explain the 5. universities
- When Professors go up for tenure, they should be required to adhere to best-practices for research methods
- Statistics 101 (survey level) into core curricula
Explain the 6. professional associations and journals
- Establish regular evaluation of disciplinary norms
- Journals should make peer review process even more transparent and rigorous
- Journals should only publish pre-registered studies
- All disciplines should establish a journal devoted to publishing negative findings
What is the WEF Code of Ethics?
- engage with the public
- pursue the truth
- minimize harm
- engage with decision making
- support diversity
- be a mentor
- be accountable
Explain the 7. scientific industry
- Industry should advocate practices that minimize irreproducible research
- Work with academic to formulate standard research practices and protocols to promote reproducible research
Explain the 8. private philantropy
- Fund scientists’ effort to replicate earlier findings
- Fund researchers who strive to develop better methods
- Funding university chairs in “reproducibility studies”
- Establish a prize for most significant negative result in various disciplines
- Improve journalism that continues to uncover the reproducibility crises
Explain the 9. government funding
- Fund scientists’ effort to replicate earlier findings
- Fund researchers who strive to develop better methods
- Prioritize funding for researchers who pre-register their plans and make data/methods open access
- Adopt new NIH Principles for funding reproducible research
- More funding to broaden statistical literacy
Scientific rigor (design)
- Foundation for achieving robust and unbiased results•Strict application of scientific method to design, method, analyses, interpretation, and reporting of results
- Standards? Sample size estimator? Randomization? Blinding? Replicates? Inclusion and exclusion criteria? Data analyses plan? Etc.
What are biological variables?
- Sex, age, weight, underlying health affect health and disease
- Variables often ignored –> incomplete understanding
- Explain how relevant biological variables are factored into research•Key now is sex
What are the questions to ask when authenticating resources?
- How do you ensure the identity and validity of your biological and chemical reagents?
- Do they differ lab to lab? Over time?
- Have varying properties that can influence data?
- Is equipment properly maintained and calibrated?
- Are protocols/SOPs documented and followed? Deviations noted and corrected?
- Are people properly trained, and is training documented?
- Are lab notebooks maintained and reviewed?
Explain the 10. government regulation
- Ensure new regulations needing scientific justification only use research that meets strict reproducibility standards
- Establish committees to determine which regulations are based on reproducible research
Explain the 11. federal legislation
- Pass a “Secret Science Reform Act” to prevent agencies from making regulations based on irreproducible research
- Strengthen Information Quality Act
- Fund programs to broaden statistical literacy
Explain the 12. State / Provincial Legislation
- Reform ‘K-12’ curricula to include courses on statistical literacy
- Use funding and oversight to encourage universities (CEGEP) to add and strengthen statistical literacy
Explain the 13. Gov’t Staff & Judiciary
- Government officials should hire trained staff in statistics and reproducible research to advise them on scientific matters
- Courts should ensure that sound science is used in judicial decision making
- Set approaches to overturn precedents based on irreproducible science
- Relevant courses in law school