Week 5 - Open Data, Reproducibility and Replicability Flashcards
What is verifiability
A statement is meaningful when it can be verified empirically
What is the induction problem
To establish a law like “All swans are white” we must observe all swans
What is falsifiability
a statement is a valid theory if it makes predictions that can be tested, and can be falsified by a counterexample
What is a non-falsifiable theory
“One day there will be a human that can breathe underwater”
What is a Desideratum
in empirical work, we want to connect observations/measuerments with a falsifiable hypothesis or theory
What is Hypothesis testing
We need to establish a null hypothesis and an alternative hypothesis for falsifiability
What is transparency
In the ideal world, a study is fully transparent, in terms of what hypothesis is being challanged, what methodology was used, and what results were obtained.
What is Open data
All data should be available in order for other researchers to evaluate the study, or reuse materials
Open data is required or not required?
required in some academic journals
What could be the reasons why data is not shared?
- No time
- NO access
- Privacy
- Propriotery data (companies dont want to share their data)
how to combat data not being shared (no open data)
enforcing open data as a journal, peer review practice
Open data is necessary but not sufficient to guarantee good research
What is replicaiblity
The ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data is collected.
What is reproducibility
The ability of a reasearher to duplicate the results of a prior study using the same maternals as were used by the original investigator.
What are research artifacts
Any concrete object that was used in the execution of a study and that is needed to reproduce the study. Examples:
* Paper/report
* Dataset
* Model
* Software
What is a taxonomy of best practices on paper/report
peer review and checklists
What is the taxonomy of best practices of the dataset
Data annotation
What is the best taxonomy of best practices of Model and Software
ML best practices
What are the problems we can encounter when doing ML paper
- The data: the way we set up our data into a splits may impact performance
- The network: many current models are DNN, meaning they consist of many layers with low interpretability
How can we combat that. Give 2 methods
- Reproducibility from within: things researchers can do to increase the quality of their research.
- Reproducibility from outside: things reviewers should pay attention to
What is generalization
your model’s ability to adapt properly to new, previously unseen data, drawn from the same distribution as the one used to create the model
What is overfitting
The model performs very well on the training data but poorly on the validation data.
what is underfitting
the model doesn’t perform well anywhere
How to prevent loss hacking
Require authors to include loss statistics
how to combat underspecification (not enough detail and not reproducible)
Provide the selected train,val,test set and ideally the code that we used to create the split. This makes sure that the report is reproducible
How to combat label imballance (situation where the distribution of labels in the dataset is skewed)
Make sure that the relevant statistical properties of the intended splits are the same.
What is cherry-picking
We could chose accidentally or not very favourable seed values
how to combat cherry picking
Seed-averaging A simple solution is to average performance results over multiple runs with different seeds
What could be the problem with classification metric like a contingency table
Table could be imballanced.
What metrics are prefered when the data is unballanced in a contingency table?
Precision and Recall
give accuracy function
Acc=correct predictions / # items
Give Precision function
use ham and spam example
Prec = correctly marked as spam / marked as spam
How can we mix precision and recall
We can define a weighted mix of precision and recall using Fb score
What are Review Checklists
Review checklists in research methods are systematic tools used to evaluate the quality, rigor, and completeness of a research study. They ensure that all necessary aspects of the research process are addressed, including study design, data collection, analysis, and reporting. These checklists help maintain consistency and transparency, aiding in the replication and validation of research findings
What are the two issues why we need review checklists
Central issue: in practice, reviewing takes palce on a pro bono basis, little time is available for reviewing
Reviewing vs reproduction: There is not always time to reproduce or replicate a given study
What is the name of system where authors and reviewers are aware of best practices
Review checklists
What are the 5 questions asked in the review checklist
- General content: contributions, intro, rq
- Scientific Artifacts: referenced? lincence?
- Computational Experiments: environment described, detailed results?
- Human participants: demographic, recruitment
- AI Assistants: use of ai
What is Academic Sin
Use of plagiarism. Copying text or data from other researchers and pretending it is your own.
What are code licenses
They might lead to lawsuits
What is a research proposal
- Motivation: main idea
- Application: thesis, for funding
There are different types of proposals: reprodude/replicate a study, propose new framework
What is the structure of a research proposal. Give 6 points
- Background/context
- Research question
- Contributions
- Methodology
- Planning: timeline
- Resources
How to set up a good proposal?
name 3 possible things
- Get creative
- Write idea first
- Be SMART
Name the SMART attributes
- Specific
- Measurable
- Achievable
- Relevant
- Time-bound
