Week 5 - Open Data, Reproducibility and Replicability

Question 1

What is verifiability

Answer 1

A statement is meaningful when it can be verified empirically

Question 2

What is the induction problem

Answer 2

To establish a law like “All swans are white” we must observe all swans

Question 3

What is falsifiability

Answer 3

a statement is a valid theory if it makes predictions that can be tested, and can be falsified by a counterexample

Question 4

What is a non-falsifiable theory

Answer 4

“One day there will be a human that can breathe underwater”

Question 5

What is a Desideratum

Answer 5

in empirical work, we want to connect observations/measuerments with a falsifiable hypothesis or theory

Question 6

What is Hypothesis testing

Answer 6

We need to establish a null hypothesis and an alternative hypothesis for falsifiability

Question 7

What is transparency

Answer 7

In the ideal world, a study is fully transparent, in terms of what hypothesis is being challanged, what methodology was used, and what results were obtained.

Question 8

What is Open data

Answer 8

All data should be available in order for other researchers to evaluate the study, or reuse materials

Question 9

Open data is required or not required?

Answer 9

required in some academic journals

Question 10

What could be the reasons why data is not shared?

Answer 10

1. No time
2. NO access
3. Privacy
4. Propriotery data (companies dont want to share their data)

Question 11

how to combat data not being shared (no open data)

Answer 11

enforcing open data as a journal, peer review practice
Open data is necessary but not sufficient to guarantee good research

Question 12

What is replicaiblity

Answer 12

The ability of a researcher to duplicate the results of a prior study if the same procedures are followed but new data is collected.

Question 13

What is reproducibility

Answer 13

The ability of a reasearher to duplicate the results of a prior study using the same maternals as were used by the original investigator.

Question 14

What are research artifacts

Answer 14

Any concrete object that was used in the execution of a study and that is needed to reproduce the study. Examples:
* Paper/report
* Dataset
* Model
* Software

Question 15

What is a taxonomy of best practices on paper/report

Answer 15

peer review and checklists

Question 16

What is the taxonomy of best practices of the dataset

Answer 16

Data annotation

Question 17

What is the best taxonomy of best practices of Model and Software

Answer 17

ML best practices

Question 18

What are the problems we can encounter when doing ML paper

Answer 18

• The data: the way we set up our data into a splits may impact performance
• The network: many current models are DNN, meaning they consist of many layers with low interpretability

Question 19

How can we combat that. Give 2 methods

Answer 19

• Reproducibility from within: things researchers can do to increase the quality of their research.
• Reproducibility from outside: things reviewers should pay attention to

Question 20

What is generalization

Answer 20

your model’s ability to adapt properly to new, previously unseen data, drawn from the same distribution as the one used to create the model

Question 21

What is overfitting

Answer 21

The model performs very well on the training data but poorly on the validation data.

Question 22

what is underfitting

Answer 22

the model doesn’t perform well anywhere

Question 23

How to prevent loss hacking

Answer 23

Require authors to include loss statistics

Question 24

how to combat underspecification (not enough detail and not reproducible)

Answer 24

Provide the selected train,val,test set and ideally the code that we used to create the split. This makes sure that the report is reproducible

Question 25

How to combat label imballance (situation where the distribution of labels in the dataset is skewed)

Answer 25

Make sure that the relevant statistical properties of the intended splits are the same.

Question 26

What is cherry-picking

Answer 26

We could chose accidentally or not very favourable seed values

Question 27

how to combat cherry picking

Answer 27

Seed-averaging A simple solution is to average performance results over multiple runs with different seeds

Question 28

What could be the problem with classification metric like a contingency table

Answer 28

Table could be imballanced.

Question 29

What metrics are prefered when the data is unballanced in a contingency table?

Answer 29

Precision and Recall

Question 30

give accuracy function

Answer 30

Acc=correct predictions / # items

Question 31

Give Precision function

use ham and spam example

Answer 31

Prec = correctly marked as spam / marked as spam

Question 32

How can we mix precision and recall

Answer 32

We can define a weighted mix of precision and recall using Fb score

Question 33

What are Review Checklists

Answer 33

Review checklists in research methods are systematic tools used to evaluate the quality, rigor, and completeness of a research study. They ensure that all necessary aspects of the research process are addressed, including study design, data collection, analysis, and reporting. These checklists help maintain consistency and transparency, aiding in the replication and validation of research findings

Question 34

What are the two issues why we need review checklists

Answer 34

Central issue: in practice, reviewing takes palce on a pro bono basis, little time is available for reviewing
Reviewing vs reproduction: There is not always time to reproduce or replicate a given study

Question 35

What is the name of system where authors and reviewers are aware of best practices

Answer 35

Review checklists

Question 36

What are the 5 questions asked in the review checklist

Answer 36

1. General content: contributions, intro, rq
2. Scientific Artifacts: referenced? lincence?
3. Computational Experiments: environment described, detailed results?
4. Human participants: demographic, recruitment
5. AI Assistants: use of ai

Question 37

What is Academic Sin

Answer 37

Use of plagiarism. Copying text or data from other researchers and pretending it is your own.

Question 38

What are code licenses

Answer 38

They might lead to lawsuits

Question 39

What is a research proposal

Answer 39

1. Motivation: main idea
2. Application: thesis, for funding
   There are different types of proposals: reprodude/replicate a study, propose new framework

Question 40

What is the structure of a research proposal. Give 6 points

Answer 40

1. Background/context
2. Research question
3. Contributions
4. Methodology
5. Planning: timeline
6. Resources

Question 41

How to set up a good proposal?

name 3 possible things

Answer 41

1. Get creative
2. Write idea first
3. Be SMART

Question 42

Name the SMART attributes

Answer 42

1. Specific
2. Measurable
3. Achievable
4. Relevant
5. Time-bound