Week 3 - Empirical Cycle

Question 1

Name 5 steps in the emprical cycle

Answer 1

1. Observation
2. Induction
3. Deduction
4. Testing
5. Evaluation

Question 2

What is observation?

Answer 2

• Question, obstacle, idea

Question 3

What is induction step?

Answer 3

• develop general rule or theoretical framework based on specific instances
• leaf of faith

Question 4

What is the deduction step?

Answer 4

• From a general rule or theory we infer a testable, falsifiable hypothesis with specific predictions

Question 5

What is the testing step?

Answer 5

• Conduct a study to test the prediction
• collect data
• anaylze data

Question 6

What are two types of studies?

Answer 6

1. Correlational
2. Experimental

Question 7

What is the correlational study?

Answer 7

Correlational: measuring variables to see how they relate to each other

Question 8

What is an experimental study?

Answer 8

Experimental: Manipulate variables, random assignment to conditions.
* should have high internal validity: the extent to which one draws accurate conclusions about cause-effect relations

Question 9

What is the goal of random sampling

Answer 9

The goal is to have high external validity: extent to which findings are generalizable

Question 10

What is Evalutation step?

Answer 10

• What did we learn about our theory?
• reflect on theory
• possible follow-up work

Question 11

What are the three types of fraud?

Answer 11

1. Plagiarism
2. Falsification
3. Fabrication

Question 12

Name three categories of QRPs

Answer 12

1. Inappropriate publication practices
2. Messing up the empirical cycle
3. P-hacking

Question 13

What is salami-slicing

Answer 13

Diving research up in smallest publishable units.
Selectively reporting experiments that ‘worked’

Question 14

How to mess up an empirical cycle

Answer 14

Reporting an unexpected finding as having been predicted from the start (HARKing)
Deduction -> Testing -> Deduction

Question 15

Name 6 p-hacking

Answer 15

1. Rounding down p-values
2. Adjusting outlier criteria
3. Selecting levels of the independent variable
4. Selecting multiple dependent variables
5. Adding/removing covariates
6. Sequential testing with optional stopping

Question 16

What is sequential testing with optional stopping

Answer 16

• at some point you’ll always find a p-value less than .05 even if the null hypothesis is true

Question 17

What is p-hacking in machine learning terms

Answer 17

P-hacking can also mean performance hacking in machine learning
* cherry picking the most optimal performance
* Comparing to poorly implented baseline
* Fine-tuning on test data to beat benchmark

Question 18

what is fabrication

Answer 18

Fabrication refers to making up data or results and recording or reporting them. It involves creating false information or experiments that were never actually conducted.

Question 19

what is falsification

Answer 19

Falsification involves manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record. This can include altering or omitting data to misrepresent the results of the research