Week 3 - Empirical Cycle Flashcards
Name 5 steps in the emprical cycle
- Observation
- Induction
- Deduction
- Testing
- Evaluation
What is observation?
- Question, obstacle, idea
What is induction step?
- develop general rule or theoretical framework based on specific instances
- leaf of faith
What is the deduction step?
- From a general rule or theory we infer a testable, falsifiable hypothesis with specific predictions
What is the testing step?
- Conduct a study to test the prediction
- collect data
- anaylze data
What are two types of studies?
- Correlational
- Experimental
What is the correlational study?
Correlational: measuring variables to see how they relate to each other
What is an experimental study?
Experimental: Manipulate variables, random assignment to conditions.
* should have high internal validity: the extent to which one draws accurate conclusions about cause-effect relations
What is the goal of random sampling
The goal is to have high external validity: extent to which findings are generalizable
What is Evalutation step?
- What did we learn about our theory?
- reflect on theory
- possible follow-up work
What are the three types of fraud?
- Plagiarism
- Falsification
- Fabrication
Name three categories of QRPs
- Inappropriate publication practices
- Messing up the empirical cycle
- P-hacking
What is salami-slicing
Diving research up in smallest publishable units.
Selectively reporting experiments that ‘worked’
How to mess up an empirical cycle
Reporting an unexpected finding as having been predicted from the start (HARKing)
Deduction -> Testing -> Deduction
Name 6 p-hacking
- Rounding down p-values
- Adjusting outlier criteria
- Selecting levels of the independent variable
- Selecting multiple dependent variables
- Adding/removing covariates
- Sequential testing with optional stopping
What is sequential testing with optional stopping
- at some point you’ll always find a p-value less than .05 even if the null hypothesis is true
What is p-hacking in machine learning terms
P-hacking can also mean performance hacking in machine learning
* cherry picking the most optimal performance
* Comparing to poorly implented baseline
* Fine-tuning on test data to beat benchmark
what is fabrication
Fabrication refers to making up data or results and recording or reporting them. It involves creating false information or experiments that were never actually conducted.
what is falsification
Falsification involves manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record. This can include altering or omitting data to misrepresent the results of the research
