Resampling and Open Science

Question 1

What are resampling methods?

Answer 1

Sets confidence intervals and critical values of tests
Non-parametric
Relatively assumption free
Computationally demanding

Question 2

Bootstrapping vs Permutation

Answer 2

Both involve computing simulations
Bootstrapping samples with replacement
Permutation samples without replacement

Question 3

Simulating Null Hypothesis Distribution Permutation Tests

Answer 3

common statistical question usually comparing two groups

Question 4

Null Hypothesis

Answer 4

States that the group means are equivalent

Question 5

Alternative Hypothesis

Answer 5

states that the groups means are not equivalent

Question 6

P-value

Answer 6

Indicates probability of obtaining a difference in the mean of the two samples at least as extreme as observed at random, if the two samples did come from the same population

Question 7

Null Hypothesis Testing using Parametric Methods

Answer 7

Simulate or select theoretical null hypothesis sampling distribution
Determine where our observed test statistic lies within this distribution and the probability of it being observed if null hypothesis was true

Question 8

Null Hypothesis Testing using Permutation Tests

Answer 8

1. calculate the real difference between the means of two groups
2. simulate or select theoretical null hypothesis sampling distribution

Question 9

Simulating H0: Random Shufflying of Observations

Answer 9

Pool sample together then draw new groups from pooled sample
Repeat process many times ensuring it is not due to chance
Build a mean that should be true under the null hypothesis

Question 10

Bootstrapping

Answer 10

Resampling method for assessing statistical accuracy of an estimate
Own sample and treat it as the entire population assessing it better
does not work for small samples
Typically used to estimate quantities associated with the sampling distribution of estimates
Original sample is resampled then drawn from which may be repeated multiple times creating a bootstrapping sample

Question 11

Basic Ideas of Bootstrapping

Answer 11

Treat particular sample as the entire population
Repeatedly sample with replacement to generate samples
Analyse to get estimate

Question 12

How Science Should Work?

Answer 12

Thought to be a reliable way to answer questions about the world
Start with hypothesis
Collect data
Do statistics to test null hypothesis
Make conclusion based on the data

Question 13

Reproducibility Crisis in Science

Answer 13

Findings do not replicate
Most significant effects should replicate
97% of 100 papers reported significant findings but only 37% were significant in the replication study
Data should reproduce as repeat experiments find significant effects replicating - rare

Question 14

Publication Bias

Answer 14

Not all research is equally publishable - editorial bias, incentivised, increased likelihood of false positives published
Wrong incentive structures - academic success tied to ‘significant’ results, publish or perish
Distorts meta-analyses - bad for estimating effect sizes
Not publishing negative results skew meta-analyese

Question 15

P-Hacking

Answer 15

Actively searching for ‘something significant’ in data
Cherry-picking - experiments, subjects, stopping rules
Analysis of degrees of freedom
Variant of multiple comparisons problem
Focus on data giving significant findings and ignore those that do not inflate the chance of it occurring
Multiple comparisons unless controlled inflate probability

Question 16

HARKing

Answer 16

Hypothesis after results are known
Related to p-hacking
Hides the reality of multiple comparison problem

Question 17

Apophenia

Answer 17

Tendency to see patterns in random data

Question 18

Confirmation Bias

Answer 18

Tendency to focus on evidence that is in line with our expectations or favoured explanation

Question 19

Hindsight Bias

Answer 19

Tendency to see an event as having been predictable only after it has occurred

Question 20

Why do Findings Fail to Replicate?

Answer 20

When low-powered studies show significant effects, these will be overestimates

Question 21

Open Science - Open Hypothesis Testing and Analysis Decision Making

Answer 21

Pre-registration - publicly commit to your hypothesis and analysis pipeline before conducting study - treats p-hacking and HARKing
Registered report - coupled to publishing can also treat public bias

Question 22

Open Science - Open Analysis Tools and Data

Answer 22

Make all materials open access so other researchers can double check conclusions
Treats honest mistakes

Question 23

Open Science - Open Evaluation: Peer Review Published with Article

Answer 23

More information is always better

Question 24

Open Science - Open Access

Answer 24

Make the research outcomes (published articles) accessible to everyone