Mindless stats

Question 1

Replication

Answer 1

Can we replicate the results form the study

Reproducibility is an essential for scientific progress and knowsledge + theory dev+ essental for application in real world

Question 2

Replication crisis

Answer 2

It involves many fields, including psychology
Failure to replicated results when implamenting the same study design

~36% of studies replicated when take same data and methods (based on whether p<0.05)
~47% studies replicated when using 95% of CI as statistical significance indicator

Question 3

Type I error

Answer 3

Incorrect acceptce of false positives

Question 4

Type II error

Answer 4

Incorrect acceptce of false negatives

Question 5

Why the results do not replicate? (4)

Answer 5

1. Sampling variability: no two samples are the same
2. Hidden moderators across studies
   * Different cultural contexts
   * Time in history
   * Demographic characteristics
   * Methods
3. Low statistical power
   * False positives in original study
   * False negatives in replication study
4. Questionable research practices (QRPs) e.g. finding meaning in noise

Question 6

Why do some researchers engage in QRPs?

Answer 6

There is a bias of publicating only novel and significant findings

Question 7

Issues with using p

Answer 7

A p-value is (sort of) a measure of how incompatible the data are with a specified statistical model, not e.g. if the hypothesis is true nor an effect size measure

The value is not based on anything real, it’s a made up threshold

When it’s not significant = we don’t know = it is not support for null hypothesis

If you can report effect sizes alongisde p (e.g. ANOVA, r, t-test)

Question 8

Statistical power

Answer 8

Probability of correctly identifying a false null hypothesis

Sample size: larger N, greater power to
detect weak effects
Effect size: larger effect size, greater power
to detect with small sample sizes

Question 9

A priori power analysis (sample size estimation)

Answer 9

Pre-specify necessary parameters to estimate how many participants you will need
* Parameters
* Type of analysis
* Type of power analysis
* Tails: one- vs. Two-tailed
* Expected correlation (be conservative)
* Power: aim for high power

Question 10

HARKing

Answer 10

Hypothesising After the Results are Known

Question 11

p-hacking

Answer 11

The practice of iteratively analysing your data in order to find significant results

Question 12

Difference between Exploration vs. Inference

Answer 12

Exploring: finding the structure in your data
* Can perform many analyses

Inference: using data to choose between different models
* Logic of p-values: 1 analysis to test a prediction

Question 13

95% Confidence Intrevals

Answer 13

They estimage a range where the correct value is

We have 95% confidence in stating that the unknown “true” mean lies within this interval range

This can provide valuable information about the lower and upper range of the likely effect
* Important when considering replications

If CIs do not overlap with 0, then the value is significantly different from 0

Question 14

Advantages of CI over p-values

Answer 14

• We don’t tend to make reasoning errors with CIs
• We see the full range - (un)certainty
• CIs provide a more reasonable prediction about future outcomes/measurement than p-values