Effect Sizes, Power, New Stats

Question 1

Effect Sizes

Answer 1

An effect size is an objective measure of the magnitude of an observed effect.

if you pull on it, it gets significantly longer‘ vs
‚if you pull on it, it gets 15% longer‘

Question 2

Raw Effect Size

Answer 2

measures express the effect in the same metric used in the particular study (e.g., RT, points on a rating scale).

Example: Participants were faster to respond to words that were preceded by a related gesture (M = 681 ms, SD = 30) than words that were preceded by an unrelated gesture (M = 696 ms, SD = 28). This difference (M = 15 ms, SD = 32, 95% CI = [3.54 26.45]) was statistically significant (t(29) = 2.93, p = 0.006).

Question 3

Standardized Effect Size

Answer 3

can be used to quantify the magnitude of an effect over a range of studies (e.g., for sample size calculation or quantitative meta-analysis).

Examples for standardized effect size measures include
Cohen‘s d

Question 4

Cohen‘s d

Answer 4

Cohen‘s d is simple, intuitive and straightforward, especially if you are comparing two groups or two conditions

It is defined as the mean difference between the two groups, divided by the pooled standard deviation

Question 5

Raw effect size benefits

Answer 5

Useful to assess whether observed effect (considering both point estimate and interval estimate) is of practical significance

Raw effect sizes indicated on a widely used outcome variable (e.g., profit in £, depression score on BDI) enable the reader to intuitively appreciate the magnitude of the observed effect

Not so useful when making comparision between studies that have used different outcome measures

Question 6

Standardized effect size measures

Answer 6

Standardized effect sizes are useful whenever one tries to quantify the size of an effect across studies (when different outcome measures are used; e.g., BDI vs. another depression questionnaire)

There are conventions for what consitutes a small, medium and large effect

Effect sizes can be transformed from one measure to another (e.g., from Cohen‘s d to r; see PDF on effect size conversion on canvas)

Question 7

Type 1 error

Answer 7

False Positive, Rejects the null hypothesis as true when there isn’t an effect in reality

Question 8

Type 2 error

Answer 8

False Negative, Accepts the null hypothesis as false when there is an effect in reality

Question 9

Power

Answer 9

Power is a probability value between 0 and 1

It represents the sensitivity of an experiment to detect an effect of a specified size (i.e., the probability of a study to detect an effect of size x if an effect of size x, or greater, exists in the population)

Question 10

a-priori sample size calculation

Answer 10

If we know the (i) power we are aiming for, (ii) the population effect size, and (iii) α, then we can calculate the required sample size

The effect size in the population (μ0 − μ1)

The sample size (N)

The probability level at which we accept an effect as being statistically significant (, typically set to 0.05)

The power of the test to detect an effect of the specified effect size (1 − )