Module 4 - Power & Effect Sizes

Question 1

Statistical power

Answer 1

What is the probability that a study will detect an
effect when there is an effect there to be detected

Question 2

Sampling to Populations

Answer 2

So when we estimate samples from populations – we accept a probability of making an error
Typically we set this as .05 (or 5%) probability that we will say there is an effect where there isn’t one to be found
We want to be 95% sure that two (or more) samples have been drawn from different populations

Question 3

Alpha

Answer 3

The probability that we will reject the null hypothesis when we shouldn’t
That we say there is an effect when there isn’t one
This is our Type I Error
<5% (<.05) means a low chance we have made this error

Question 4

Statistical power

Answer 4

The likelihood that a study will detect an effect when there is an effect there to be detected
What is the probability of a correct decision of rejecting the null
hypothesis when it is false
Power = 1 – Probability of a false negative (Type II error)
Power = 1 – β

Question 5

Factors Affecting Statistical Power

Answer 5

*Alpha level
*Error variance
*Sample size
*Effect size

Question 6

How Sample Size affects Power

Answer 6

Sample size works in the same way that error variance works
As we test more people we are able to better describe a distribution
Our hypothetical distributions (based on our samples) gets smaller/narrower

Question 7

Effect Size

Answer 7

Effect size is the relative distance between our null and true distributions
This distance is measured in standard deviation units
An effect size of 0 (zero) would mean no difference between groups (a “perfect” null result)
Effect size increases as two or more groups become “more” different from each other
This can help tell us if differences are practically meaningful

Question 8

Effect Size Measurements

Answer 8

Main Effect (ANOVA)
- Eta Squared
- Omega Squared
Multiple Comparisons (Planned contrast or Post-hoc)
- r
- Cohen’s d

Question 9

ETA squared

Answer 9

• Used for main effect
• Small (.01); Medium (.09); Large (.25)
  n2 = SSbetween/SStotal

Question 10

Omega squared

Answer 10

• Used for main effect
• Most accurate measure of effect size for main effect
• Small (.01); Medium (.06); Large (.14)
  w^2 = SSb -(df b*MSw)/ SSt+MSw

Question 11

Effect Sizes for Planned-Contrasts

Answer 11

• r
• Used for follow-up tests
• Particularly useful for planned contrasts
• Small (.10); Medium (.30); Large (.50)
  r = Square root (t^2 / (t^2+df)

Question 12

Effect Size for Post-Hoc Tests

Answer 12

Cohen’s d
* Used for follow-up tests
* Can be used for Tukey’s post-hoc tests
* Small (.20); Medium (.50); Large (.80)

Question 13

Post-Hoc Tests - Cohen’s d

Answer 13

Step 1. Spooled = Square root of (n1-1)s1^2+(n2-1)s2^2 / n1+n2
Step 2. D = x1-x2/Spooled

Question 14

Cohen’s d - Reporting example

Answer 14

Post-hoc tests using Tukey HSD revealed that taking no drugs (M = 13.5) led to significantly higher rated Ikea build quality than taking Marijuana (M = 3.20, p < .001, d = 8.25) or LSD (M = 6.75, p < .001, d = 4.75). Furniture ratings after taking LSD were significantly higher than Marijuana (p < .001, d = 2.88). All effects were large.

Question 15

Calculating required sample size

Answer 15

Before running an experiment we want to ensure that if there is an effect present, that we will observe it (Power)
Part of that will rely on testing sufficient numbers of participants (see effect of Sample Size on Power)
We can use effect size and desired Power to estimate how many people we will need to test

Question 16

Calculating required sample size in other words…

Answer 16

In other words, by making predictions of what our null, and alternative hypothesis distributions will look like (i.e., Mean, SD), and using that to estimate effect size (difference between null and alternative), we can estimate how many people we will need to describe those distributions sufficiently, to meet a certain power specification
You might not need to do this in this course, but you may need this for your honours project
Free software known as G*Power

Question 17

Power and Effect Sizes Wrap-up

Answer 17

The probability that we will find an effect if there is one to be found is known as statistical power
The probabilities of making inferential errors are related to observed power
Certain factors affect power (e.g., sample size)
Effect sizes are standardised metrics of difference
For ANOVA we can use Eta Squared, or Omega Squared
For multiple comparisons, we have different effect sizes
We can calculate the sample size we would need to observe a certain effect (size) at a certain statistical power

Question 18

G* Power - Calculating Sample Size

Answer 18

We can determine our mean difference using different types of tests (including ANOVA)
If we are looking for a smaller mean difference between groups, we need a larger sample size
If we are looking for a larger mean difference between groups, we need a smaller sample size