Power and sample size

Question 1

What is a type 1 error?

Answer 1

• Type I error: the incorrect rejection of a true null hypothesis (a “false positive”).
• Alpha: an alpha level of .05 means that there is a 5% chance of determining that there is an effect, when actually the null hypothesis is true: i.e. a false positive or a type 1 error. Alpha level is the probability of making a type 1 error.

Question 2

What is a type 2 error?

Answer 2

• Type II error: is incorrectly retaining a false null hypothesis (a “false negative”)
• Beta: the probability of making a type 2 error.

Question 3

How is power defined?

Answer 3

• Statistical power is defined as the probability of correctly rejecting a false null hypothesis (H0). In other words, the probability of detecting an effect that is really there.
• More formally, power = 1 – β, where β equals the probability of making a Type II error.
• Power estimates are important to examine whether a completed statistical test had a fair chance of rejecting an incorrect H0 .

Question 4

How does effect size affect power?

Answer 4

• Effect size: the difference between the parameter values associated with the null and alternative hypotheses. A larger effect size increases power.

Question 5

How does alpha level affect power?

Answer 5

• Decreasing alpha decreases power, increasing alpha increases power but also increases chance of making a type 1 error.

Question 6

How does the variance of the distribution affect power?

Answer 6

Decreasing the variance of the distributions (ie increasing sample size cf central limit theorem) will increase power.
Infinite points have enough to make a perfect estimate. As we add more and more new sample points, the difference between the information we need to have a perfect estimate and the information we actually have gets smaller and smaller.

Question 7

How do 1/2 tailed tests affect power?

Answer 7

• Using a one tailed test increases power
o This is for the same reason that increasing alpha increases power i.e., when you used a 1 tailed test, you double the alpha at one end of the distribution (while you remove the alpha at the other end of the distribution).

Question 8

What do you need to calculate power?

Answer 8

We need to know or estimate the following ingredients:
o Effect size
 The difference between the two means, divided by the standard deviation of one of the groups (or the average of the 2 standard deviations).
 We can estimate effect size from: Previous research/ Meta-analyses or using conventional labels of effect size magnitude for d originally provided by Cohen.
o Sample size
o Significance level (α) and whether 1 or 2 tailed test used