Lecture 2Probability, power & effect size, t-tests & confidence intervals

Question 1

Statistical Power

Answer 1

Statistical Power:
Probability to reject the null hypothesis when it’s not correct / to find an effect when there really is an effect.
Ideally 80% – but depends on the sample size and effect size.
This equates to a 20% chance of a Type 2 error (thinking there are no differences when there are)
You can calculate your power based on sample size and expected effect.
You can calculate the sample size you need to have a certain power (given an expected effect size).

Question 2

Significance

Answer 2

Statisticalsignificance:
probability that the observed difference between two groups is due to chance. It is dependent on your sample size.

Effect size
how big the effect/difference is between your groups/conditions. It is independent of sample size.
E.g. Cohen’s (1988) d
d = 0.3  small effect
d = 0.5  medium effect
d = 0.8  large effect
You could have a large effect which is not significant and small effects which are significant.

Examples:
You find a significant difference between the IQ of Teachers and Researchers: Teachers’ IQ = 111, Researchers’ IQ = 114. That’s probably a tiny effect and mostly due to a massive sample size.

Question 3

Effect Size - Formula

Answer 3

(Look at Powerpoint)

Question 4

Sample Size and Errors

Answer 4

The size of your sample influences your test statistic:

Test statistic = effect size x sample size

Large samples but small effects can yield significant results! (Type 1 error?)

Small samples but large effects can yield non-significant results! (Type 2 error??)

Question 5

How to avoid this?

Answer 5

Power analysis:
How many participants do I need? OR What is my power?
Calculate an effect size from your data (or previous similar data from publications) using the means and standard deviations of your groups.
Select a power for your test (usually .80).
Use power tables to see how many participants you need to reach significance.
Too many participants: Danger of Type 1 error
Too few participants: Danger of Type 2 error

Question 6

Power Tables

Answer 6

(Look at Powerpoint)

Question 7

Confidence intervals (CI)

Answer 7

Essentially CIs tell us the range of scores that we can be 95% certain that a population mean will be in
This is extrapolated from our data and takes into account mean scores, variance, and sample size
Basically we know that if scores are normally distributed we can be 95% certain that a mean for the whole population will be +1.96 - -1.96 standard deviations from the mean

Question 8

Why Bother?

Answer 8

Another way to analyse results beyond p values
If 95% CIs for two (or more) conditions overlap (as for bottom left) it indicates that population scores for those conditions probably won’t be different
If 95% CIs for two (or more) conditions don’t overlap (as bottom right) it suggests population scores for those conditions probably will be different

Question 9

Reporting 95% CI ranges

Answer 9

(Look at Powerpoint)

Question 10

T-test

Answer 10

A t-test is a parametric equivalent of the Wilcoxon and Mann-Whitney U tests you covered last year
They tell us if the differences in scores between two conditions are statistically significant or not
Data need to be at least interval/ratio level, normally distributed, have no outliers, and homogenous in variance (parametric assumptions)
As with last year’s tests you need to identify differences between conditions (and their direction) with descriptive statistics to interpret

Question 11

Formula of T-test

Answer 11

(Look at Powerpoint)