Effect size and p-values

Question 1

t-distribution as a null model

Answer 1

-null describes distribution we’d expect to see due to random noise if there was no true difference in our data
- This is conditional on parametric assumptions being met.

Question 2

What does the shape of the null depend on

Answer 2

• shape of null depends on number of observations
• more extreme values are associated with smaller sample sizes
• Several different ‘t’ distributions change shape subtly - specified by df on analysis (number of observations)

Question 3

Degrees of freedom

Answer 3

the number of independent values that can vary in an analysis without breaking any constraints

Question 4

Degrees of freedom equation

Answer 4

One sample= N-1
Independent sample= N1 +N2 -2
Paired sample= N-1

Question 5

P values

Answer 5

Probability of observing a result at least as extreme as the one from our data.

Significance = <5% (p<0.05) chance of observing result same size or larger by pure chance, under the assumption null is correct.

Question 6

What p values not

Answer 6

It is not:
- probability null is false
- probability experimental hypothesis is true
- statistically significant result does not mean result is practically significant or useful
- it is probably not certainty

Question 7

How to report p values

Answer 7

-after test statistics
-specify the Degrees of freedom of the test
-report exact p values two or three decimals
-specify the significance threshold used

Question 8

Effect Sizes (and its difference between t tests)

Answer 8

-t-values combine size of difference with precision of estimate

-Cohen’s D provides ‘pure’ measure of size of the difference - (effect size)
-It measures the strength of the difference, irrespective of how sig the effect may be

-No info about confidence and not affected by sample size.

Question 9

Reporting effect sizes

Answer 9

Report effect sizes after p-values e.g. t(28) = 1.13, p<0.32, Chosen’s d = 0.15

Report exact effect sizes to 2dp

Question 10

comparing means with t test

Answer 10

Example 1:
t(1476) =8.32, p<0.0001, Cohen’s d = 0.32
- very large sample
- strong evidence for difference in means
- very low probability of obtaining a result this large by chance, if null were true
- very small effect size, difference not likely to be of practical importance