Week 3: ANOVA assumptions, power and mean comparison

Question 1

what is the underlying ANOVA equation?

Answer 1

Score = Grand mean + treatment effect + residual error (measurement and individual differences)

Question 2

Essentially, how does ANOVA break down variance? and what does the comparison of this breakdown produce?

Answer 2

Into 2 parts

• Variance due to treatment
• Error variance

Comparison of these two variances produces the F STATISTIC

Question 3

What are the assumptions of a between ANOVA?

Answer 3

Homogeneity of variance: SD for all groups is about the same
Normality: error is normally distributed
Independence of observation: truly between design

Question 4

How do you test the assumption of homogeneity of variance?

Answer 4

Levenes test
- if statistically significant the null is rejected. this means that there is a difference somewhere do homogeneity is violated

Question 5

What do you do if Levenes test is significant?

Answer 5

You can look up density plots to find the culprit

can be violated without grave consequences as long as sample sizes are equal

Question 6

What can you use to test the assumption of normality?

Answer 6

Shapiro WIlk
If significant, assumption of normality is violated

can also look at histograms, distribution plots, skewness and kurtosis

Question 7

What happens if normality is violated?

Answer 7

Typically its not a big issue

ANOVA still tends to be robust in terms of normality violations

Question 8

How do you report that you have used a non-parametric version of an ANOVA?

Answer 8

‘Similar results were found using non-parametric….’

Question 9

What are quantile-quantile plots?

Answer 9

They look at normality assumption.

Chop data up into how many scores are in each quantile and compare it against what we would expect in that quantile for a normal distribution.

Straight line = perfect
Deviations = bad

Question 10

What happens to the ANOVA if assumptions are violated?

Answer 10

If you arent confident, can use non-parametric versions of the test
This can then increase confidence in results

Question 11

What is skewness of data?

Answer 11

Deviations from typical bell curve of data distribution

- asymmetrical

Question 12

Explain positive and negative data skewness

Answer 12

Negative skew: Have a long tail towards low values in the data (more low scores than expected)

Positive skew: long tail towards high values (more high scores than expected)

Question 13

No skew value?

Answer 13

0 - perfectly symmetrical

Question 14

What values are considered moderately skewed?

Answer 14

Between -1 and -0.5
or
Between 0.5 and 1

Question 15

What values are considered highly skewed?

Answer 15

(-/+) 1+

Question 16

Skewness in Quantile-Quantile plots?

Answer 16

If data drops below the line, it is skewed to the left or negatively skewed

If data rises above the line, it is skewed to the right or is positively skewed

Question 17

What is kurtosis?

Answer 17

It is a measure of tailedness in data distribution curve

how light or heavy-tailed the data is

Question 18

What is leptokurtic distribution (kurtosis)?

Answer 18

People scoring closer to the average (light tailed)

Question 19

What is platykurtic distribution (kurtosis)?

Answer 19

Wider spread of scores so maybe more heavy tailed

Question 20

What is the optimal kurtosis value?

Answer 20

It is reported in terms of how much excess skew there is - so the optimal would be 0!!!

Question 21

How do you know when to reject normality in terms of kurtosis?

Answer 21

You are given a standard error as well as a skewness value - these can be used to determine a z-score

z score = Value of kurtosis/standard error

If z score is less than -2 or more than +2, reject normality

Question 22

What is power?

Answer 22

The probability of correctly rejecting the null

finding a difference between means if it is there

Question 23

What is power associated with?

Answer 23

Type 2 errors

Question 24

when do type 2 errors typically occur?

Answer 24

when alpha level is too strict
or
outlier increases error variance so hard to see treatment effect clearly

Question 25

What is the power equation?

Answer 25

power = 1-B

The probability of finding a real difference = 1 - the probability of not finding a real difference

Question 26

What increases power?

Answer 26

• Increase in magnitude of difference between means (effect size): makes it more obvious
• Alpha level (more generous): increases power but also increases type 1 errors or false positives
• variance in scores: less variance increases power as if too much variance, makes it harder to see effects clearly
• larger sample size: gives more accurate means - more power as can see more clearly

Question 27

A priori power estimate?

Answer 27

Estimate before the experiment is run

- helps to ensure have adequate numbers of participants to detect any real differences between treatments

Question 28

Post hoc power estimate?

Answer 28

Estimate after data has been gathered
Estimates the likelihood of being able to replicate a significant difference if repeated
- value between 0 and 1
eg 0.82 - 82%

Question 29

How can you obtain a post hoc power estimate?

Answer 29

G*power - REFER TO SLIDES
Can provide you with a power estimate if you feed it means, average variance (MSE) and sample size
- Tell it what kind of test you’ve done
- gives an effect size that can then be used to find power

Question 30

How do we find a priori power estimates?

Answer 30

Estimation of effect size (cohens d or f) - or you can use the effect size from other studies
With this effect size you can compute a non-centrality parameter (used to find the power of an experiment for any effect size)

Can then plan sample size etc (can use g*power again)

Question 31

What happens when a statistically significant comparison has more than 2 means?

Answer 31

Need further tests to determine which of the means actually differ from each other
- typically multiple comparisons

Question 32

What are the two approaches to follow up tests?

Answer 32

A priori: chosen before data is collected

Post hoc: planned after data is examined

Question 33

Errors in multiple comparison tests?

Answer 33

Major type 1 error rate risks

Question 34

Per comparison error rates vs. family-wise error rates?

Answer 34

Per comparison:
the probability of making a T1 error on any comparison - this is alpha

Family-wise:
the probability that a family of comparisons will contain at least 1 T1 error
- per comparison rate stacks up
- error rate per comparison X number of comparison

Question 35

A priori comparisons?

Answer 35

Usually when have very specific hypothesis - creates a smaller number of comparisons so T1 probability is reduced

• contrasts

Question 36

How to do a priori comparisons in jamovi?

Answer 36

These are ‘constrasts’ and this is a tab within jamovi under the analysis set up

Question 37

What are the different types of contrasts?

Answer 37

Deviation - each level to grand mean
Simple - mean of one to each other one
Difference - mean of level to previous
Helmert - mean of level to all subsequent combined
Repeated - mean of one to each subsequent individually

Question 38

T1 error rates in contrast tests?

Answer 38

Need to do bonferonni adjustments

alpha/number of comparisons

Question 39

Critics to bonferonni?

Answer 39

Claim it may be too conservative and we may make T2 errors instead and miss actual effects

Question 40

What can we do here if we are worried that bonferonni is too conservative?

Answer 40

Linear step up!

Question 41

What is linear step up?

Answer 41

Focuses more on false discovery rate (FDR)

Rank comparisons according to their p values - divide this rank by the number of comparisons
Then multiply this by FDR which is alpha

Gives you a critical p value to determine significance against

Question 42

What other post hoc tests may be appropriate when comparing means?

Answer 42

Tukey or Holm