Comparisons and contrasts

Question 1

omnibus F statistic (beyond a one-way ANOVA)

Answer 1

If the IV has more than three levels, the omnibus (overall F statistic) is ambiguous

A significant main effect doesn’t tell WHERE the difference are (which groups differ)

Question 2

Comparison

Answer 2

A statistical comparison of 2 chunks of data

e. g. young vs middle, middle vs old, young vs old
e. g. young and middle vs old, young vs middle and old, you and old vs middle

Question 3

Type 1 error rate

Answer 3

probability of making a type 1 error (false positive- saying there is an effect when thee isn’t)

Question 4

Type 2 error rate

Answer 4

false negative (saying there isn’t an effect when there is)

Question 5

Per comparison (PC) error rate

Answer 5

Probability of making a type 1 error on ANY comparison

aPC=.05

Question 6

Familywise (FW) error rate

Answer 6

The probability of at least one Type 1 error in a family of comparisons

1-(1-a)to the power of c
c= no. of comparisons

e.g. 5 comparisons aFW= 1- (1-.05) to the power of 5= .27

There is a 27% chance of making a Type 1 error (around 1 in 4 will be a an error)

Question 7

Relationship between PC and FW

Answer 7

FW error goes up and a and c go up

PC=or < FW

Question 8

Multiple Comparisons (MC) and corrections

Answer 8

To reduce the chance of a Type 1 error:

Reduce the number of comparisons (c)

Lower the significance threshold (alpha level)

FDR (false discovery rate)- used with large number of comparisons or large number of significant effects

Question 9

A priori comparisons

Answer 9

Planned before looking at the data

Based on hypotheses/ theory (what are you interested in)

:) more liberal thresholds (more sensitive to identifying a genuine effect- more powerful)

In SPSS will generally have one less comparison than number of IV levels

Question 10

Post hoc comparisons

Answer 10

Made after looking at the data

Exploratory analyses

:( more conservative alpha (less prone to Type 1 errors but harder to identify a genuine effect (Type 2 error)- less powerful

in SPSS: post hoc test will give ALL the possible comparisons

Question 11

Myths busted about multiple comparisons

Answer 11

1) just because you used planned comparisons, it doesn’t mean you don’t need to protect against Type 1 error inflation
(BUT Type 1 error inflation will be less than in post hoc (because post hoc is all possible tests, planned is a subset) so a correction will reduce power less)

2) Omnibus F doesn’t need to be significant to do multiple comparisons
(logic behind MCs doesn’t require this- Wilcox (1987) says not much use for F test at all)