Multiple Comparison & A Priori Comparison

Question 1

F-Test

Answer 1

One overall (omnibus) comparison
F simply indicates that not all the populations are equal
Tests null against all possible alternatives

Question 2

Error Rate Per Comparison

Answer 2

Probability of making a Type 1 error on any given comparison increases as you increase the number of tests

Question 3

A Priori

Answer 3

Comparisons hypothesized prior to the outset of a project
Theoretically derived or driven hypotheses

A priori typically does not involve all possible comparisons
If theoretically motivated, can use .05 per comparison, within reason
Type 1 error rater will be lower!

Question 4

Post Hoc Comparisons

Answer 4

Post hoc comparisons are planned after the experimenter collects data and looks at the means
Data driven comparisons (think of new tests after you see the results, not pre registered)
Can capitalize on chance findings

Question 5

10 Comparisons A Priori vs. Post Hoc

Answer 5

Assume null is true
By chance two of the means are far enough apart that they would lead to rejection of null
A priori: probability of Type 1 error is 1 in 10 (10%)
Post hoc: given the same situation, probability of Type 1 error is 100%

Question 6

A Priori Comparisons with Homogeneity of Variance Assumed

Answer 6

In running individual t-tests, replace individual variances with MSerror from the overall omnibus analysis of variance
MSerror is a better estimate of standard deviation
Evaluate the observed t using the dferror from the omnibus F test

Question 7

Omnibus

Answer 7

Comprising several outcomes

Question 8

Linear Contrasts

Answer 8

Linear contrasts allow us to compare one group or set of groups with another group mean or set of group means
Procedures can be post hoc tests, but a priori is more common
Typically not used for t test style comparison, more used for comparing in groups

Question 9

Linear Combination

Answer 9

Linear contrasts are based on linear combinations
Weighted sum of means
Use small coefficients that sum to 0
Use coefficient 0 for means not being used in test

Question 10

Orthogonal Contrasts

Answer 10

Each contrast provides unique and independent information from the others
Contrasts are completely independent, doing one won’t tell you anything about the possibility of how the others will come out
To determine if contrasts are orthogonal, multiply the corresponding coefficients, sum of multiplications should sum to 0

Question 11

Number of Orthogonal Contrasts

Answer 11

Number of orthogonal contrasts you can make are k-1
SSgroups = sum of SS for each orthogonal contrast

Question 12

A Priori Comparisons

Answer 12

Linear contrasts/orthogonal contrasts

Question 13

Post Hoc Comparisons

Answer 13

Fischer’s least significant difference
Bonferroni t (Dunn’s test)
Studentized range statistic
Tukey HSD
Student Newman-Keuls

Question 14

Fischer’s Least Significant Difference

Answer 14

Also known as Fisher’s protected t
Use when you have no more than 3 groups and a significant omnibus F

Question 15

Bonferroni t (Dunn’s Test)

Answer 15

Change the per comparison rate
Can be applied to any test
The probability of occurrence of one or more events can never exceed the sum of the individual probabilities
E.g. make 3 comparisons with a = .05…probability of at least one type 1 error will not exceed .15

Question 16

Studentized Range Statistic

Answer 16

Can be applied to any number of groups
No 2 in equation
Basis of many post hoc tests (e.g. Newman-Keuls, Tukey)
Critical value of q is adjusted based on the number of means
Q test

Question 17

Tukey HSD

Answer 17

Very similar to Newman Keuls
qhsd will always be the maximum value of qr

Question 18

Student Newman-Keuls

Answer 18

Newman-Keuls is a layered test in that it adjusts criteria depending on the number of steps
Less strict, more liberal than Tukey
Uses the studentized range statistic

Question 19

Power of Bonferroni t

Answer 19

Not powerful
Using more conservative test reduces power
But want to reduce power when trying to correct for type 1 error in post hoc manner

Question 20

Adjacent Means in Q Test

Answer 20

For adjacent means: q.05 = t.05 SQRT(2)
For nonadjacent means: q will increase as the number of intervening means increases

Question 21

Bonferroni t Calculation

Answer 21

Set a’ = a/c
a = .05
c = number of comparisons
E.g. if making 5 comparisons, a’ = .05/5 = .01
Take alpha level of .05 and divide by number of comparisons
Tells us what p level we need for significance