week 5 A priori and post-hoc comparisons

Question 1

error rate

Answer 1

In multiple comparisons, the error rate is the probability of making at least one type 1 error (erroneously rejecting the null hypothesis) across all our comparisons. Two ways of calculating: a)error rate per comparison (PC) or

b)familywise error rate (FW)

Question 2

calculating error rate

Answer 2

How many comparisons? (c)If doing all possible pairwise comparisons, c=n(n-1)/2. eg if have 5 groups, there are 5(4)/2=20/2=10 possible pairwise combinations. To calculate the error rate: a) PC=alpha x c, eg for 10 comparisons at alpha=0.05. alpha 0.05 =5%. Thus 10 x 5%=50% error rate!

b).FW=1-(1-a)^c

where c=number of comparisons and a =alpha level.

if do 10 comparisons, with alpha=0.05,

then FW=1-(1-0.05)¹⁰=.40=40%.

Question 3

linear contrasts (psi)

Answer 3

Is a form of comparison test (usually a priori type).

A linear combination uses weights to compare groups.

is an alternative to running multiple t tests.

When we impose that the weights must sum to 0, it becomes a linear contrast (psi). This can be used to compare 1 group to a few groups.

ψ=a₁x^-₁+a₂x^-₂+a₃x^-₃

Question 4

assigning coefficients to linear contrasts and completing the contrast

Answer 4

1. set up the contrasts. assign coefficients for each contrast. these are used to “weight” each mean. Sum of the coefficients needs to equal zero.

Any set of coefficients that fits the ratio will suffice BUT if you want the coefficient to represent the actual mean difference, you must use fractions (1/number of sub groups).

eg. groups (A,B,C).
a) compare group c with groups A &B
b) compare groups A and B

A B C

Coefficients contrast 1 -1 -1 2

cofficients contrast 2 1 -1 0

2.Calculating psi for each contrast.

psi=a₁X^-₁ + a₂X^-₂ +a₂X^-₃

a=contrast coefficient. X^- =group mean

3. once we have a psi value for each contrast, need to convert this to a SS_contrast then an MS_contrast and then an F test.

SS_contrast=n(psi)²/Σa_j² n=sample size per group.

MS=SS/df.

For linear contrasts the df will equal 1 (number of groups being compared -1).

Therefore MS_contrast=SS_contrast

F_contrast=MS_contrast/MS_error MS_error is obtained from the ANOVA table.

If F(dftreatment,dferror) >Fcritical, then we reject the null hypothesis.

(compare positives to negatives etc in regards coefficients)

Question 5

a priori comparisons

Answer 5

Due to the fact that the type 1 error rate significantly increases the more comparisons are run, a priori comparisons aims to keep the number of comparisons to a minimum. A priori comparisons are planned prior to data collection. With careful consideration, only those comparisons required to answer the research hypotheses are conducted.

By contrast, post hoc comparisons are not planned prior to data collection, and with the post-hoc approach, all possible pairwise comparisons are conducted.

Question 6

orthogonal comparisons

Answer 6

A type of a priori.The chosen contrasts are independent of each other. There is absolutely no overlapping information.

eg if have 3 groups, and firstly compared group 1 and 2, and 2 scored higher, then secondly compared group 1 and 3 and found 1 scored higher, there is therefore no need to compare group 2 and 3 as already know that 2 has scored higher. So would not be done in an orthogonal study.

The maximum number of orthogonal studies is the same as df _treatment or number of groups -1.

We can never compare groups on the same branch in orthogonal studies.

All coefficients in an orthogonal contrast must sum to zero.

Σa_j=o

and

all pairwise products of the coefficients must sum to zero.

A B C

eg contrast 1 coefficient 2 -1 -1

contrast 2 coefficient 0 1 -1

0=(2x0) + (-1x1) +(-1x-1)

Question 7

linear combination

Answer 7

6

Question 8

psi

Answer 8

7

Question 9

Bonferroni t or Dunn test

Answer 9

The Bonferroni corrected p value keeps the familywise eror rate the same, but adjusts the p value used such that the type 1 eror rate is reduced.

Bp=original p/c

c=number of comparisons.

ie if p was 0.05, and wish to run 5 comparisons then new Bp is 0.05/5=0.01.

The disadvantage of this is that it is very strict(conservative) and whilst decreases risk of type 1 error, it has an increased risk of a type 2 error.

Question 10

modified Bonferroni

Answer 10

As Bonferoni adjustments are so conservative, there are various methods which do a more modified adjustment. eg Holm and egHolland-Copenhaver procedures. These both rank the p-values obtained for all the comparisons run, and adjust them based on their ranking.

Question 11

Keppel’s recommendations (form of modified Bonferoni)

Answer 11

a)do not adjust the significance level if the number of comparisons is NOT greater than the df. eg if have 5 groups then df_treatment=5-1=4.

ie if planning up to 4 comparisons(i n this example)then no adjustment is necessary (use 0.05 asPC error rate).

b)if the number of comparisons exceeds the df, then fWxdf=newFW

eg 4 x .05=.20=new FW

divide new FW by c to determine new P value

eg if doing 5 comparisons .20/5=0.04 (compared with Bonferroni would have done 0.05/5=0.01)