Module 3 - Multiple Comparisons

Question 1

Family-wise Error

Answer 1

Each time you test at = 0.05, you increase your chance of error

Question 2

Types of multiple comparisons - Planned comparisons (contrasts)

Answer 2

• Before study
• A priori
• Break down variance into component parts
• Test specific hypotheses

Question 3

Types of multiple comparisons - Post-hoc analyses

Answer 3

• After study
• Compare all groups using stricter alpha values
• This reduces Type I error rate

Question 4

Planned Contrasts

Answer 4

Involves breaking down the variance according to hypotheses made ‘a priori’ (i.e., before the data were collected)
RULES
* Once a group has been singled out – it cannot be used in another contrast
* Each contrast must only compare 2 “chunks” of variation
* There should always be 1 less comparison than the number of groups (i.e., number of contrasts = k-1)

Question 5

Planned Contrasts - Rules

Answer 5

• Compare positive against negative weights
• The sum of weights for a comparison should be zero
• If a group is not in a comparison it should be assigned zero
• For any contrast, the weights assigned to the groups or group in one chunk of variation should be equal to the number of groups in the opposite chunk of variation

Question 6

Planned Contrasts - Orthogonal Contrasts

Answer 6

Compare unique “chunks” of variance

Question 7

Planned Contrasts - Non-orthogonal Contrasts

Answer 7

Overlap or use the same “chunks” of variance in multiple
comparisons
Require careful interpretation
Lead to increased type 1 error rate

Question 8

Planned Contrasts - Standard Contrasts

Answer 8

Orthogonal: Helmert and Difference
Non-Orthogonal: Deviation, Simple, Repeated

Question 9

Planned Contrasts - Polynomial Contrasts

Answer 9

Linear, Quadratic, Cubic and Quartic trends

Question 10

Planned Contrasts - Helmert

Answer 10

Compare each category to the mean of subsequent categories (based on the order they are coded in SPSS, which might be alphabetical!)

Question 11

Planned Contrasts - Difference

Answer 11

Compare each category to the mean of previous categories

Question 12

Planned Contrasts - Polynomial contrasts

Answer 12

Used only when your IV is ordinal

Question 13

Research Scenario

Answer 13

Researcher interested in exploring the influences of drawing conditions on drawing quality
* Total sample of 60 with 20 participants in each group
* Independent Variable: Drawing condition
- 3 levels: Normal, Non-dominant Hand, Blindfolded
* Dependent Variable: Drawing quality
- Rated by an independent group of observers with the average score being the drawing quality
- The possible range of scores is 0-10 with higher scores indicating higher drawing quality

Question 14

Research Questions

Answer 14

Does the conditions in which you draw an object affect the quality of your drawings?
Hypotheses:
* H1: Drawing with your dominant hand will produce higher quality images than other conditions
* H2: Drawing while blindfolded will be more difficult than drawing with your non-dominant hand

Question 15

Post-hoc Tests

Answer 15

Involves comparing all possible differences between pairs of means
* Good approach with exploratory research or where there are no pre-defined specific hypotheses
* Simplest post-hoc test is Bonferroni
* Bonferroni correction means:
𝐵𝑜𝑛𝑓𝑒𝑟𝑟𝑜𝑛𝑖 𝛼= 𝛼 /𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑒𝑠𝑡𝑠

Question 16

Post-hoc Tests - Tukey’s HSD (Honestly Significant Difference)

Answer 16

• The cumulative probability of a type 1 error never exceeds the specified level of significance (p < .05)
• Supplies a single critical value (HSD) for evaluating the ‘significance’ of each pair of means
• The critical value (HSD) increases with (i.e., each additional group mean)
• It becomes more difficult to reject the null hypothesis as a greater number of group means are compared
• If the absolute (i.e., obtained) difference between two means exceeds the critical value for HSD, the null hypothesis for that pair of means can be rejected

Question 17

Module Summary

Answer 17

ANOVAs tell us if there is a main effect of an independent variable on a dependent variable, but not where that effect is
Running multiple t-tests isn’t sensible, because of family- wise error
Multiple comparisons allow us to compare levels of the independent variable
Planned contrasts are designed a priori to test specific hypotheses
Post-hoc tests compare all conditions to each other
You should know how to run these tests, and how to report them