Week 7 - planned comparision and post hoc tests

Question 1

Why does the F ratio not paint the whole picture?

Answer 1

only tells there is a difference somewhere between the means We need an analysis that helps to determine where the difference(s) are

Question 2

What are the two basic approaches to comparisions?

Answer 2

• A priori (or planned) comparisons
• Post hoc comparisons

Question 3

What is a priori (or planned) comparisons

Answer 3

• If we have a strong theoretical interest in certain groups and have evidence-based specific hypothesis regarding these groups then we can test these differences upfront
• Come up with these before you do your study
• Seek to compare only groups of interest
• No real need to do the overall ANOVA we do it because of tradition. Hence, reports often start with the F test and progress to planned comparisons

rather be in a prior hypothesis then be in post hoc

Question 4

What is a Post hoc comparisons?

Answer 4

• If you cannot predict exactly which means will differ then you should do the overall ANOVA first to see if the IV has an effect, then
• Post hoc comparisons (post hoc = after the fact/ANOVA)
• seek to compare all groups to each other to explore differences.
• Less refined – more exploratory.

Question 5

What are the two types of A priori/ Planned comparisons

Answer 5

Simple

Complex

Question 6

What is a simple a priori comparison?

Answer 6

comparing one group to just one other group

Question 7

What is a complex a priori comparison?

Answer 7

comparing a set of groups to another set of groups

*In SPSS we create complex comparisons by
assigning weights to different groups

Question 8

How to conduct a priori comparison (how to weight it)

Answer 8

Create 2 sets of weights

• 1 for the first set of means
• 1 for the second set of means
• Assign a weight of zero to any remaining groups
• Set 1 gets positive weights
• Set 2 gets negative weights
• They must sum to 0

A simple rule that always works –> The weight for each group is equal to the number of groups in the other set

Question 9

What are the assumptions of a priori/ planned comparisons

Answer 9

• Planned comparisons are subject to the same assumptions as the overall ANOVA - particularly homogeneity of variance as we use pooled error term.
• Fortunately, when SPSS runs the t-tests for our contrasts it gives us the output for homogeneity assumed and homogeneity not assumed
• If homogeneity is not assumed SPSS adjusts the df of our F critical to control for any
  inflation of TYPE 1error

Question 10

What are Orthogonal contrasts

Answer 10

• One particularly useful kind of contrast analysis is where each of the contrasts tests something completely different to the other contrasts

Principle:
Once you have compared one group (e.g., A) with another (e.g., B) you don’t compare
them again.

Example
Groups 1,2,3,4
Contrast 1 = 1,2 vs 3,4
Contrast 2 = 1 vs 2
Contrast 3 = 3 vs 4

Question 11

Cool things about orthogonal contrasts

Answer 11

• A set of k-1 orthogonal contrasts (where k is the number of groups) accounts for all of the differences between groups
• According to some authors a set of k-1 planned contrasts can be performed without adjusting for type-I error rate

Question 12

Post-Hoc comparisons

Answer 12

• Let’s say we had good reason to believe that sleep deprivation would impact performance but did not know at exactly what level of sleep deprivation this would occur. So, we had no specific hypothesis about what difference would emerge between which conditions.
• In this case, planned comparisons would not be appropriate
• Here you would perform the overall F analysis first
• If overall F is significant, we need to perform post-hoc tests to determine where the differences actually are

Question 13

What do post hoc comparisions seek to compare?

Answer 13

Post-hoc tests seek to compare all possible combinations of
means
* This will lead to many pair-wise comparisons
* e.g., With 4 groups, 6 comparisons
* 1v2, 1v3, 1v4, 2v3, 2v4, 3v4

Question 14

How does post hoc comparisions increase the risk of type 1 errors?

Answer 14

• So, as we know when we find a significant difference there is an alpha chance that we have made a Type I error.
• The more tests we conduct the greater the Type I error rate

Question 15

What is the error rate per experiment (PE)

Answer 15

the total number of Type 1 errors we are likely to make in conducting all the tests required in our experiment.
* The PE error rate <=  x number of tests
* <= means it could be as high as that value

Question 16

How do you restore a type 1 error rate back to .05 (5%) when conducting multiple tests

Answer 16

So when we need to conduct several tests, what should we do about the rising Type I error rate?
* If many tests is required, then a Bonferroni Adjusted
apha level may be used

Question 17

What is a Bonferroni adjustment?

Answer 17

• Divide  by the number of tests to be conducted (e.g., .05/2 = .025 if 2 tests are to be conducted).
• Assess each follow up test using this new  level (i.e. .025)
• Maintains PE error at .05 But this will reduce the power of your comparisons a lot!

Remember as we decrease alpha (by making our test more conservative) we also decrease power (chances of detecting a true effect)

Question 18

What are alternatives to the Bonferroni test (alternatives to reducing type 1 error rate)

Answer 18

• There are several statistical tests that systematically compare all means whilst controlling for Type 1 error
• LSD - least significant difference (actually no adjustment) (where you just ignore the problem -> not recommended)
• Tukey’s HSD - Honestly Significant Difference, popular as best balance between control of EW error rate and power (ie Type 1 V Type 2 error)
• Newman-Keuls: gives more power but less stringent control of EW error rate
• Scheffe Test most stringent control of EW error rate as controls for all possible simple and complex contrasts
• And many others you can find out about at your leisure

Question 19

What is the best one of these tests to use?

Answer 19

Tukey’s test is very common and recommended.

Question 20

What to do with post hoc tests (when do you use them and how)

Answer 20

• If your hypothesis predicts specific differences between means;
• Assess assumptions
• Perform ANOVA
• Consider what comparisons will test your specific hypotheses
• Perform planned comparisons needed to test these predictions
• If your hypothesis does not predict specific differences between means;
• Assess assumptions
• Perform ANOVA
• If ANOVA is significant then perform post-hoc tests
• If ANOVA is not significant then don’t do post-hoc tests

Question 21

What is a meta analysis?

Answer 21

When a researcher finds a lot of papers in the literature about a specific topic then you take their individual statistics and put it in a spreadsheet and then you aggregates these statistics and do a statistical test on this aggregated data

Question 22

Effect size philosophy

Answer 22

A significant F simply tells us that there is a difference between means. I.e., that the IV has had some effect on the DV
* It does not tell us how big this difference is.

• It does not tell us how important this effect is.
• An F significant at .01 does not necessarily imply a bigger or more important effect than an F significant at .05.
• The significance of F is dependent on the sample size and the number of conditions which determines the F comparison distribution

Question 23

What does effect size tell us?

Answer 23

If I took the overall variability in my criterion variable (example here target accuracy) How much of that variability could I explain on the basis of how much sleep deprivation you’ve had?

summarizes the strength of the treatment effect:

• Eta squared (n2)
• Indicates the proportion of the total variability in the data accounted for by the effect of the IV.

Question 24

what does n2 tell us

Answer 24

• This result says that __% of the variability in errors is due to the effect of manipulating whatever our IV is

For example, one could say that 65% of the variability in errors is due to the effect of manipulating sleep deprivation.

Question 25

What are the limitations of n2 (eta squared) given by SPSS

Answer 25

• It is a descriptive statistic not an inferential statistic so not the best indicator of the effect size in population
• It tends to be an overestimate of the effect size in the population

Question 26

Criteria for assessing eta squared

Answer 26

Cohen (1977) proposed the following scale for effect size;
* .01 = small effect (1%)
* .06 = medium effect 6%)
* >.14 = large effect (14%)

Question 27

Interpreting effect size

Answer 27

• The effect sizes typically observed in psychology may vary from area to area.
• The levels of the IV used are important in determining the observed effect size.
• A theoretically important IV may still only account for a small proportion of the variability in the data.
• A theoretically unimportant IV may account for a large proportion of variability in the data

Question 28

What is power?

Answer 28

• Sensitivity is the ability of an experiment to detect a treatment effect when one actually exists.
• Power is a quantitative index of sensitivity which tells us the probability that our experiment will detect this effect.

Question 29

What is the ideal power

Answer 29

*Keppel (1992) argues that ideally power should be > .80. to ensure an experiment can pick up a moderate effect.

• Ensuring adequate power is a research design issue

Question 30

Power is a function of…?

(what are the things you can tweak/change that will change your experiments overall power?)

Answer 30

1. The size of the treatment effect (we have better power to detect stronger effects)
2. The size of the error variance (the more noise in the data, the harder to detect an effect)
3. The alpha level (the more conservative the test, the greater chances you will reject the H1 even when it is true)
4. Sample size
   * Greater sample size – greater power

Question 31

How do you know what the power of your experiment is

Answer 31

power of your experiment is 1 minus the type 2 error rate

Question 32

Why does sample size effect the amount of power you have?

Answer 32

When the df is larger (so larger N and more people) then there becomes less overall error so F becomes larger

Question 33

Why not use the largest n possible?

Answer 33

• Not always cheap or easy to use large samples,
• We need to know what is the acceptable minimum sample size to pick up a specific effect

Question 34

What are the two main situations we are concerned about power in?

Answer 34

When we do not find a significant effect but there is evidence that we may have made a Type II error.

• When we are planning a new experiment and wish to ensure that we have adequate power to pick up the effect of our IV

Question 35

Power and sample size

Answer 35

• Ideally, we should determine the sample size that will give our experiment adequate power (> .8 ) before we run it.
• Conducting a study without an indication of the sample required to achieve desired power may be an expensive waste of time

Question 36

What are the ways to estimate the required sample size.

Answer 36

To do this you need an estimate of the magnitude of the treatment effect

• You can get this either from:
• past research
• a pilot study
• an estimate of the minimum difference between means that you consider
  relevant or important (often used in clinical experiments) .