ANOVA

Question 1

basic model (SS_T)

Answer 1

is comparing observed DV scores against the grand mean (i.e., the mean for all participants, ignoring the fact that they are may be further grouped according to their scores on the IV);

In summary, the basic model seeks to evaluate how much variability there is in the DV in total (this is the starting point, as per regression)

Question 2

Best model (SS_M)

Answer 2

(i.e., the one that considers the IV(s) we have chosen;

is comparing group means against the grand mean

the best model is determining whether if we split participants into groups based on their IV score (i.e., for gender, we would split into male and female) whether the group means differ from the grand mean,

Question 3

The residuals (SS_R)

Answer 3

Can be viewed as comparing a participant’s DV score against the mean for the group s/he belongs to (i.e., their group mean).

the residuals determine whether the group mean adequately represents scores for individuals in that group.

Question 4

Assumptions of ANOVA:

Answer 4

• Homogeneity of variance.
  
  • As long as the disparity ratio from smallest to largest is less that 4:1 we can accept the differences in the variances.
• Normality (covered in module 3).
  
  • skew can distort group means, seeing as ANOVA is based around differences between means this can impact accuracy of the test
• And independence of observations.
  
  • All scores must be independent of each other, observations within a group should be unrelated (independant) to each other
  • Designs with repeated measures can violate this assumption

Question 5

Logic of ANOVA

Answer 5

In ANOVA, as in t-tests, we compare the difference between group/treatment means with a measure of the variability within groups. However since we are dealing with a design with more than two treatment means, we can’t use a single difference score as in a t-test. What is used instead of a simple difference between means (which is the top line of a t-test formula) is an estimate of how much variability occurs between groups (MS_model or MS_between), and this is compared with the amount of variability within groups (MS_residual or MS_within).

So for an ANOVA, we calculate an F-ratio which is:

Question 6

Calculations in the analysis of variance:

Answer 6

he first step in calculating ANOVA is to use the sums of squares approach to compute the three different sources of variance:

1. SS_total = sum of squares total = Σ (X_ij – X̄_.. )²

(Where X_ij represent each of the individual scores and X̄_.. represents the grand mean).

Note: the i in X_ij designates the individual, and j designates the group, such that person 1 in group 2 would have the code X₁₂. This also explains why the i subscript knocks out in the SS_M equation (below).

2. SS_model = sum of squares model (between groups) = nΣ (X̄_j – X̄_.. )(n is the sample size for a given group, X̄_j represents the group’s means. Note that we multiply the sample size for a given group by the group mean-grand mean value. This ensures flexibility in case the groups have different sample sizes.)

3. SS_residual = sum of squares residual (within groups) =

SS_total – SS_model =

Σ (X_ij – X̄_j )²

The next step is to calculate the variances corresponding to these sums of squares which are also called mean squares or MS (short for mean squared deviation) because they are a form of average of the squared deviations. This is done by dividing each SS by its degrees of freedom or df.

The three dfs are:

1. df_T = N – 1 (where N is total number of observations)

2. df_M = k – 1 (where k is the number of groups)

3. df_R = df_T – df_M = N – k (where N is the total number of participants)

The resulting mean squares are:

1. the treatment mean square: MS_M = SS_M/df_M

2. the error groups mean square: MS_R = SS_R/df_R

The final step is to calculate the F-ratio. We test the hypothesis by making a ratio of the MSs called an F-ratio, which is:

F = MS_M/MS_R

Question 7

If the null hypothesis is true:

Answer 7

If the null hypothesis is true then both mean squares are estimates of the same population variance and if we make an extra assumption that the scores in each sample are normally distributed, then the statisticians tell us that the ratio.

MS_M/MS_R

will have what is called an F-distribution, which is a type of distribution whose mathematical properties are well known. As with the t-distribution, the F-distribution has a different shape depending on the number of subjects tested, and also on how many groups are tested. So there are two degrees of freedom for each F-ratio, which are the df_model and df_residual which correspond to the numerator (or top line) of the ratio and denominator (or bottom line) respectively.

If the null hypothesis is true then the expected value of F would be 1 or less.

Question 8

If the null hypothesis is false:

Answer 8

If the null hypothesis is false, MS_model will be an overestimate of the population variance because it is also influenced by the treatment variability. Therefore the expected value of the ratio

MS_M/MS_R

would be greater than 1.

If the null hypothesis is false then the expected value of F > 1.

We can therefore use what is known about the F-distribution to tell us whether the particular value of F we have found is significantly greater than 1 or not, given the number of degrees of freedom we have, just as we did in a t-test using the t-distribution. It’s desirable for F to be considerably larger than 1 so that we are not getting excited over meaningless fluctuations in the data due to sampling or measurement error. The exact amount by which we need to exceed 1 depends on our degrees of freedom. With bigger samples, we are more confident that even subtle deviations above 1 are due to genuine effects rather than sampling perturbations, and therefore have a lower critical value for F. On the other hand, with smaller samples, it is easy to get big F values by chance alone and, therefore, we have a more stringent F value hurdle to claim significance.

Question 9

Magnitude of experimental effect:

Answer 9

t is easy to think that if you have a very big F-value you must have a very big result, but, if you have a large enough sample size the most trivial effects produce big F-values. The magnitude of the difference between two groups in standard deviation units, known as the effect size, tells you how big the effect is. The greater the difference then the larger the effect size which means that the finding is more important.

In Research Methods A you were introduced to one measure of effect size, Cohen’s d. This is used when you compare a pair of means. A d of .20 is considered a small effect, a d of .50 is medium, and a d of .80 and above is large. The d actually represents standard deviation difference in the means of the two groups. A d of .50 means that, on average, the two groups being compared have means that are .50 standard deviations apart from each other.

In addition to d, there are various other measures that assess the magnitude of the experimental effects. Two measures that are used in analysis of variance are Eta-Squared (η²) and Omega Squared (ω²). Both provide an estimate of how much of the total variance is accounted for by the treatment effect. Note also that ω² has been shown to be a less biased estimate of the magnitude of the experimental effect than η².

It is also important to note that there are varying views about how to interpret the magnitude of effect measures. Some researchers argue that even small effects can be important in some contexts, especially when only small changes are expected. On the other hand, a large effect can easily be obtained if we study a trivial variable.

Guidelines for interpreting η² and ω² have been provided by Cohen. Values of .02 are considered small, .13 medium and .26 large. However, any effect also needs to be interpreted in relation to the previous research in any particular topic area.

Question 10

Power

Answer 10

In ANOVA, as with t-tests, you set an alpha level beforehand which determines your probability of a Type I error (i.e., rejecting the null hypothesis, and concluding that there is a significant treatment effect, when the null hypothesis is true). This is usually .05 or 5 percent. There is another type of error, called a Type II error (which entails failing to reject the null hypothesis when it is in fact false; i.e., there is a significant treatment effect). This probability of a Type II error is called beta or b, and is used to calculate the power or sensitivity of an experiment by means of the equation.

power = 1 – b

An experiment which is powerful or sensitive is one which has a low probability of a Type II error, that is, an experiment in which it is easy to find a significant difference, even if the treatment effect is small. If a is made larger then β will be smaller (and hence the power of the experiment will be greater).

In particular, factors which increase power are:

1. Increasing the a-level (e.g., using .1 instead of .05 which is not advisable in practice); and
2. Increasing the number of subjects per condition