ANCOVA AND MANCOVA

Question 1

List 3 general points about ANCOVA

Answer 1

ANCOVA can be used with all types of ANOVA

Can even have a changing covariate in repeated measures design (but not in spss)

ANCOVA is equivalent to multiple regression

Question 2

What is a covariate?

Answer 2

• a potential covariate is any variable (continuous as otherwise you would add it as a factor)that is significantly correlated with the outcome variable (DV)
• we assume a linear relationship between the covariate (x) and the DV (y)

(Null hypothesis: means on the DV do not differ significantly across groups after adjustment for scores on one or more covariate)

Question 3

What is a common design that would use an ANCOVA method?

Answer 3

In a two group pretest-posttest design, the pretest can be used as a covariate because how a subject scores before treatment is usually correlated with how they score after treatment

Or
Two groups compared on some score of achievement, with iq as a covariate, because IQ may be correlated with achievement

Question 4

If you remove a covariate, which will remove error from the error term underneath the f ratio, the f ratio will get bigger, what will happen to the t value?

Answer 4

It will get bigger and may then lead to a sig. Result

Question 5

What do you do to a correlation to get the amount of variance in the DV explained by the iv?

In the context of ANCOVA: IQ and achievement

Answer 5

Square the correlation

E.g. IQ and achievement - 0.80

Squared = 64% of the variance in achievement is explained by IQ

So if you remove this variance you go from trying to explain a change in 100% of the variance in achievement by your intervention, to explaining 36% of achievement- so explaining some of that will have a bigger effect. Say 5% or a 100% compared to 5% of 36%

ANCOVA removes that portion from the error term and thus increases statistical power

Question 6

For an ANCOVA analysis what happens to the sums of squares error term from analysis 1 to analysis 2

Answer 6

It gets smaller

Question 7

Error term underneath the f test is what?

Answer 7

The effect/error

Question 8

How are the effects of the IV’s assessed in an ANCOVA?

Answer 8

By holding covariate a constant (i.e. Treating each subject as if they scored at the mean of the covariate… So basically you adjust their achievement score to be at where it would be if they had scored at the mean of the covariate IQ)

Question 9

What is an additional test that ANCOVA produces?

Answer 9

Provides a test of significance for the regression of the covariate (s) on the DV ignoring group effects.

There would be no point doing an ANCOVA if there wasn’t a significant relationship between covariate and the DV as you would actually be adding noise into your analysis not taking it away. You would probably have tested this already with a correlation

Question 10

What are the assumptions of ANCOVA?

Answer 10

Usual ANOVA assumptions
-absense of outliers (both Univariate and multivariate outliers amounts DV’s and covariates)

• homogeneity of variances for DV and covariate (not so important if you have equal group sizes and not wildly disparate variances)

Usual MR assumptions:
Eliminate highly correlated covariates (multicollinearity and singularity)
Relationship between DV and covariate, and between covariates, should be linear

Additional for ANCOVA
Covariate is independent of treatment (in a random allocation this has to be true if this wasn’t the case it would be sampling error and one time in 20 this would be the case).

Homogeneity of the regression slopes (slopes for 1 covariate, planes for 2 covariates, hyper-planes for multiple covariates)

The covariate is measured without (much) error (reliable covariate)

Question 11

If the assumption of outliers is violated in ANCOVA what does this mean?

Answer 11

This is a serious violation

Question 12

What can you do if the relationship between the DV and the covariate is curvilinear? I.e. The assumption that this relationship should be linear is violated

Answer 12

• adjustment of the means will be improper it will be biased as it will assume linearity
• transform the data (only possible when the relationship is monotonic)
• fit a polynomial ANCOVA model to the data

Question 13

What if the assumption that the covariate should not have much error is violated? i.e. The covariate has high measurement error

Answer 13

For randomised designs, power is reduced

For nonrandomised designs, the effect is serious

Question 14

What are the preliminary data checks for an ANCOVA?

Answer 14

Examination of histograms for the DV on the covariate
(All distributions should be approximately normal & there should be no extreme outliers)
- check the homogeneity of variance assumption, esp. If group sizes are very different
-examination of scatter plots between the DV and the covariate (should be approx. linear)
- covariate should be measured before the onset of treatment (this makes sure the covariate is independent of the DV)
- covariate was measured reliably

Question 15

How can you test the homogeneity of regression (slopes) in spss?

Answer 15

• remember that ANCOVA will fit a single regression equation to the analysis so if the slopes are different this can be a serious issue

include covariate-by-IV interaction term (s) in the model, as well as main effects (basically just like a moderation analysis in regression)

• if there interactions are sig. Then there is heterogeneity of regression and ANCOVA is inappropriate
• in spss, the model button allows you to specify the model
• note a ‘full factorial model’ (spss default) does not include interactions between covariates and IV’s (this model is full in the context of the factors but it doesn’t include covariates) you need to test the covariance of the covariate and the IV’s

Question 16

How do you test homogeneity of variances in ANCOVA?

Answer 16

Levenes statistic in the DV and the covariate

If sig. Then this maybe an issue, but if there is a good sample size,equal groups and the largest sd is not twice as big then it’s okay (the largest variance needs to not be 4 times as big)

Question 17

What theoretical issues are there in the context of choice of covariates

Answer 17

Every time you include a covariate you pay a price and that price is that the DF for the error term decrease which makes it harder to have a sig. Effect… So there is a point of diminishing returns.

Ideal is small number of orthogonal (uncorrelated) covariates, this means they are explaining different bits of the DV - each correlated with the DV.

Goal - maximum adjustment of the DV with minimum loss of DF

Covariates must be independent of treatment
(Data on covariates must be gathered before the treatment is started)

• covariates should be reliable (i.e. Covariate measurement is not much contaminated by noise since measurement noise distorts the resulting effects and it is very problematic for small samples)

Question 18

What are the 3 uses for (M)ANCOVA

Answer 18

• to increase power by reducing error term in experimental work (with random assignment to groups)
• to adjust for mismatch on nuisance variables in non experimental work (this is the tricky case)
• stepdown analyses to follow-up MANOVA

Question 19

If the covariate was measured post treatments what might this do?

Answer 19

Lose independence

If covariate is measured post-treatment and is affected by treatment, then the change in covariate will be correlated with the DV

Covariate adjustment like then remove part of the treatment effect

Question 20

How many covariates?

Answer 20

For small group sizes max. Two or three covariates

Rule of thumb

Number of covariates + number of groups-1

Divided by

Total sample size

Should be less that 0.10

(If too many covariates are used, the estimates of adjusted means will be unstable)

Question 21

Discuss theoretical issues in ANCOVA surrounding ransom vs. nonrandom assignment

Answer 21

In random assignment (experimental) designs, group differences in covariates will be due to chance (as long as covariates measures before assignment)

With nonrandom assignment (common in psychology) covariate differences may reflect meaningful substantive differences related to group membership (I.e. Depression and controls with a covariate of anxiety, non random groups but those with depression may be stronger anxiety scores)

Question 22

Why is ANCOVA invalid when groups differ on covariate

Answer 22

ANCOVA looks at the relationship between the DV and the group (iv)

Don’t know what group represents when covariate and group are related
Group variable altered so that it may no longer measure what it was intended to measure
ANCOVA may move part of treatment effect or produce a spurious effect

Question 23

What is lords (1967) paradox

Answer 23

E.g.

Weight of boys and girls at the beginning and end of an academic year

-observations i) boys weigh more than the girls at the start and at the end, I) the average weight of neither group doesn’t change over time

Question
Does diet affect boys and girls differentially?

Conclusion 1 - normal ANOVA diet has no effect on weight

ANCOVA initial weight as covariate
End weight as DV
Gender as iv

Conclusion 2: boys show more weight gain than girls when initial weight differences are adjusted

Basically it ends up being an artificial comparison which is what ANCOVA does of comparing the weight gain of underweight boys with the weight gain of overweight girls

You end up comparing a group variable that you are not interested in as its nonrandom assignment of groups

Question 24

Can ANCOVA ever be valid with group differences on covariate?

Answer 24

If group differences arise by chance (e.g. In experiments with nonrandom assignment)

Overall and Woodward (1977): if group factor could not have caused the covariate differences

As a useful means of exploring the dataset and clarifying the relationship between the variable (but can’t say anything about causation if you do this)

Question 25

What are alternatives to ANCOVA?

Answer 25

You can incorporate the covariate into the analysis - no longer as a covariate but as another substantive variable

-propensity score matching (basically employs a predictable probability of group membership (e.g. Treatment vs controls group) based on a set of observed covariates - requires large samples with substantial group overlap, hidden bias may remain)

You can also extend the regression equation in the control/healthy group for the DV and the covariate - analyse residual scores for other / patient group (extending the regression line requires broadening the range of covariate scores in the control group)

You can also do something called blocking

Subjects are measured on potential covariates and the grouped according to the scores (e.g. Into groups of high medium and low)

Groups then become level of another full-scale iv that are crossed with the levels of IV’s in factorial design

(Advantages - none of the assumptions of ANCOVA or within subjects ANOVA - the relationship between covariate and DV need not be linear - can be extended to multiple covariates)

Question 26

What are the assumptions for MANCOVA?

Answer 26

A sig. Relationship between the set of DV’s and the set of covariates
Homogeneity of the regression hyperplanes

Null hypothesis: equality of adjusted population mean vectors

Question 27

What are the practical issues around MANCOVA?

Answer 27

MANCOVA serves as a noise-reducing operation where variance associated with covariates is removed from error variance

In non- experimental designs, MANCOVA provides statistical matching of groups (but beware of difficulties in interpretations, just as with ANCOVA)

Check the assumptions (linearity, homogeneity of regression etc) beforehand

Violation has serious effects

If covariates are not reliable, increased type 1 and 2 error

Check for multicollinearity and singularity

Question 28

What is specification error?

Answer 28

This is the term given to the idea that if preexisting groups differ systematically on more than the covariate. The covariate will leave those differences intact, thus biasing the estimate of the treatment… This is a problem with nonrandom assignment to groups

Question 29

What is a common issue with interpretation on nonrandom group assignment?

Answer 29

It is impossible to determine whether a given pretreatment difference reflects random error or a true group difference. This uncertainty complicates interpretation of treatment effects because it’s impossible to distinguish a main effect of treatment from an interaction between the effects of treatment and the pretreatment difference, and from meaningful overlap (variance shared) between the treatment and the pretreatment characteristics.

Often overlooked is the overlap between pretreatment difference and the grouping factor

Question 30

What was ANCOVA developed for?

Answer 30

To improve the power of the test of the independent variable not to CONTROL for anything

Question 31

How can you tell if the covariate and the grouping variable share variance and what sort of design is most likely to give this result?

Answer 31

They will be correlated

A quasi-experimental design (nonrandom assignment of groups)

Question 32

Why is it important to measure the covariate as reliably as possible?

Answer 32

To maximise its ability to capture noise variance in the DV

To ensure that the adjusted DV, DVres is not contaminated by noise associated with measurement of the covariate

This issue is particularly important in small sample sizes

Question 33

If the assumption of homogeneity of regression slopes is violated i.e. Heterogeneity of regression slopes is found, how severe is the violation and what is a way around it?

Answer 33

Not that serious

Investigator is encouraged simple to frame the analysis as a hierarchical or simultaneous regression and in that context to include an interaction term consisting of the product of the covariate X group

(The covariate then becomes a meaningful part of the analysis)

Question 34

What does the observed power tell us?

Answer 34

The observed power (aka post hoc power) is the chance of getting a sig. Result if you reproduced the study, if there was a difference between the populations which was exactly the same as the difference you observe in the samples, and if everything else was the same about the experiment including sample size.

Basically it doesn’t tell you anything more than the significance test. If you observed power is low p will not be sig.

Better to use an estimated effect size prior to the study to identify how much power you have to detect an effect of that assumed size by your experiment

Question 35

When running an ANCOVA what aspects of the experimental design give you confidence to run the analysis?

Answer 35

Random allocation to groups gives confidence

Covariate tested before allocation

And check the correlations between the covariate and the DV if highly correlated then it suggests that an ANCOVA may be useful.

Question 36

Where do you look in the output for the homogeneity of regression assumption in ANCOVA?

Answer 36

You look in the between-subjects effects box (univariate output) and you look at the interaction term

Question 37

What do the estimated marginal means show?

Answer 37

They show the group means after adjusting for the covariate

Question 38

From the multivariate output of an ANCOVA (I.e. MANCOVA) can you determine whether there is the necessary homogeneity of regression for this analysis?

Answer 38

Yes you look a the interaction term and if it’s not sig. Then there is homogeneity of regression slopes

Question 39

What is the Roy-bargmann step down analysis?

Answer 39

These are carried out after finding a sig. Effect in manova

The RB tries to find out the group differences on the individual DV allowing for group differences on the DV.

One has to have a priori priority ordering of DV’s, begin with highest priority and test simple ANOVA (adjusting for total number of comparisons) then next highest and so on

Seeing if there is an effect of a particular DV even after removing the influence of a higher priority DV (like hierarchical regression)