Weeks 13 and 14

Question 1

This question refers to the diagram below showing the results of 5 different studies on the same treatment. Assume that for this treatment, effect sizes less than 0.2 standard deviations from d = 0 are unimportant.

Which of the studies has a treatment effect that is…
1) … beneficial, significantly different from zero, and important

a) long line from -0.4 to 0.4, M at -.005;
b) line between 0.25 and 0.42, M at 0.37;
c) line from -0.1 to 0.7, M at 0.22;
d) line from -0.15 to 0.1, M at -0.05;
e) line from 0.05 to 0.15, M at 0.1.

Answer 1

b) line between 0.25 and 0.42, M at 0.37;

Question 2

This question refers to the diagram below showing the results of 5 different studies on the same treatment. Assume that for this treatment, effect sizes less than 0.2 standard deviations from d = 0 are unimportant.

Which of the studies has a treatment effect that is…
2) … adverse, not significantly different from zero, and unimportant

a) long line from -0.4 to 0.4, M at -.005;
b) line between 0.25 and 0.42, M at 0.37;
c) line from -0.1 to 0.7, M at 0.22;
d) line from -0.15 to 0.1, M at -0.05;
e) line from 0.05 to 0.15, M at 0.1.

Answer 2

d) line from -0.15 to 0.1, M at -0.05;

Question 3

This question refers to the diagram below showing the results of 5 different studies on the same treatment. Assume that for this treatment, effect sizes less than 0.2 standard deviations from d = 0 are unimportant.

Which of the studies has a treatment effect that is…

3) … significant but unimportant

a) long line from -0.4 to 0.4, M at -.005;
b) line between 0.25 and 0.42, M at 0.37;
c) line from -0.1 to 0.7, M at 0.22;
d) line from -0.15 to 0.1, M at -0.05;
e) line from 0.05 to 0.15, M at 0.1.

Answer 3

e) line from 0.05 to 0.15, M at 0.1.

Question 4

This question refers to the diagram below showing the results of 5 different studies on the same treatment. Assume that for this treatment, effect sizes less than 0.2 standard deviations from d = 0 are unimportant.

Which of the studies has a treatment effect that is…

4) Which of the studies has the lowest power to reject the null hypothesis that d = 0?

a) long line from -0.4 to 0.4, M at -.005;
b) line between 0.25 and 0.42, M at 0.37;
c) line from -0.1 to 0.7, M at 0.22;
d) line from -0.15 to 0.1, M at -0.05;
e) line from 0.05 to 0.15, M at 0.1.

Answer 4

a) long line from -0.4 to 0.4, M at -.005;

Question 5

A significant effect must also be an important effect.

a) true
b) false

Answer 5

b) False

Explanation …. Consider an effect of treatment that is real, but very small. So small, in fact, that the changes associated with the effect are not clinically important. Even the tiniest effect, however, can be shown to be significant if the design is made powerful enough (for instance by using a massive number of participants). Thus, significance does not signal importance. These sorts of considerations can come up when one dealing with the side effects of some treatment. The main treatment can be considered safe if the side effects, although real, are clinically unimportant.

Question 6

The results of a large number of studies, all of which find there is no significant effect of some treatment, may nonetheless reveal a significant effect once the studies are aggregated in a meta-analysis

a) true
b) false

• graph of 4 lines all the same length at -0.15 to 0.3, with a metaCI that is between 0.01 to 0.17

Answer 6

a) True

Explanation … The point of this question is that a meta-analysis allows for a new estimate of population properties that is based on information from all of the observations in all the studies being aggregated together. The new estimate is, therefore, higher power than any one of the original studies it is based on. The diagram below demonstrates this for the highly artificial scenario where 4 identical, nonsignificant studies are aggregated into a single result with a significant effect of treatment.

Question 7

7) Suppose that the following standardized confidence intervals are from 4 studies on the same treatment. In a meta-analysis, which study will play the greatest role in determining the value of the aggregated treatment effect?
a) CI.95 = 0.8 ± 1.3
b) CI.95 = 1.8 ± 1.3
c) CI.95 = 0.8 ± 0.3
d) CI.95 = -0.8 ± 1.3

Answer 7

c) CI.95 = 0.8 ± 0.3

Explanation … The aggregated effect size in a meta-analysis is a weighted sum of all the effects sizes from all the studies that go into it. The study that will have the greatest weight in this sum will be the most exact one … in other words the one with the smallest confidence interval In this question, studies a), b) and d), all have the same sized confidence interval (each one is 2.6 standard deviations wide). Study c) in contrast has a much smaller confidence interval (only 0.6 standard deviations wide). Therefore study c) will be the most influential one in creating the aggregated effect size.

Question 8

Whenever you have to conduct more than one t-test as part of a multiple comparisons procedure, the FW Type I error rate will be larger than the PC error rate.

a) true
b) false

Answer 8

a) True

Explanation … Each time you conduct a test in a multiple comparisons procedure you expose yourself to the possibility of committing a Type I error. If you undertake 2 tests then you have exposed yourself to this possibility twice. The overall probability of committing a Type I error therefore has top be larger than for just a single test. This goes on and on. Each time you undertake a new test in the family you once again expose yourself to the possibility of committing a Type I error and these possibilities add together. The overall Type I error rate over the entire family of tests is therefore larger than in any single tests. In other words the Familywise error rate (FW) must be larger than the per comparison error rate (PC).

Question 9

When you make multiple comparisons using t-tests, one way to control the FW error rate is to make it more difficult to find significance in each individual comparison.

a) true
b) false

Answer 9

a) True

Explanation … When you perform multiple tests the rate of Type I errors aggregates into an overall larger number – the so-called FW error rate. A general strategy for making sure the FW error rate doesn’t become too large is to make the error rates judged acceptable for each individual test – the PC error rate – smaller. But this corresponds to using a smaller than usual alpha-value value in each test. It is the alpha-value that ultimately decides whether each test result will be judged significant and making the alpha-value smaller results in making it more difficult to find significance in each test.

Question 10

The Sidak formula for the FW error rate is more exact than the Bonferroni formula.

a) true
b) false

Answer 10

a) True

Explanation … Bonferroni’s formula relates the FW Type I error rate expected from c comparisons to the PC Type I error rate used in each comparison … FW = c x PC. This formula is highly intuitive because it says, for instance, that if you perform 10 comparison tests and use an error rate of .05 to decide significance in each one, then the rate of Type I errors is expected to be 10 x .05 = .5 over the entire family. This is too simplistic, however. Bonferroni’s formula really only gives an upper bound on the possible FW error rate and, in many cases, the true FW error rate may be smaller. The problem with this is that it may cause you to overcompensate when trying to adjust the PC error rate. Sidak’s formula is harder to use but is much more accurate, particularly for a large number of comparisons.

Question 11

If you want to perform pairwise t-tests to find out which means are different from which, but you haven’t decided ahead of time which pairs you will test, then you must perform all possible comparisons.

a) true
b) false

Answer 11

a) True

Explanation … A 2-sample t-test can be seen as a device for picking up differences between pairs of means. But if you don’t figure out ahead of time which pairs of means you would like to test, the temptation is to pick the pair of means that looks the most different from each other. The problem with this procedure can be seen by thinking about what would happen if, truly, none of the means in a design are really different from each other. In that case you would expect most means to be close to each other but every once in a while, just by accident, one pair of means would differ by quite a bit. Let’s say that this happens once in every 20 pairs. Normally this is accommodated by using alpha = .05 when performing tests. But if you insist on never doing a test unless you think you see a difference in means that is big enough to be significant, then you will almost always find significance even though we started off this whole line of reasoning by supposing that none of the means were really different from each other. In other words, your Type I error rate is no longer .05, it is almost 1! It is incredibly tempting to fall into this sort of fallacious procedure following an ANOVA. You can protect yourself, and your Type I error rate, by performing a full post hoc analysis that performs all possible pairwise comparisons.

Question 12

How many pairwise comparisons between means are possible in a 1-way design with 7 levels for the independent variable?

a) 1
b) 7
c) 21
d) 42

Answer 12

c) 21

Explanation … The number, c, of pairwise comparisons possible in a 1-way design goes up steeply as the number, k, of levels increases. The formula is c = k(k – 1)/2. When k = 7, as here, the number of comparisons is c = 7x (7 – 1)/2 = 42/2 = 21.

Question 13

For a post hoc analysis of a 1-way design with k = 5, what Bonferroni-corrected PC error rate would you use to limit the overall Type I error rate to FW = .05?

a) .005
b) .01
c) .05
d) .25

Answer 13

a) .005

Explanation … In a design with k = 5, there are c = (5x4)/2 = 10 possible pairwise comparisons that need to be made in a family of post hoc tests. If PC = .05 is used in each test the Bonferroni formula says that the familywise Type I error rate would build up to FW = c x PC = 10 x .05 = .5 over this family of tests. If each PC error rate is made smaller, however, then they won’t build up so much. Rearranging the Bonferroni formula tells us how small PC should be to achieve a desired FW error rate. If FW = .05 is desired, then the rearranged equations says the correct PC to use is PC = FW/c = .05/10 = .005.

Question 14

In an a priori family of tests following an ANOVA, how many comparisons need to be made?

a) more than for a post hoc analysis of the same data
b) the same number as for a post hoc analysis of the same data
c) fewer than for a post hoc analysis of the same data

Answer 14

c) fewer than for a post hoc analysis of the same data
Explanation … In a post hoc analysis you are forced to make all possible pairwise comparisons of means. If, however, you have the ability to figure out ahead of time which comparisons to make then you can get away with making fewer of them. Since controlling the FW error rate amounts to giving up power in each of the tests that form a family, getting away with having to test fewer comparisons is often a good idea. This is exactly why a priori families of test are used.

Question 15

For a 1-way design with k = 3 levels, which of the following sets of coefficients specifies a legitimate contrast of the means?

a) [1, 1, 1]
b) [-1, 1,1]
c) [-1/3, 1/3, 1/3]
d) [-1, 1/2, 1/2]

Answer 15

d) [-1, 1/2, 1/2]
Explanation … A contrast is a sum of means from an experiment where each term in the sum is multiplied by a coefficient. There is an additional constrain beyond this, however, that any sum has to satisfy if it is to be regarded as a contrast. The constraint is that the sum of the coefficients must be zero. We often specify contrasts by just giving the list of the coefficients involved and that is what you see offered in the possible answers to this question. Of all the sets of coefficients offered, however, only the final one “[-1, 1/2, 1/2]” satisfies the constraint that the sum of the coefficients is zero and so this is the only one that could form a contrast.

Question 16

As in the previous question, here are 4 more sets of coefficients for k = 3 means. Which one specifies a legitimate contrast of the means?

a) [1, 1/2, 1/2]
b) [-1, 1,1]
c) [-3, 1/3, 1/3]
d) [-30, 10, 20]

Answer 16

d) [-30, 10, 20]
Explanation … This is questions is very similar to the previous one. The only set of coefficients that sums to zero is the last one. It doesn’t matter how large the coefficients are. Only the fact that the positives balance out the negatives matters.

Question 17

For a given set of experimental results, which of the following contrasts would be expected to yield a different t-value than the others?

a) [1, -1/2, -1/2]
b) [1/2, -1/4, -1/4]
c) [-1, 1/2, 1/2]
d) [1/2, -1, 1/2]

Answer 17

d) [1/2, -1, 1/2]
Explanation … If the coefficients of one contrast can be turned into the coefficients of another simply by multiplying everything 2 by a constant then both contrasts will yield exactly the same t- and p-values in a test. In this question, the coefficients for answers b) and c) can be turned into the ones for answer a) by multiplying by -1 and 2, respectively. Answers a), b), and c) thus all represent the same basic contrast and will yield the same test results. The coefficients in answer d) on the other hand, cannot be turned into the coefficients of answer a) just by multiplying everything by the same constant. For instance you have to multiply +1/2 (the 1st coefficient in answer d)) by 2 to get the value of the 1st coefficient in answer a). But you have to multiply -1 (the 2nd coefficient in answer d)) by ½ to get the value of the 2nd coefficient in answer a). These are different multiplicative constants and so answers d) and a) cannot represent the same contrast.

Question 18

A pairwise comparison between means is, itself, a contrast.

a) true
b) false

Answer 18

a) True

Explanation … The numerator in a 2-sample t-test is Mi – Mj. This can be interpreted as a sum of means where one of the means, mean Mi, has a coefficient of +1, another mean, mean Mj, has the coefficient -1, and all the rest have coefficients equal to zero. Different pairwise comparisons could be picked out in this way. For instance, in a 1-way design with k = 3, a set of 3 contrast corresponding to 3 different pairwise comparisons could be [1, -1, 0], [1, 0, -1], and [0, 1, -1].
Thus, of all the contrasts that are possible for a design, a subset of them correspond to traditional pairwise comparisons.

Question 19

In a study (Science 332:1146-1148, 2011) on how gossip affects what we perceive, Anderson et al. showed participants faces which had previously been paired with socially positive, neutral, or negative statements. In addition, participants from time to time saw novel faces that had not been paired with any statement. The researchers then measured how much time participants spent attending to the various faces. If the results for the various experimental conditions in this design are presented in the order [Positive, Neutral, Negative, Novel], which of the following contrasts would best answer the question … “Do faces paired with positive gossip capture people’s attention more than other faces?”

a) [3, -1, 1, -3]
b) [-3, 1, 1, 1]
c) [1/2, 1/2 , -1/2, -1/2]
d) [1/3, 1/3, 1/3, -1]

Answer 19

b) [-3, 1, 1, 1]

Explanation … Contrasts allow you to ask interesting questions of experimental results. Here, the question is about whether hearing something negative about a person whose face is shown results in their face getting more attention than any other. The corresponding contrast should, therefore single out the Negative condition. The only contrast to do this is shown in answer b) because in that contrast the means for the Positive condition have one sign and the means for all other conditions have the other sign.

Question 20

For the same study, by Anderson et al., as in the previous question, which contrast would best answer the question … “Do novel faces capture more attention than known faces?”

a) [3, -1, 1, -3]
b) [1, 1, -3, 1]
c) [1/2, 1/2 , -1/2, -1/2]
d) [1/3, 1/3, 1/3, -1]

Answer 20

d) [1/3, 1/3, 1/3, -1]
Explanation … The answer here, d), is different from before but the reasoning is the same as in the previous question. We wish to know if novel faces are different from known faces in terms of capturing the attention of the participants. The corresponding contrast should therefore single out the Novel condition and compare it to all the others. Contrast d) is the only one that does this.

Question 21

Suppose that in a 1-way design with k = 3, the results of an experiment are M1 = 1, M2 = 2, and M3 = 4. Calculate the value of the contrast specified by [-1, 1/2, 1/2].

a) Y = 0
b) Y = 1
c) Y = -1
d) Y = 2

Answer 21

d)  = 2

Explanation … A contrast is a sum of means. The coefficients in the sum specified in the question are [-1. 1/2, 1/2] and the means themselves are given by are M1 = 1, M2 = 2, and M3 = 4 The sum is therefore Y = -1x1 + (1/2)x2 + (1/2)x4 = -1 + 1 + 2 = 2.

Question 22

15) Calculate the t-value for a test to determine if the contrast in the previous question is statistically different from zero. Assume that spooled = 1 and n = 150
a) t = 0.2
b) t = 1
c) t = 1.96
d) t = 20

Answer 22

d) t = 20

The formula for the t-test is:

t = Y / (spooled x square root of (sum of coefficients squared)/n))

The right hand side of this equation contains 4 terms and we know the values of 3 of them … Y = 2 (from the previous question), spooled = 1, and n = 150. The only thing missing is ∑▒a_i^2 , the sum of the squares of the contrast coefficients. But since the contrast in this problem has coefficients, [-1, 1/2, 1/2], the sum of the squares of these coefficients is ∑▒a_i^2 =∑▒〖[(-1)^2+(1/2)^2+(1/2)^2 ]=1+1/4+1/4〗=1.5.

Now we can fill in everything on the right-hand side of the formula…

t = Y / (spooled x square root of (sum of coefficients squared over n))
= 2 / 1 x (square root of 1.5/150))
= 2 / (square root of 1 / 100)
= 2 / 0.1
= 20.