Chapter 13: Null Hypothesis Significance Testing

Question 1

What is a goal of statistical evaluation?

Answer 1

To provide an objective or at least agreed-upon criterion to decide whether the results we obtained are sufficiently compelling to reject the null hypothesis.

Question 2

What is a statistically significant effect?

Answer 2

This indicates that the probability level is equal to or below the level of confidence selected.

Question 3

What is the use of statistical evaluation?

Answer 3

It provides consistent criteria for determining whether an effect is to be considered veridical.

Question 4

How is statistical significance a function or measure of sample size?

Answer 4

The larger the sample size, the smaller the group differences needed for statistical significance for a given level of confidence.

Question 5

What is the minimum required power of a study?

Answer 5

At least .80
- If power is .80, this means that the investigator’s chance of detecting a difference in their study is 4 out of 5, if there is a real difference in the population.

Question 6

Why are underpowered studies a problem for the field as psychology?

Answer 6

It leads researchers to believe that the topic was studies and that a reasonable answer came out of it. As a result, researchers might quit investigating this area, even though there is an effect to be found.

Question 7

How can you identify your needed sample size?

Answer 7

Fix the parameters of:
1. Alpha
2. Power
3. ES

Question 8

How can we decide on an effect size?

Answer 8

1. You can consult meta-analyses to identify a likely ES for the study you want to undertake.
2. If meta-analyses are unavailable, you can consult one or more studies that have compared the groups of interest or that used the measures (DV) in a related way.
3. Another study on the topic or closely related can be consulted.
4. When meta-analyses and individual studies are unavailable, ES can be estimated on a priori grounds.
   
   • It is helpful to select a conservative estimate.

Question 9

What is the difference between n and N?

Answer 9

N refers to the overall sample for the entire study, while n refers to the sample size within each condition.

Question 10

Why do methodologists advocate attention to power?

Answer 10

1. There is a long tradition of most studies being underpowered to detect the likely effects their manipulations will show.
2. Power is easy to calculate given statistical software.

Question 11

What is the error variance?

Answer 11

The heterogeneity of the subjects (as reflected in within-group variance).
- This is directly related to ES and statistical significance.
- The larger the error variance, the less likely the results will be statistically significant.

Question 12

What are ways to increase power?

Answer 12

1. Increase the sample size (N or n/group).
2. Increase expected differences by contrasting conditions that are more likely to vary (stronger manipulations, sharper contrasts).
3. Use pretests/repeated measures that reduce the error term in the effect size.
4. Vary alpha (a priori) if the case can be made to do so if, e.g.:
   
   1. Classification of groups is imperfect.
   2. Measures are not well-established.
   3. Small effects/differences are predicted.
   4. Consequences of the decision vary markedly as a function of the direction and hence we wish to detect difference in one direction rather than another.
5. Use directional tests for significance testing,
6. Decrease variability in the study as possible by:
   
   1. Holding constant vs controlling sources of variability.
   2. Analyzing the data to extract systematic sources of variance from the error term.

Question 13

How can you set up conditions that might show a stronger effect?

Answer 13

By comparing groups that are more extreme, since that might show a stronger effect.
- E.g. comparing none vs. a lot, rather than a bit vs a lot.

Question 14

What are statistical advantages of a pretest?

Answer 14

With various analyses, the error term in evaluating ES is reduced.
- With repeated assessment of the subjects, the within-group variance can be taken into account to reduce the error term.

Question 15

What circumstances may lead the investigator to anticipate specific constraints that will reduce the likely ES and the differences between groups or conditions?

Answer 15

1. The criterion for selecting groups in a case-control study might be known to be imperfect or somewhat tenuous.
   
   • Thus, some persons in one group might, through imperfect classification, belong in the other group. Imperfect classification is analogous to diffusion of treatment as a threat to internal validity.
2. The measures in the area of research may not be very well established. This may introduce variability into the situation that will affect the sensitivity of the experimental test. The predicted relation might be more evident with a more sensitive/reliable measure.
   
   • Thus, it might be reasonable to have a less stringent p-value.
3. The specific comparison of interest may be expected to generate a small difference between groups. Usually, a large sample size is required to find this effect, but we might not have access to that. Thus, a lower p-value might be useful.
4. We might alter alpha based on consideration of the consequences of our decisions.

Question 16

What do you need to remember when considering the relaxation of alpha?

Answer 16

You should not relax levels after the fact or when the results just miss conventional significance levels.
- This violates the model on which significance testing is based.

Question 17

Why is there resistance to using one-tailed t-tests?

Answer 17

There is often an implicit assumption that investigators who have used one-tailed tests may have done so because the results would otherwise not be statistically significant.
- Often it is unclear to the reader of the research report that the use of one-tailed tests was decided in advance of seeing the results.

Question 18

How can you be transparent when using a one-tailed t-test?

Answer 18

Clarify the basis of this use.
- Why such tests are used should be embedded in the goals of the study and described in the introduction to the study and the rationale for the hypotheses.

Question 19

How can you control variability?

Answer 19

By monitoring many facets of the study; by holding many sources of variability constant.
- This translates to a stronger ES.

Question 20

How can one reduce error variance of a study?

Answer 20

By analyzing the data by one or more of the variables embedded in the error variance.
- You select variables that are worth evaluating because they are used to test, explore, or generate hypotheses of interest.

Question 21

What are common concerns of NHST?

Answer 21

1. All-or-none decision making.
2. H0 is rarely or never true.
3. Significance is a function (and measure) of N.
4. Tests are more subjective than meets the eye.
5. Says nothing about the strength or importance of effects.

Question 22

What are misinterpretations of NHST?

Answer 22

1. That p reflects the likelihood or degree to which H0 is true.
   
   • Instead, p refers to the likelihood that the finding would be obtained by chance if a large number of tests were run.
2. Higher p-value is more potent/stronger effect.
3. Higher p-value is an effect more likely to be replicated.
   
   • Significance fallacy: the interpretation of statistical significance as being a measure of real significance or importance.
4. No difference means that there is no real effect but a difference means that there is.
5. There are nonsignificant “trends” or a difference that “approached” significance or is of “borderline significance”.

Question 23

What does a Bayesian analysis do?

Answer 23

It pits H0 against Ha and asks which hypothesis is more credible given the data that has been collected.

Question 24

What is the likelihood (in Bayesian stats)?

Answer 24

The probability of obtaining the exact data we obtained given our hypothesis.

Question 25

What is the Bayes factor?

Answer 25

The ratio of the probability of obtaining the observed data given the null hypothesis, divided by the probability of obtaining the observed data under the alternative hypothesis.