Week 3: Hypothesis Testing and Significance

Question 1

What is the main goal of hypothesis testing?

Answer 1

To assess how compatible sample data are with a certain hypothesis about the population parameter.

Question 2

What are the two types of hypothesis in hypothesis testing?

Answer 2

Null hypothesis (H0): Assumes no effect or no difference (e.g., μ = 0).
Alternative hypothesis (H1): Assumes an effect or difference exists (e.g., μ ≠ 0).

Question 3

What are the steps in hypothesis testing?

Answer 3

1. Define the null hypothesis
2. Define the alternative hypothesis
3. Choose a significance level (α)
4. Select the correct statistical test and calculate the test statistic
5. Find the critical value or p-value
6. Interpret the results

Question 4

What is the significance level (α), and what does it represent?

Answer 4

The probability of a Type I error (rejecting H0 when it is true), commonly set at 0.05.

Question 5

What is the difference between a one-sided and two-sided test?

Answer 5

One-sided: Tests if a parameter is either greater or less than a critical value
Two-sided: Tests if a parameter is different from a certain value, without specifying direction

Question 6

What are Type I and Type II errors?

Answer 6

Type I error (α): Rejecting H0 when it is true
Type II error (β): Failing to reject H0 when H1 is true
Note: Setting a lower significance level decreases a Type I error risk, but increases a Type II error risk. A Type I error is often considered to be more important to avoid. The hypothesis test procedure is therefore adjusted so that:
α = P(type I error) = significance level = 0.05

Question 7

How is a test statistic calculated for a hypothesis test?

Answer 7

By comparing the sample estimate to the null hypothesis value, scaled by its standard error. In other words, the number of SEs a sample estimate is away from H0. It shows the size of the estimate relative to its precision.
Example:
z = x̄ - μ / SE

Question 8

What is a p-value?

Answer 8

The probability of obtaining a result as extreme or more extreme than the observed result, assuming H0 is true.

Question 9

How do you interpret a p-value?

Answer 9

p < 0.05: Evidence against H0 at the 5% level
p ≥ 0.05: Insufficient evidence against H0

Question 10

What is the relationship between p-values and CIs?

Answer 10

Both describe the role of chance in the data. If a CI excludes the null value, the corresponding p-value is less than the significance level.
The p-value is the probability that a sample is consistent with H0. Unlike CIs, it gives little indication of the likely size of the population parameter.
The CI is a range of intervals within which the true population parameter is likely to lie.

Question 11

What is the critical value in hypothesis testing?

Answer 11

A threshold used to compare the test statistic to determine whether to reject H0. It depends on α and the type of test (one-sided or two-sided)

Question 12

Why is statistical significance not always practical significance?

Answer 12

Large samples can detect trivial effects as statistically significance, while small samples may miss meaningful effects.

Question 13

What is “p-hacking” and why is it problematic?

Answer 13

Manipulating data or analyses to obtain significant p-values, leading to misleading or non-reproducible results.

Question 14

How should p-values be used in reporting?

Answer 14

Treat p-values as a continuum and present them alongside CIs and study context (one piece of evidence alongside other factors e.g., prior evidence, study design, data quality, real-world costs and benefits, etc.
P-values represent the probability of the data given H0, not the probability of H0 given the data. Avoid using binary language to interpret p values (reject/don’t reject) in favour of no/weak/strong/some evidence against H0.

Question 15

How do you decide based on a 95% CI?

Answer 15

If the CI contains the null value, there is no evidence against H0.
If the CI excludes the null value, there is evidence against H0.
Note: We can use 95% CI for one sample tests. However, for two-sample tests, overlapping CIs does not preclude lack of support for H0.

Question 16

What are common misinterpretations of p-values?

Answer 16

• No evidence of difference does not mean there is evidence of no difference.
• No evidence against H0 is not sufficient to state H0 is true.
• Thinking significance tests are about individuals, when they’re really about central tendency of the population (mean, proportions).
• Thinking tests cannot be statistically significant by chance. By probability, 5% of the time you will get a value in the critical region, even if H0 is true. It is misleading to report results only if ‘statistically significant’. True effects may not be as large as initial estimates, e.g., due to publication bias.
• Lack of “statistical significance” equates with a lack of evidence for an effect
  (false)
• The p-value is the probability that the H0 is true. The p-value assumes the H0 is true; it indicates the degree to which the data conform to the pattern predicted by H0.
• Misinterpreting p < 0.05 as proving H0 is false/to be rejected. Rather, a discrepancy from H0 would be as large or larger than that observed by no more than 5% of the time if only chance were creating the discrepancy.
• Misinterpreting p > 0.05 as proving H0 is true/to be accepted. See point above.
• A statistically significant result is scientifically or substantively important, indicating large effect size. Rather, when a study is large, very minor effects can lead to significant results; CIs should be used to determine which effect sizes of scientific importance are compatible with the data.

Question 17

Why are one-sided tests controversial?

Answer 17

They yield smaller p-values and require clear justification before testing, as they only consider one direction of effect.

Question 18

What are key limitations of hypothesis testing?

Answer 18

• Results depend on sample size. When n is small, large deviations from H0 may go undetected. Where n is very large, tiny deviations from H0 may be found to be statistically significant. CIs are more informative.
• Tests (especially p values) can indicate statistical significance without practical relevance.
• Selective reporting can lead to publication bias.

Question 19

What is the “power” of a test?

Answer 19

The probability of correctly rejecting H0, when H1 is true (1 - β). Some tests may not be powerful enough to detect an effect.

Question 20

What steps should be taken to avoid “p-hacking”?

Answer 20

• Pre-register hypotheses and analysis plans
• Avoid selective reporting
• Consider the study context, prior evidence, and data quality

Question 21

How should p-values and CIs complement each other?

Answer 21

Use p-values to assess evidence against H0, and CIs to gauge the magnitude and direction of the effect.

Question 22

What is a test statistic?

Answer 22

Quantity based on sample data and H0 used to test between H0 and H1.

Question 23

What is meant by rejection?

Answer 23

Values of the test statistic used to assess how ‘strong’ the evidence is against H0.

Question 24

What are examples of hypothesis tests?

Answer 24

Hypothesis test for a mean i.e., how does the sample mean compare to some pre-specified value?
Hypothesis test for a difference in mean i.e., is the mean in group 1 different from that in group 2?

Question 25

Outline the logic underlying hypothesis testing:

Answer 25

• Look for evidence against H0 by evaluating its consistency with the data
• If the sample estimate is consistent with H0, then we have evidence in favour of H0
• If the sample estimate is not consistent with H0, then we have evidence against H0
  Note: We cannot prove or disprove a hypothesis

Question 26

Step 1 - Null hypothesis:

Answer 26

Assumption of no difference in the population from which the sample is drawn:
Mean: μ = μ0 (usually μ0 = 0)
Difference in means: δ = μ1 - μ2 = 0

Question 27

Step 2 - Alternative Hypothesis:

Answer 27

H1 is a statement of what a statistical hypothesis test is set up to establish.
Assumption that a difference exists in the population:
Mean: μ ≠ 0 or specified value μ0
Difference in means: δ = μ1 - μ2 ≠ 0 (or H1 can be directional - e.g. μ > 0)

Question 28

What will the choice of the test statistic depend on?

Answer 28

The assumed probability model and the hypotheses under study.
Commonly used: z-score (mean, proportion); t-statistic, chi-square; F-statistic

Question 29

What is the logic underpinning a 5% significance level?

Answer 29

If the test is two-sided, 95% of the z-scores fall within 1.96 SE away from the mean.
If the test is one-sided, the critical value is (+ or -) 1.645 (95% of the distribution lies below/above z=1.645).

Question 30

What does a large test statistic indicate?

Answer 30

Evidence against H0. For a mean -
>1.96 or <-1.96 data do not support H0 at 95% confidence level

Question 31

Do not fall into this trap when n is small…

Answer 31

You shouldn’t assume that because n is small, you cannot believe any results of a significance tests. Or conversely, that because n is large, the results must be true.
Statistical tests account for sample size when calculating probabilities and CIs. Study design (including sampling and method of randomisation) is critical: Statistical analysis cannot remedy basic flaws in data production, such as voluntary response samples, uncontrolled experiments, and measurement bias.