Hypothesis Testing, Statistical Inference and Estimation

Question 1

What does a Test of Significance determine?

Answer 1

It determines the probability that the observed difference between a sample statistic value and the parameter value stipulated by the Null Hypothesis (p hat - population proportion) is due only to random sampling variation.

Question 2

Is the test of significance based in the sampling distribution for the statistic of interest?

Answer 2

Yes, it is. This is why the formula is (sample statistic - hypothesized population parameter) / Standard Deviation of the sample statistic

Question 3

What is the center (expected value) of the distribution of the sample statistic (statistic of interest)?

Answer 3

The center is assumed to be equal (or no real difference) to the null hypothesis value for the true population parameter

Question 4

How do you compute the spread of the sampling distribution of the statistic of interest?

Answer 4

The spread is computed differently depending on your statistic of interest. For sample proportion (sigma p hat), it is calculated as square root of Po (1-Po)/n

Question 5

If we assume the null hypothesis is true (which we do when conducting all tests of significance), the shape of the sampling distribution of the statistic of interest will be approximately normal if the following conditions are met:

Answer 5

• n*Po is greater than or equal to 10
• n*(1-Po) is greater than or equal to 10

Question 6

What is the overall output from a test of significance and how do you get it?

Answer 6

The p-value! After completing the test of significance formula, you get a test statistic. Test statistics always have a known probability distribution that allows you to assign a probability to the observed value of the sample statistic if the value of the population parameter stated in the Null Hypothesis is true.

Question 7

What is the p-value?

Answer 7

It is the probability that the observed difference between the sample statistic value and the value of the population parameter stipulated by the Null Hypothesis is due only to random sampling variation, assuming the Null Hypothesis is really true.

Question 8

What are One-tailed tests of significance and when are they appropriate to use over Two-tailed tests?

Answer 8

These tests evaluate sample data as evidence for a difference in a specific direction. These tests are appropriate only if there is a theory-based reason, well documented in the scientific literature, for expecting a specific kind of difference.

Question 9

Provide additional interpretations/definitions of the p-value?

Answer 9

• It is a measure of the strength of the evidence against the Null Hypothesis
• It is also the probability that a conclusion to reject the Null Hypothesis value is incorrect

Question 10

Is the difference between statistically significant and practically important relevant?

Answer 10

Yes! Even though a difference between a (observed) sample statistic and a hypothesized population parameter (the null hypothesis value) may be statistically significant, the magnitude of the difference between them must be big enough to have real-world consequences; thus, be practically important

Question 11

What is Effect Size?

Answer 11

It is the difference between an observed sample statistic and the hypothesized population parameter. It measures the magnitude of the effect (the difference) documented by a study.

Question 12

Are the p-values of One-tailed tests smaller or larger than that of Two-tailed tests? And will your overall conclusion be affected between which one you pick?

Answer 12

They are smaller, which also makes them more likely to support a conclusion that a significant difference exists. Also no, picking either a one-tailed or two-tailed won’t affect the nature of your final conclusion as to whether there is a significant difference. ONLY in cases where the evidence against the null hypothesis is borderline (p-value is close to 0.05)

Question 13

When do you usually used Two-tailed tests of significance?

Answer 13

When there is insufficient basis to predict the direction of the difference prior to the study.

Question 14

What means that a sample statistic is “significant”?

Answer 14

It means that the more likely explanation for the observed difference between the statistic value and the null hypothesis value is that the true population parameter differs from the null hypothesis value.

Question 15

Why are the only two possible explanations for why the value of a sample statistic differs from the hypothesized parameter value ?

Answer 15

1. Difference due to Random Sampling Variation only (Null Hypothesis)
2. Real Difference (Alternative Hypothesis aka p-value below 0.05)

Question 16

What does the Type 1 Error Rate (alpha) give you?

Answer 16

It gives you the probability that a conclusion to reject the null hypothesis value of the true population parameter is actually incorrect. It is quantified by the p-value (meaning it is the same value as the p-value)

Question 17

When is the maximum allowable Type 1 Error rate (alpha) fixed?

Answer 17

When the investigator decides how small the p-value must be before he or she will conclude that there is sufficient evidence to reject the Null hypothesis value for the population parameter.

Question 18

What is the Power of a test of significance (1 - beta)?

Answer 18

The power is the probability that a test of significance will correctly reject the Null hypothesis value for the population parameter when there actually is a specified real difference between the Null hypothesis value and the true population parameter value.

Question 19

What is the specified real difference based on when discussing Power?

Answer 19

It is based in the investigator’s judgment about what would constitute the minimum important difference.

Question 20

What does is the Type II error rate (beta)?

Answer 20

It is the probability that a test of significance will fail to reject the null hypothesis value when there really is a difference. In other words, the test will fail to detect a real difference.

Question 21

What are the two possible reasons for why a test of significance fails to reject the null hypothesis?

Answer 21

1. There really is no difference.
2. The test of significance had insufficient power to detect it, usually due to inadequate sample size.

Question 22

Absolute Effect Size vs Relative Effect Size

Answer 22

The first one is the actual difference between the observed sample statistic and the null hypothesis value of the population parameter; while the second one, is this difference but it is presented as a percent.

Question 23

Absolute Effect Size vs Relative Effect Size

Answer 23

The first one is the actual difference between the observed sample statistic and the null hypothesis value of the population parameter; while the second one, is this difference but it is presented as a percent.

Question 24

What two steps you need to complete in order to calculate Power? Explain them.

Answer 24

1. Determine the MSD: The minimum significant difference refers to the rejection region, depicted by the critical value of the sample statistic which defines the range of values for the sample statistic that would provide sufficient sufficient evidence to reject the Null hypothesis. This first step only states that it is unlikely that the difference isn due only to random sampling variation.
2. Determine the probability of rejecting the Null hypothesis value if a specified MID exists: the minimum important difference is defined by the investigator and it refers to a difference that is sufficiently large and interesting to prompt a change in theory or standard procedures.

Question 25

Write the mathematical formula for the two-step process of computing power.

Answer 25

1. Critical value of the sample statistic= (Z or t score) [based on the type 1 error and whether it is a one-tailed or two-tailed] * (standard deviation of the sample statistic) + (the null hypothesis value)
2. P[Z or t is greater than or equal to OR less than or equal to (the critical value of the sample statistic - minus the alternative hypothesis value) divided by the standard deviation of the sample statistic (recomputed using the alternative hypothesis).

Question 26

What is a confidence interval?

Answer 26

It is an interval that specifies a range of values, within which the true value of the population parameter will occur with a specified probability (confidence level)

Question 27

What is the confidence level?

Answer 27

It is the probability that confidence intervals produced by the specified method will include the true population parameter value.

Question 28

What is the error rate of a confidence interval (alpha)?

Answer 28

The error rate (alpha) is the probability that a confidence interval produced by the specified method will NOT include the true population parameter value.

Question 29

What is the confidence level [100*(1 - alpha)] and the error rate (alpha) fixed on?

Answer 29

They are fixed by the investigator’s decision regarding how “confident” that person wants to be that the interval will include the true parameter value. The investigator’s choice of error rate determines the critical value used to calculate the interval.

Question 30

Are the confidence level and the error rate affected by the sample size or by the size of random sampling variation in the statistic?

Answer 30

Nope, they are not affected these things.

Question 31

How are 100*(1 - alpha)% confidence interval computed?

Answer 31

1. Compute the standard error of the sample statistic (which is the sample standard deviation)
2. Compute the margin of error= (Z_alpha/2)*(standard error of the sample statistic)
3. Compute the confidence interval for the true population parameter value= observed sample statistic +- the margin error.

Question 32

What happens if we want to be more confident that the interval will include the true value of the population parameter?

Answer 32

The interval widens, which then makes the estimate of the parameter value less precise. This happens because higher confidence levels have lower rates, but larger critical Z-values (which you use to compute the margin of error). In other words, higher confidence reduces precision.

Question 33

How can you compute the sample size required to obtain a confidence interval with a desired margin of error (aka m* aka desired precision)?

Answer 33

Two ways:
1. Depends whether a prior estimate of the sample statistic for the population parameter is available:

n= (Z_alpha/2 divided by m)^2 *[sample statistic (1 - sample statistic)]

2. Depends whether there is not an available prior estimate of the sample statistic for the population parameter:

n= (Z_alpha/2 divided by m)^2 * [0.5*(1 - 0.5)]

Question 34

When there is no prior estimate of the sample statistic for the parameter value, what do you do?

Answer 34

You use the sample statistic = 0.5 . Doing this, will produce confidence intervals with a margin of error that is always less than or equal to the specified value of m*.

For example, a margin of error of plus minus 0.05 is associated with the 95% confidence level.

Question 35

What happens to the p-value when you increase the effect size (the difference between the sample statistic and the null hypothesis value)

Answer 35

The p-value from a test of significance decreases. Large effects are unlikely to be due to random sampling variation only.

Question 36

What happens to the p-value when you increase the sample size?

Answer 36

Since increasing the sample size decreases the random sampling variation (e.g., the standard deviation of the sample statistic), the p-value also decreases (all other things being held constant). Remember that a smaller p-value indicates that the difference between a sample statistic and the hypothesized population parameter is less likely due to random sampling variation.

Question 37

What happens to power when you increase the sample size?

Answer 37

Random sampling variation is reduced (e.g., the standard deviation of the sample statistic); hence, the the power of a test of significance increases.

Question 38

Do one-tailed tests have greater power than two-tailed tests of significance?

Answer 38

Yes, since they have a higher probability of rejecting the Null hypothesis (smaller p-values).

Question 39

What happens to the power of a test of significance when you specify a smaller minimum important difference (MID)?

Answer 39

It decreases since small differences are less likely to be detected.

Question 40

What happens to the power of a test of significance and the Type II error rate (beta) when you decrease the allowable Type I error rate (alpha)?

Answer 40

It sets a more rigorous standard of evidence for rejecting the Null hypothesis. By making it more difficult to reject the null hypothesis, the power is reduced and the Type II error rate (beta) is increased.

Question 41

What happens to the width of confidence intervals when you increase sample size?

Answer 41

The width decreases since the standard error of the sample statistic decreases by increasing the sample size.

Question 42

What happens to the error rate (alpha) when you increase the desired confidence level?

Answer 42

It decreases the error rate (alpha) of confidence intervals.