Confidence Intervals and Hypothesis Testing

Question 1

What is Hypothesis Testing

Answer 1

Setup hypothesis about population parameter that you want to test, gather sample as see if it confirms or rejects our assumptions.

Question 2

Null hypothesis (H0)

Hypothesis Testing

Answer 2

Population parameter we assume to be true (want to confirm).
Example: Assume average population IQ is 100.

Question 3

Ha / H1 - Alternative hypothesis

Hypothesis Testing

Answer 3

Unexpected result (that we want to check for significance).
Example: Population IQ is larger than 100.

Question 4

Significance level

Hypothesis Testing

Answer 4

• Alpha (α)
• What percent in H0 is so unexpected that we reject H0 and assume Ha.
• Example, if our mean is in 5% of normally distributed sample assuming H0, it means that our H0 is wrong and we reject it.

Question 5

Z-stat vs T-stat

Hypothesis Testing

Answer 5

• We use the z-distribution (z-statistics and z-tests) with proportions.
• And we use the t-distribution (t-statistics and t-tests) with means.
• With a large sample size, results from the t-distribution are very similar to results from the z-distribution. So sometimes statisticians use the z-distribution with means. However, we should avoid that practice (even for large samples).
• In general, when population mean is not known, use T-statistic.

Question 6

P-value

Hypothesis Testing

Answer 6

Probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct.

Question 7

General formula for Hypothesis Test

Answer 7

(BestEstimate - H0_estimate) / SE_h0

Question 8

Type I Error

Hypothesis Testing

Answer 8

• We reject H0 although it’s true
• (we received sample data that falls under significance level but H0 is true)

False Positive

Question 9

Type II Error

Hypothesis Testing

Answer 9

We don’t reject H0 although it’s not true.
Data we received didn’t fall under significance level, but it should because H0 is false.

False negative

Question 10

Power

Hypothesis Testing

Answer 10

P(rejecting H0 | H0 false)
1 - P(Type II Error)
Probability of correctly rejecting H0 (i.e. accepting Ha) when H0 is false.

Question 11

How To Increase Power

Hypothesis Testing

Answer 11

• Increase significance level (alfa). Big downside - P(Type I Error) goes up as well.
• Increase sample size (n)

In general, Power is higher when population:
* Has less variability.
* True (alternative) parameter (Ha) is far from H0 - less intersection area.

Question 12

How to check normallity assumption of sample data

Answer 12

• Plot it and check the shape (if large sample size)
• Use QQ Plot to check for normality.
• Compare box plots to see if IQR are roughly the same.

If not sure then normality requirement can be ignored if n is large enough (because of CLT),

Question 13

Welch’s t-test

Hypothesis Testing

Answer 13

Method of Testing for a Difference in Population Means of independent groups in Unpooled Approach

Question 14

What is P-hacking and how to prevent it

Hypothesis Testing

Answer 14

• P-hacking polega na łamaniu założeń używanych modeli statystycznych, takich jak stosowanie niezależnych prób losowych, oraz na popełnianiu błędów logicznych.
• Metody przeciwdziałania takiemu zjawisku obejmują między innymi prerejestrację planów badawczych.

Question 15

Why is it called Student’s t-test

Hypothesis Testing

Answer 15

Gets its name from William Sealy Gosset who first published it in English in 1908 in the scientific journal Biometrika using the pseudonym “Student” because his employer (Guinness Brewery) preferred staff to use pen names when publishing scientific papers (also, didn’t want to let competitors know they use t-statistics).

Question 16

General formula for confidence interval

Answer 16

Best Estimate ± (T-Multiplier (Z, T) * Standard Error)

Question 17

What is confidence level

Answer 17

The confidence level refers to the long-term success rate of the method, that is, how often this type of interval will capture the parameter of interest.

Długoterminowe prawdopodobieństwo zwierania przez próbkę szukanej statystyki populacji, jeśli byśmy wielokrotnie powtarzali próbkowanie.

Question 18

Sample conditions to calculate confidence interval or hypothesis test

Answer 18

• Random: The data needs to come from a random sample or randomized experiment (best: SRS)
• Normal: The sampling distribution of p ̂ needs to be approximately normal or has large-enough size to use CLT (at least 10 expected successes and 10 expected failures)
• Independent: Individual observations need to be independent. If sampling without replacement, our sample size shouldn’t be more than 10%, percent of the population.

Question 19

How to find Conservative Standard Error and when to use it

Confidence Interval

Answer 19

• Use worst-case MoE that does not depend on our p̂ estimation.
• Calculate for worst case scenario, i.e when nominator is maximized.

Use when p̂ is not accurate or to determine needed sample size.

Question 20

How to find needed sample size to achieve specific MoE with confidence level.

Confidence Interval

Answer 20

Derive from Conservative Standard Error

Question 21

What is T-distribution

Answer 21

• T-Distribution converges to Z-statistic as n grows larger.
• T-Distribution has fatter tails (larger values on tails) than normal distribution.
• Use degree of freedom to find it’s value

Question 22

Apporoaches of estimating CI or hypothesis test for a Difference in Population Means (for Independent Groups).
Describe

Answer 22

We have two possible approaches depending on two sample variances.

Pooled Approach
The sample variance is very close, variance of the two populations are assumed to be equal (Sx1 == Sx2)

Unpooled Approach
The assumption of equal variances is dropped.

Both uses different formulas and different df calculation methods.