Hypothesis Testing

Question 1

Step 1: Choosing Hypotheses

Answer 1

A hypothesis test has two hypotheses: the null hypothesis (H0) and the alternative hypothesis (HA)

Hypotheses make inferences about the population as a whole.

H0:
can be rejected, assumes that there is no difference

Ha:
Never accepted Posits that there is a difference

Question 2

Step 2: Calculating the test statistic

Answer 2

The test statistic is a single value that measures the evidence for or against our hypotheses.
In this case, our test statistic is simply the differences between the means of each ad. In other words, the observed difference of sample means

test statistic = sample_mean_a - sample_mean_b

Question 3

Step 3: The permutation test

Answer 3

Remember, our null hypothesis states that there is no difference between the mean sales of ads A and B
Another way to say this is, “the ad shows has no effect on mean sales”
To test this, we carry out a permutation test, in which we randomly shuffle the “ad shown” column and find the observed difference. Repeating this process will approximate the true mean difference (remember the CLT)

Question 4

Finding the p-value

Answer 4

To find our p-value, we need to calculate the number of differences equal to or greater than our test statistic, divided by the total number of differences.

Question 5

Step 4: Drawing a conclusion

Answer 5

When conducting a hypothesis test, we need to specify a significance level alpha, that is, the value at which we reject out null hypothesis if p<= alpha
Note that we never “accept” the alternative hypothesis! Evidence against the null hypothesis is not proof of the alternative hypothesis.