A/B Testing Flashcards
What is a type 1 error?
Rejecting the null hypothesis when it’s true
What is a type 2 error?
Failing to reject the null hypothesis when it’s FALSE
What is a p-value?
A p-value is the probability that, given the null hypothesis is true, what is the probability that this observation is due to random chance? If there’s a low probability, you reject the null hypothesis!
It can also be thought of as the observed significance level ->
What is POWER?
Power is P(reject the null | null is false)
In words, it’s the probability you identify an effect when it’s actually there
Power describes what proportion/percentage of the time you correctly reject the null hypothesis when it is false (how often do you detect the treatment effect when it’s actually there)
What are degrees of freedom?
They are the number of independent values that are free to vary
Why is a type 1 error generally more serious than a type 2 error?
From a theoretical perspective, this ties into why we say “fail to reject” as opposed to accept. We aren’t necessarily accepting the null hypothesis, we just don’t have enough evidence to the contrary.
From a more practical perspective, a type 1 error means you have drawn or come to a conclusion that likely requires some sort of action/cost/investment, and you obviously don’t want to be incurring costs to pursue things that don’t actually work or have the desired effect.
What is Simpson’s paradox? What about the UCB admissions example?
Simpson’s paradox describes how aggregating data can mask important differences between groups
1970s, UCB was sued for gender discrimination in admissions. Lawsuit pointed to the fact that women were underrepresented in admissions. However, once an appropriate breakdown was made and people dug a little deeper and looked at acceptance rates of individual departments, women actually did better in many cases!
