AB Testing

Question 1

Sample Size Equation

Answer 1

Z-value * pop w/ feature * pop w/o feature / margin error ^ 2

Question 2

Type I Error

Answer 2

The Null Hypothesis should be accepted but is rejected

Question 3

Type II Error

Answer 3

The Null Hypothesis should be rejected but is accepted

Question 4

Confidence Interval

Answer 4

Confidence of not making a Type I Error, Common value is 90% - 95%

Question 5

Null Hypothesis

Answer 5

Hypothesis that control and treatment having the same impact or disapproval of the testing feature

Question 6

Statistical Power

Answer 6

Probability of finding a statistically significant results when Null Hypothesis is false.

Question 7

Rejecting the Null Hypothesis

Answer 7

Creates statistically significant result for the tested feature, that there is a difference between the treatment and the control.

Question 8

Relationship between CI and Test Sample

Answer 8

If you want higher CI, you will need a larger sample size.

Question 9

Definition of Power

Answer 9

Formula to figure out the chance for the null hypothesis to be rejected, bigger the sample size, generally bigger the power. Power = 1 - Beta where beta is the type II error

Question 10

Alpha - Power

Answer 10

p - value, this is the value that determines the likelihood for your feature data to have a type I error. The likelihood or p-value needs to be below the determined CI in order for us to reject the null hypothesis.

Question 11

Beta - Power

Answer 11

beta is the type II error or failure to account for the effect of the feature in the population sample.

Question 12

Assumptions of Power

Answer 12

Generally, the Beta will tell us the chance that the feature is ignored in the sample set [beta = .2, 20% of the time the feature is missed, Power will be .8 in this case]

Question 13

Jacob Cohen

Answer 13

States that for most researchers, type I error is about 4 times more significant than type II errors

Question 14

General practice for Power

Answer 14

Generally a power of .8 is enough for the experiment. As the requirement for bigger sample to increased CI is exponential.

Question 15

Critical value

Answer 15

t-statistic, this will let us know the degree of freedom and how much type II error is present within the dataset

Question 16

6 pillars of Power Analysis

Answer 16

1. Difference of the biological interest
2. Variability (std) of the data
3. desired Power of the experiment (.8) or 1 - Beta
4. Significant Level (.05) or alpha
5. Sample Size or Power Forumula
6. Alternative Hypothesis (one sided and two sized test)

Question 17

What is the difference of biological interest?

Answer 17

it is the minimum meaningful effect of biological relevance. Or how effects of the experiment is measured. The smaller the effect size, the smaller the need for sample size

We determine this through previous research or pilot studies.

Question 18

what is the variability or Standard Deviation of the data?

Answer 18

the variability of the data is how likely the population we sample is random.

to determine this, we use Data from previously related research on the sample population or the baseline of the sample population

Question 19

Significant Level

Answer 19

the Alpha of the experiment or often times the critical values (t-statistic or p-value) based on the size of the sample size.

Question 20

One-tailed test

Answer 20

One-tailed test will have the significant value to be closer to 0 and is easier for the data to be significant.

also known as direction test.

Question 21

Two-tailed test

Answer 21

Two-tailed test will have the significant value to be closer to 0 and is harder for the data to be significant.

also known as non-directional test.

Question 22

Difference between directional and non-directional test and use case.

Answer 22

1. the difference between the test is the way the type of statistics used. t-distribution is for non-directional while the f distribution is for directional.
2. you can use the directional test if you are only interested in one quantity over another rather than general difference.

Question 23

Difference between directional and non-directional test and use case.

Answer 23

1. the difference between the test is the way the type of statistics used. t-distribution is for non-directional while the f distribution is for directional.
2. you can use the directional test if you are only interested in one quantity over another rather than general difference.

IE, if you are interested in group A testing higher than Group B, then you can use directional test. If you are looking for the differences between the test score of Group A and Group B, then you should use the non-directional test.

non-directional is harder to be significant.

Question 24

Difference between t-statistic and z-statistic

Answer 24

If you don’t have information on the population sample such as mean, std, sample mean, sample size. Then you use the t-statistic.

you can use the z-statistic if you have those information

Question 25

what is the p-value and how is it derived

Answer 25

the likelihood that your results happened by chance. it is derived from the t-statistics

if the tested feature is significant and the likelihood of it being a fluke is low (p-value < .05) then the feature is likely to be significant

Question 26

What is one sample t-test and what is an example of it?

Answer 26

the most common type of t-statistic test. You are testing the means of your sample population against the average.

IE. Compare the SAT score of your tutor class to the average SAT score to a broader population.

Question 27

what is the paired t-test and what is an example of it?

Answer 27

The paired t-test compares the mean between the same sample size over a feature. Good to measure the effect of the feature on the same population over time or some other fixed variable

IE. Test if a new learning program is raising the SAT score of your population. Two features: time and learning program.