3. Randomized Experiments

Question 1

Run an experiment

Answer 1

1. Randomly assign customers to different treatment groups
2. Compare differences in behavior among treatment groups

Question 2

A/B Tests

Answer 2

Divide customers into two groups:

• Control group A
• Treatment group B

Analyze if treatment group behaves differently

Question 3

Overall Evaluation Criterion (OEC)

Answer 3

OEC is a quantitative measure of the experiment’s objective

Examples:

• Active days per user
• Successful sessions
• Time to success

OEC must be measurable in the short-term yet believed to causally drive long-term strategic objectives.

Question 4

Parameter

Answer 4

A parameter is a controllable experimental variable that is thought to influence the OEC or other metrics of interest.

• In a simple A/B test there is usually a single parameter with two variables
• But multivariable test are also possible

Question 5

Randomization unit

Answer 5

A randomization unit is the entity to which the randomization is applied

• you must map units to variants in a persistent and independent manner
• it is common to use users as a randomization unit
• important to ensure that the populations are similar statistically, allowing causal effect to be determined with high probability

Question 6

Online Shopping Funnel

Answer 6

• Users may not progress linearly through a funnel, but instead skip, repeat or go bach-and-forth between steps

Question 7

Experiment Design:

Questions to be answered

Answer 7

1. What is the randomization unit
2. How do we measure success?
3. What population of randomization do we want to target?
4. How large does our experiment need to be?
5. How long do we run the experiment?

Question 8

How to perform the randomization?

Answer 8

• Pure randomization
• Stratified
  
  • Randomize within a group of users
  • e.g. make sure that all age groups have the same representation

Question 9

Measuring the impact on the OEC

Answer 9

To measure the impact of the change, we need to define a goal metric -> OEC

One obvious choice might be revenue:

• Revenue per user is preferred to overall revenue as it accounts for the number of users in each condition

Question 10

OEC: Which users to consider?

Answer 10

It depends on the business context:

• all users who visited the site
• only users who complete the purchase process
• only users who start the purchase process

All three can be right/wrong depending on the context

Question 11

Central Limit Theorem

Answer 11

The CLT states that the average from a random sample, when standardized, approximates a standard normal distribution, independently of the population distribution.

With the CLT we have information about the distribution of an estimator even without knowing the distribution of the original population.

Question 12

Hypothesis testing

Answer 12

Hypothesis testing is a method to draw insights from the population, based on a sample.

Steps:

• Formulate a hypothesis about the population
  
  • null hypothesis H₀
  • alternative hypothesis H₁
• _​Assess how likely the hypothesis is to be true, based on available data
• Reject or fail to reject the null hypothesis

Question 13

Statistical Power

Answer 13

Statistical power is the probability of detecting a meaningful difference between the variants when there really is one:

–> Reject the null hypothesis when there is a difference.