Lecture 3 Flashcards
A/B testing steps:
- Randomly divide people into groups
- Expose(излагам) each group to a different stimuli
- Measure the outcome & Compare the outcome
What makes A/B testing different from experiment
Bigger : scales up to million users
More control: easier to reach the user in the digital environment
Mechanism
Usage of A/B testing
Email marketing
Website marketing
Social media advertising
Intermediaries
come between the advertiser and the customer (the person who will see the stimuli)
How intermediaries affect the advertiser
the more intermediaries, the less control the advertiser has
During randomization, the outcome is affected by the treatment
internal validity
What does A/B testing to confounding variabels
eliminates them (bc of the randomizing)
How we can check if randomized is working during A/B testing
the variables outcomes should have similar %
When Multivariable testing is useful
- when you want to know which variables contributed the most
- suspect interactions
Average treatment effect (ATE) of A/B testing
how much the outcome changes as a result of AB
Calculating ATE
the mean outcome given A - the mean outcome given B
Multi-Armed Bandit (MAB)
speed up your experiments, by the proportion who is receiving the treatment is updated dynamically
Exploration
explore what you like
Exploitation
once you have an idea what you like, you start pursuing it
How Multi-Armed Bandit (MAB) can make experiments more efficient
by cutting down on inferior treatments
