ch8

Question 1

Field experiment

Answer 1

Natural environment where manipulation is possible

• Problems with randomisation
• Problems to exclude external influences

Question 2

Advantages of field experiments

Answer 2

• Real world behaviour = real world impact, high external validity
• Authenticity - field experiments provide authentic (a) context, (b) treatments (c) participants and (d) outcome measures
• Novel insights - (a) answer questions that cannot be answered in the lab, (b) to check if lab-results hold in real-world situations and (c) to capture second-order and long-term effects

Question 3

Disadvantages of field experiments

Answer 3

• Time consuming
• Challenging to implement
• Focused on observed behaviour - limited to observable behaviour, low ability to investigate underlying/psychological
• High degree of noise
• Ethical considerations

Question 4

2 main objectives of experimental studies

Answer 4

•To draw valid conclusions about the effects of IV(s) on DV -> requires internal validity
•To make valid generalisations towards a broader group/population -> requires external validity
*without internal validity, no external validity

Question 5

Threats to internal validity

Answer 5

• Poor timing and unexpected situational factors - changes in the environment unrelated to the study, e.g. weather, technology, news, politics
• Failure to randomise - no randomisation of participants in groups (due to targeting, technical failures etc)
• Non-compliance/Failure-to-treat - subjects that are supposed to receive the treatment do not receive it
• Spillovers&side effects - one participant is affected by the treatment of other participants; no consideration of unexpected side effects
• Insufficient sample size - insufficient power to detect effects (main and interaction)

Question 6

Internal validity best practice

Answer 6

• Randomisation checks
• Unit of randomisation
• Power calculations
• Outcome measures

Question 7

A/B testing

Answer 7

• Field experiment in a digital/online environment (usually large scale)
• Testing on large-scale enables to detect small effects
• Results scale easily in the online environment
• Amazon, FB, etc each carry out/conduct more than 10k tests per year
• Focus on conversion rates
• Learning process how to optimise key metrics (e.g. CTR, sales)

Question 8

A/B testing: Industry Wisdom

Answer 8

•A/B tests are an excellent opportunity to replace intuition with tests
•Small changes can have a big impact
- But 1: most tests fail: only about 10-30% generate positive results
- But 2: Changes rarely have a large positive impact
- Consequence: need to make a lot of several small wins to improve
•A/B testing is about learning
•A/B vs A/B/n vs Multifactorial tests
- Multifactorial designs are more complex and require larger sample size (“traffic”)

Question 9

Slicing the cake - heterogeneous treatment

Answer 9

•Instead of focusing on overall effects only, consider to look at the effects in subgroups/segments
-> Heterogeneous treatment effects
•Effects in segments might vary; e.g. for gender (men vs women), historical purchase patterns (heavy vs light shopper) etc
•Focusing on aggregate data might lead to the incorrect conclusion that there are no effects on any participant
•Important 1: partition subjects in subgroups based on pre-treatment covariates
•Important 2: Need sufficiently large sample in each condition to get valid results

Question 10

What is an experimental culture

Answer 10

•In company practice, it is not about designing the best experiments, it is about making better decisions
- False belief that only successful experiments are good experiments
- Need to shift companies from a culture of decision making based on intuition to a culture of decision making based on experimentation
•In research, one can learn from unsuccessful experiments; unsuccessful experiments can even open new research avenues
- However, rigor is crucial in the design of experiments