Field Experiments

Question 1

What is a Factor of an Experiment and what are levels?

Answer 1

Factor: A controlled independent variable.

Levels: categories of the factor.

Question 2

How can one test the causal effect of a factor?

Answer 2

One must compare at least two levels of the factor.

Question 3

Describe an A/B test?

Answer 3

between subjects experiment with one factor and two levels.

We test wether the means of the control or test group are the same, using a t-test.

Question 4

Describe between-subjects experiments with one factor and three levels.

Answer 4

How to test the differences?

Either test the different levels pair by pair, or use ANOVA (Analysis of Variance) on a single factor.

Question 5

How does Single Factor ANOVA work?

Answer 5

Assume each outcome y is a linear combination of:
mean of treatment level + mean error of single treatment level + error of the observation.

test that means of the treatments are all the same
OR
test that all the mean errors of the treatment levels are zero.

If either fo the two alternative hypotheses are rejected, the variance is simply given by sample variance, and there is no difference between treatment levels.

Question 6

What is the probability of making a type I error, if you have 6 hypothesis with significance level 0.05?

Answer 6

P = 1 - (0.95)^6 = 0.26

Question 7

What is a within-subject design?

Answer 7

When you test the same subjects on more than one level.

Question 8

How to test hypotheses within-subjects design ?

Answer 8

Repeated measures ANOVA.

Now the outcome y is a combination of:
the mean of the level + the mean error of the level + mean error of the subject + error of observation

We can remove variance due to difference in subjects, and isolate the effect of the treatment level.

Question 9

Pros and Cons of within-subjects?

Answer 9

Pros:
- Need less subjects

Cons:

• Demand effect
• Carryover effect

Question 10

What is a factorial experiment design?

Answer 10

Multiple factors and multiple levels.

main effect of a factor is the effect produced by change in the level of the factor.

Question 11

What is mixed design?

Answer 11

When you have both between and within subjects design. The factor related to the between. Levels are related to the within.

Question 12

What are Limitations of Lab Experiments?

Answer 12

• Self-selection

- Threats to external validity

Question 13

What are the types of experiments in general?

Answer 13

In order, from controlled to non-controlled:

• Lab
• Artefactual Field Experiment
• Framed Field Experiment
• Natural Field Experiment
• Natural Experiment
• Prosperity Score Estimation
• Instrumental Variable Estimation
• Structural Modeling

Question 14

Considerations when designing a field experiment

Answer 14

1. Non-compliance
2. Attrition/Mortality
3. Two-sided non-compliance
4. Contamination/Interference with experimental units