External validity Flashcards
What is external validity?
Lab findings hold in the field/real-world
What benchmark should lab findings be held against?
The theory, not the real world
What is the Durham-Quine thesis?
No hypothesis can be tested in isolation. Always simultaneously testing auxiliary hypotheses.
What is the name of the thesis regarding the inability to test any hypothesis in isolation?
Duhem-Quine thesis
How does the lab overcome the Duhem-Quine thesis?
Randomised treatments can be compared against each other, solving for any differences in the auxiliary assumptions.
What is the intuition of the all causes model?
A regression model that contains all explanatory variables for the lab and the field
What are the 5 key criticisms of the external validity of lab experiments?
- Observer effects - nature and extent of scrutiny/obtrusiveness.
- The context of the game.
- Stakes of the game.
- Self-selection of participants.
- Artificial outcome space of choices
How to counter argue concerns regarding observer effects?
The 5 precepts regarding incentives should overcome this
How to counter argue concerns regarding the context of the game?
Does the researcher either have too tight control or too little? Logically can’t be both.
How to counter argue the stakes of the game criticism?
Stakes can be raised, and effects can be tested for.
How to counter argue the self-selection concern?
Subjects are paid, so not necessarily attracting more pro-social people.
How to counter argue the concern of the limited outcome space of choices?
Can test for these effects in the lab.
Name the 4 kinds of experiments
- Lab experiment.
- Artefactual field experiment.
- Framed field experiment.
- Natural field experiment.
What changes to go from a lab to an artifactual field experiment?
Using a non-standard subject pool.
What changes to go from an artifactual to a framed field experiment?
Any of the information, commodity, task or stakes of the experiment.
