Generalizabiity Flashcards
True/False: Rachel argues that when a program has been evaluated in country A, but not in country B, policymakers in country B should always do their own impact evaluation before scaling it up there
False. Rather, Rachel argues for a framework that helps one think through how to take existing evidence and interpret it in one’s context as one is considering whether or not to run a new impact evaluation and/or to take a program to scale.
We want to know whether the government in Sierra Leone can implement the same program previously implemented by an NGO in India. When rolling out the program in Sierra Leone, we should probably conduct a:
Process Evaluation
Questions around whether or not a program can be properly implemented are evaluated using a process evaluation i.e., evaluating whether the process of delivering inputs and outputs, and conducting activities, happens as described. This comes before any questions around impact or impact evaluation. A process evaluation can be agnostic on the needs and theory of change; what it seeks to do is determine whether a program (whether effective or not) operates in the way it is purported to do.
Meta-analysis
combines the results of many RCTs and looks at the average effect. For example..”On average, what did we get when we have 50 different RCTs that all gave people this certain cancer treatment?”
Literature Review
which is more common in economics, is a little bit more freewheeling in, saying, looks at theory and tries to draw general lessons across
What are the possible concerns about meta-analyses as opposed to literature reviews?
As Rachel argues, meta-analyses can obscure study-specific and context-specific information by averaging across multiple studies. Even if there are strict criteria under which studies are admitted to meta-analyses, some studies may be more relevant to one context than another, or may play into the theory of change in a different way; by averaging across all studies, meta-analyses can obscure these differences.
When thinking about whether results generalize to a new context, rigorous evidence and knowledge of local conditions should be seen as
Complements
Knowledge of local conditions can inform how one should interpret and apply the results of rigorous evidence; for instance, knowledge about prevailing estimates of HIV prevalence in Rwanda help inform how one uses the results of the Sugar Daddies study in that context. Thus, they should be viewed as complements.
When thinking about whether lessons generalize to a new context, the main reason we worry about general behaviors is because:
People may not respond to the program the same way in different contexts
Our understanding of general behaviors is that they are common to people regardless of context. However, to the extent that the reaction people have to a program is context-bound, one cannot call such reactions general behaviors. Questions around such behaviors stand independent of the capacity or resources that we may have to implement and evaluate the program.
Evidence from other RCTs that people are willing to pay to tie their own hands/commit to a course of action is evidence in support of:
People procrastinate
Committing to a certain course of action, even at a cost, implies that people recognize the importance of that course of action even if they are psychologically not able to. For instance, one may recognize the value of going to the gym, but keep procrastinating doing so. Buying a gym membership could be construed of as a form of paying to commit to the gym to get over this procrastination. Committing in this way can be incentivized, but the evidence as presented in this question does not say anything about the need, or size, of the incentives.
