Cost-effectiveness Analysis Flashcards
Cost-effectiveness analysis
Cost-effectiveness analysis measures the ratio of the costs of a program to the effects it has on one outcome
Measure the cost for a given level of effectiveness (cost to increase school attendance by 1 year) OR measure level of effectiveness for a given cost (years of more schooling after spendng $100).
Comparative cost-effectiveness
compares this cost-effectiveness ratio from one program to a similar ratio for many other programs
Pros of Comparative CEA
Good way to help decision makers synthesize information from many evaluations
Provides a summary of a single program in terms of its costs and effects on one outcome.
Can be used to compare many programs, find the most cost-effective option
Must compute costs and benefits using similar methodology for all programs being compared
Cost-Effectiveness (CEA) vs. Cost-Benefit Analysis (CBA)
CEA: Ratio of costs to effect on one outcome
CBA: Ratio of costs to monetary value of effects on all
outcomes; Can deliver absolute judgment on whether a program is worth the investment. Requires assumptions about the monetary value of all the
different benefits. (cost of life, disability, lower crime among school kids)
Advantage of CEA is its simplicity - Allows user to choose an objective outcome measure (e.g. cost to induce an additional day of schooling) – no need for
making judgments on monetary value of that schooling.
Easier for policymakers to compare programs when they are primarily concerned about one outcome of interest (e.g. increasing school attendance, not child health)
Which of the following is NOT a necessary
ingredient in a comparative CEA?
A. Several different programs looking at the same outcome
B. Accurate impact estimates
C. An estimate of monetary value of programs’ benefits
D. Information on the cost of the program
In a CEA, you do not need an estimate of the monetary value of the program’s benefits
What info is needed to do a CEA?
• Take impact measures from rigorous impact evaluations. Need some other info, like number of beneficiaries, when impacts were measured
• Take cost data from…?
– Most projects don’t record their implementation costs
– Need fairly disaggregated specific data on exactly what items were purchased, how much staff time was spent (on what), transportation costs, etc.
When might a CEA be useful?
When you have multiple programs that all have a positive impact on an outcome of interest, and you’re trying to choose between them
You want to convince a decision-maker that a non-obvious program is a good idea
You want to understand how the CE of a program could vary with contextual and implementation factors
Common CEA Uses
Retrospective analysis of pilot programs
– “Exactly how cost-effective was that pilot program?”
• Prospective analysis of pilot programs
– “Roughly how cost-effective could this proposed pilot be?” “How big an impact must this achieve to meet our threshold?”
• Prospective analysis of programs at scale
– “Roughly how cost-effective might this proposed national program be?”
Retrospective Analysis of Pilot Programs
List necessary data, strengths and weaknesses
Necessary data: cost data from an exact program that was evaluated & rigorous impact estimates
Strengths: Gives precise estimates of how cost-effective a program was in the context.
Weaknesses: Still suffers from external validity problem for cost and impact estimates
Prospective Analysis of Pilot Programs
List necessary data, strengths and weaknesses
Necessary data: Projected costs; impact estimates from a similar program
Strengths: Even rough calculations can help rule out programs that can’t be cost-effective
Weaknesses: Cost projections and impact estimates from similar programs may not be accurate
Prospective Analysis of Programs at Scale
List necessary data, strengths and weaknesses
Necessary data: Projected cost data for program at scale; rigorous impact estimates from pilot evaluation
Strengths: Producing customized prospective estimates are a powerful tool when speaking with country governments
Weaknesses: Impacts from small-scale pilots may not generalize to at-scale programs
Three Key Challenges in Doing CEAs
- Absence of incentives to do CEA
- Costs are hard to gather
- To be actionable for large funding decisions, analysis needs to be much more precise & that is not straightforward
CEA Challenge - Absence of incentives to do CEA
What if the program was effective but not really cost effective?
No editorial requirement to show CEA in most social science journals
CEA Challenge - Costs are hard to gather
Collecting cost data not seen as key part of evaluation
unlike impact measures
Cost data is surprisingly hard to collect from implementers (budgets different from implementation costs; hard to divvy up overhead and existing costs to project)
Hard to get cost data from other authors for a
comparative CEA
Impact measures and cost collection often not harmonized
CEA Challenge - Precise analysis required
– Numerous assumptions are needed to complete the
analysis (e.g. multiple outcomes, transfers, spillover effects, exchange rates, inflation etc.)
– No one “right” way, but consistency is important!
Which method does JPAL use primarily when gathering cost data retrospectively?
The “ingredients” method - gathering data from multiple sources
Research paper for description of program structure, ingredients and local conditions like wages;
Interview researchers for additional ingredients, their costs, additional documents like budgets;
Program staff and field research staff for unit cost data;
Supplement with public sources (e.g. local wages, transportation costs etc.)
What are some challenges with the retrospective CEA approach?
Data not originally collected by implementer or evaluator and key field staff are hard to locate or do not respond
Many important costs are forgotten, or hard to estimate after long lag
Program as implemented may be very different from how it was budgeted
Aggregate cost data is much less useful for sensitivity analysis or scale-up
How can prospective CEA help remediate some of the challenges with retrospective CEA?
Providing standard templates helps in data collection
Harmonization (outcomes, costs, methodology) makes it easier to do comparative CEA
What are some possible things you may be calculating the cost-effectiveness of?
The program, during the pilot phase
The program, if it was scaled up
Some component of the program
Aside from calculating the cost-effectiveness, what other assumptions should you include in the analysis?
How will you deal with…
– Exchange, inflation, discounting
– Spillover effects
– Multiple outcomes
– Costs shared with a partner organization
– Fuzzy costs: administration, overhead, and management
Issues to consider in CEA - Present value
Present value - real discount rate of 10% is used to discount costs and benefits to control for time
Issues to consider in CEA - Inflation
Inflation - adjust costs to today’s prices
Issues to consider in CEA - Across countries
standard exchange rates are used to adjust to US$
Issues to consider in CEA - multiple outcomes
can only examine one type of benefit at a time, which is how many policies are framed anyway
Issues to consider in CEA - Total vs. sunk costs
only considers the incremental cost to the existing infrastructure (material, personnel, oversight)
Issues to consider CEA - Proximal Success vs. Final Impact of Programs
Use global measures to translate proximal outcomes into final outcomes
Which of the following, if changed, would likely NOT influence the cost-effectiveness estimate?
Decision to use nominal exchange rate versus PPP exchange rate Population density The impact estimate Discount rate Local cost of living
None of the above.
Options 1 through 5 all play into how one accounts for the costs of the program in one’s CEA. These are amongst the factors that determine how the impact estimate plays into one’s CEA. Thus, all of these factors would have some bearing on one’s cost effectiveness estimate.
Should participant “opportunity costs” be included when conducting a CEA?
What costs one takes into account depends on what the purpose of one’s CEA. Is the question “which program is most cost effective for society” or “which program is most cost effective for a specific organization to run”? The former might want to take participant opportunity costs into account while the latter might not.
One challenge of CEA is that:
There is no one way to do CEA
As Iqbal illustrates, there are multiple decision points involved with determining how to conduct a CEA. There is thus no single way in which to do a CEA; it is critical however that one stay consistent in one’s methodology across programs when conducting a comparative CEA.
Issues to consider CEA - Transfers
Not a cost to the society but are they a part of the program cost?
International donors vs. local governments
Additional problems of non-cash transfers
Basically administrative costs vs. transfer costs
Issues to consider CEA - Significance of Effects
Only report results at 10% level of significance and show confidence intervals
Issues to consider CEA -Point Estimates vs. Range
Show range around point estimates to make distinction between a set of cost effective programs vs. a
set of not so cost efficient programs
Issues to consider CEA - Context
If costs depend a lot on specific contexts (e.g. population density) provide ranges of cost effectiveness based on these parameters
Things to consider in “Scale-Ups”
– Spillover Effects: Spillovers may be different in a pilot vs. scaled program.
– Partial vs. General Equilibrium: Very hard to measure
precise nature or direction of such effects
– Experimental vs. Scalable Mode: Costs of inputs may
become endogenous to the scale up
– Hard to Control Contextual Differences: Quality of
infrastructure, motivation of local partners and
beneficiaries, price differences, cultural differences, local parameters
