Cost-effectiveness Analysis

Question 1

Cost-effectiveness analysis

Answer 1

Cost-effectiveness analysis measures the ratio of the costs of a program to the effects it has on one outcome

Measure the cost for a given level of effectiveness (cost to increase school attendance by 1 year) OR measure level of effectiveness for a given cost (years of more schooling after spendng $100).

Question 2

Comparative cost-effectiveness

Answer 2

compares this cost-effectiveness ratio from one program to a similar ratio for many other programs

Question 3

Pros of Comparative CEA

Answer 3

Good way to help decision makers synthesize information from many evaluations

Provides a summary of a single program in terms of its costs and effects on one outcome.

Can be used to compare many programs, find the most cost-effective option

Must compute costs and benefits using similar methodology for all programs being compared

Question 4

Cost-Effectiveness (CEA) vs. Cost-Benefit Analysis (CBA)

Answer 4

CEA: Ratio of costs to effect on one outcome

CBA: Ratio of costs to monetary value of effects on all
outcomes; Can deliver absolute judgment on whether a program is worth the investment. Requires assumptions about the monetary value of all the
different benefits. (cost of life, disability, lower crime among school kids)

Advantage of CEA is its simplicity - Allows user to choose an objective outcome measure (e.g. cost to induce an additional day of schooling) – no need for
making judgments on monetary value of that schooling.
Easier for policymakers to compare programs when they are primarily concerned about one outcome of interest (e.g. increasing school attendance, not child health)

Question 5

Which of the following is NOT a necessary
ingredient in a comparative CEA?

A. Several different programs looking at the same outcome
B. Accurate impact estimates
C. An estimate of monetary value of programs’ benefits
D. Information on the cost of the program

Answer 5

In a CEA, you do not need an estimate of the monetary value of the program’s benefits

Question 6

What info is needed to do a CEA?

Answer 6

• Take impact measures from rigorous impact evaluations. Need some other info, like number of beneficiaries, when impacts were measured

• Take cost data from…?
– Most projects don’t record their implementation costs
– Need fairly disaggregated specific data on exactly what items were purchased, how much staff time was spent (on what), transportation costs, etc.

Question 7

When might a CEA be useful?

Answer 7

When you have multiple programs that all have a positive impact on an outcome of interest, and you’re trying to choose between them

You want to convince a decision-maker that a non-obvious program is a good idea

You want to understand how the CE of a program could vary with contextual and implementation factors

Question 8

Common CEA Uses

Answer 8

Retrospective analysis of pilot programs
– “Exactly how cost-effective was that pilot program?”

• Prospective analysis of pilot programs
– “Roughly how cost-effective could this proposed pilot be?” “How big an impact must this achieve to meet our threshold?”

• Prospective analysis of programs at scale
– “Roughly how cost-effective might this proposed national program be?”

Question 9

Retrospective Analysis of Pilot Programs

List necessary data, strengths and weaknesses

Answer 9

Necessary data: cost data from an exact program that was evaluated & rigorous impact estimates

Strengths: Gives precise estimates of how cost-effective a program was in the context.

Weaknesses: Still suffers from external validity problem for cost and impact estimates

Question 10

Prospective Analysis of Pilot Programs

List necessary data, strengths and weaknesses

Answer 10

Necessary data: Projected costs; impact estimates from a similar program

Strengths: Even rough calculations can help rule out programs that can’t be cost-effective

Weaknesses: Cost projections and impact estimates from similar programs may not be accurate

Question 11

Prospective Analysis of Programs at Scale

List necessary data, strengths and weaknesses

Answer 11

Necessary data: Projected cost data for program at scale; rigorous impact estimates from pilot evaluation

Strengths: Producing customized prospective estimates are a powerful tool when speaking with country governments

Weaknesses: Impacts from small-scale pilots may not generalize to at-scale programs

Question 12

Three Key Challenges in Doing CEAs

Answer 12

1. Absence of incentives to do CEA
2. Costs are hard to gather
3. To be actionable for large funding decisions, analysis needs to be much more precise & that is not straightforward

Question 13

CEA Challenge - Absence of incentives to do CEA

Answer 13

What if the program was effective but not really cost effective?
No editorial requirement to show CEA in most social science journals

Question 14

CEA Challenge - Costs are hard to gather

Answer 14

Collecting cost data not seen as key part of evaluation
unlike impact measures

Cost data is surprisingly hard to collect from implementers (budgets different from implementation costs; hard to divvy up overhead and existing costs to project)

Hard to get cost data from other authors for a
comparative CEA

Impact measures and cost collection often not harmonized

Question 15

CEA Challenge - Precise analysis required

Answer 15

– Numerous assumptions are needed to complete the
analysis (e.g. multiple outcomes, transfers, spillover effects, exchange rates, inflation etc.)
– No one “right” way, but consistency is important!

Question 16

Which method does JPAL use primarily when gathering cost data retrospectively?

Answer 16

The “ingredients” method - gathering data from multiple sources

Research paper for description of program structure, ingredients and local conditions like wages;
Interview researchers for additional ingredients, their costs, additional documents like budgets;
Program staff and field research staff for unit cost data;
Supplement with public sources (e.g. local wages, transportation costs etc.)

Question 17

What are some challenges with the retrospective CEA approach?

Answer 17

Data not originally collected by implementer or evaluator and key field staff are hard to locate or do not respond

Many important costs are forgotten, or hard to estimate after long lag

Program as implemented may be very different from how it was budgeted

Aggregate cost data is much less useful for sensitivity analysis or scale-up

Question 18

How can prospective CEA help remediate some of the challenges with retrospective CEA?

Answer 18

Providing standard templates helps in data collection

Harmonization (outcomes, costs, methodology) makes it easier to do comparative CEA

Question 19

What are some possible things you may be calculating the cost-effectiveness of?

Answer 19

The program, during the pilot phase
The program, if it was scaled up
Some component of the program

Question 20

Aside from calculating the cost-effectiveness, what other assumptions should you include in the analysis?

Answer 20

How will you deal with…
– Exchange, inflation, discounting
– Spillover effects
– Multiple outcomes
– Costs shared with a partner organization
– Fuzzy costs: administration, overhead, and management

Question 21

Issues to consider in CEA - Present value

Answer 21

Present value - real discount rate of 10% is used to discount costs and benefits to control for time

Question 22

Issues to consider in CEA - Inflation

Answer 22

Inflation - adjust costs to today’s prices

Question 23

Issues to consider in CEA - Across countries

Answer 23

standard exchange rates are used to adjust to US$

Question 24

Issues to consider in CEA - multiple outcomes

Answer 24

can only examine one type of benefit at a time, which is how many policies are framed anyway

Question 25

Issues to consider in CEA - Total vs. sunk costs

Answer 25

only considers the incremental cost to the existing infrastructure (material, personnel, oversight)

Question 26

Issues to consider CEA - Proximal Success vs. Final Impact of Programs

Answer 26

Use global measures to translate proximal outcomes into final outcomes

Question 27

Which of the following, if changed, would likely NOT influence the cost-effectiveness estimate?

Decision to use nominal exchange rate versus PPP exchange rate
Population density
The impact estimate
Discount rate
Local cost of living

Answer 27

None of the above.

Options 1 through 5 all play into how one accounts for the costs of the program in one’s CEA. These are amongst the factors that determine how the impact estimate plays into one’s CEA. Thus, all of these factors would have some bearing on one’s cost effectiveness estimate.

Question 28

Should participant “opportunity costs” be included when conducting a CEA?

Answer 28

What costs one takes into account depends on what the purpose of one’s CEA. Is the question “which program is most cost effective for society” or “which program is most cost effective for a specific organization to run”? The former might want to take participant opportunity costs into account while the latter might not.

Question 29

One challenge of CEA is that:

Answer 29

There is no one way to do CEA

As Iqbal illustrates, there are multiple decision points involved with determining how to conduct a CEA. There is thus no single way in which to do a CEA; it is critical however that one stay consistent in one’s methodology across programs when conducting a comparative CEA.

Question 30

Issues to consider CEA - Transfers

Answer 30

Not a cost to the society but are they a part of the program cost?

International donors vs. local governments
Additional problems of non-cash transfers

Basically administrative costs vs. transfer costs

Question 31

Issues to consider CEA - Significance of Effects

Answer 31

Only report results at 10% level of significance and show confidence intervals

Question 32

Issues to consider CEA -Point Estimates vs. Range

Answer 32

Show range around point estimates to make distinction between a set of cost effective programs vs. a
set of not so cost efficient programs

Question 33

Issues to consider CEA - Context

Answer 33

If costs depend a lot on specific contexts (e.g. population density) provide ranges of cost effectiveness based on these parameters

Question 34

Things to consider in “Scale-Ups”

Answer 34

– Spillover Effects: Spillovers may be different in a pilot vs. scaled program.
– Partial vs. General Equilibrium: Very hard to measure
precise nature or direction of such effects
– Experimental vs. Scalable Mode: Costs of inputs may
become endogenous to the scale up
– Hard to Control Contextual Differences: Quality of
infrastructure, motivation of local partners and
beneficiaries, price differences, cultural differences, local parameters