Module 2 Flashcards
Wilson identifies 4 types of agencies based on whether activity and outputs of the agencies and outcomes are observable. Describe the characteristics of the four types of agencies.
- Production agencies
- Outputs and outcomes are observable. Performance evaluation is easiest in production organization where quantifiable outputs are transformed into quantifiable outcomes.
ex) tax collection agencies/ postal services - Procedural agencies
- Most activities and outcomes are observable. ex) armed forces - Craft agencies
- Activities are hard to observe. Most outcomes are observable
ex) research and auditing agencies - Coping agencies
- Neither outputs not outcomes are observable. Performance evaluation is virtually impossible for coping agencies because of the limited visibility.
ex) Central gov agencies
Define varying concepts of public sector performance and performance management
Performance management is a broad term used to characterize an organizational focus on results-based management.
A performance management cycle includes strategic planning, annual or multi-year planning, results-measurement, monitoring and assessment, and performance reporting.
Components of performance can focus on economy, effectiveness, and efficiency.
Performance management can be defined as “an ongoing, systematic approach to improving results through evidence-based decision making, continuous organizational learning, and a focus on accountability for performance.”
Three key performance constructs include:
economy, effectiveness, efficiency
Define economy, effectiveness, and efficiency.
# Economy - the price paid for resources for a given level of quality # Effectiveness - is achieved when the activities or outputs that best achieve its intended outcomes are selected. Effectiveness can be thought of as the use of best practices. This form of effectiveness contributes to the efficiencies if its outputs and activities are also cost-effective in achieving intended outcomes. Effectiveness may also refers to an ability to achieve planned outcomes. # Efficiency - A unit of output of a given quality is produced at minimum cost # Economy, efficiency, and effectiveness yield the best value for money in using resources to achieve outcomes.
Discuss the definition of efficiency.
Efficiency means that unit of outputs of given level of quality is produced at a minimum cost. It implies economy and embodies technical efficiency and allocative efficiency. # Technical efficiency is achieved when no resources are wasted in production. # Allocative and allocation efficiency requires the mix of inputs to achieve a certain level of output and the mix of inputs are selected to minimize the costs given the relative prices of input. The least costly mix of inputs for a given level of output depends on relative input prices and relative marginal productivity. # Allocative efficiency can be achieved when we choose the right mix of outcomes. Allocative efficiency relates to maximizing the social value of outcomes rather than choosing the mix of inputs that minimize the costs of inputs.
Transformation of inputs into outcomes as two separate processes.
Describe the two separate processes.:
- One that involves the creation of outputs (activities, programs, policies and regulation)
- One in which these outputs act on or with a subject or object to produce a desirable outcome
To maximize or optimize economic performance, a government needs to:
- have strong procurement policies and procedures.
- have policies and procedures that reduce the waste of resources.
- choose models and techniques for the production of services that maximize outputs for given levels of resource inputs.
- choose models and techniques for the production of services that take into account the comparative cost of substitute inputs
- choose policies and services that are the most cost-effective in producing desired outcomes.
- choose a mix of outcomes and objectives that maximize social welfare
Some of challenges with the use of performance measurement techniques are:
- a lack of fit between the organization’s needs and the performance-measurement solutions
- the complexity of accountability relationships in public and non-profit organizations.
- The attribution problem - replacing program evaluation with performance measurement
- The degree of analysis needed - organizational vs program performance
- gaming - the unintended consequences of performance measurement.
Exam Questions
- Describe differences between program evaluation (PE) and performance measurement (PM)
- PE and PM should not be viewed as competing approaches. Instead, they should be viewed as complementary evaluation strategy. Both of them are components of performance management cycles and both are intended to be part of the feedback loop that reports, accesses, contributes outcomes and programs.
- PE is discreet; PM is on-going
- PE addresses specific issues and questions; PM is more general
- PE involves development of program-specific, individualized measures and data;
PM uses existing data, routine measures and processes - PE requires specific, targeted resources ex) $, staff: PM integrates ongoing resources into program infrastructure
- PE tries to isolate specific program outcomes (by using comparison groups); standard in PM
- PE objectives clarified at outset (Terms of Reference); PM objectives and measures evolve in response to changing needs, priorities, and what has been learned.
- Program managers have more direct involvement in PM; in PE, they are usually a stakeholder but not an evaluator.
- PM is less complex and more wide-spread, but also much less informative as PM cannot tell you whether an agency is being economical, effective or efficient.
Explain the reasons why PM is less informative ?
PM is less informative as it cannot usually tell you whether an agency is being economical, effective, or efficient. PM cannot tell whether the entity was efficient in the delivery of its programs or even whether the agency is actually responsible for observed outcomes. This is due to several factors. First, agency performance cannot be aggregated into one single measure.
Second, Joint production issues arise and
there is effective environmental factors and there is generally incomplete characterization of the production process and a lack of justifiable benchmark.
Define the following:
a. Performance management
Performance management is a broad term used to characterize an organizational focus on results-based management.
Define the program evaluation.
Program evaluation is a systematic process for gathering and interpreting information about a program. Program evaluations may be formative, undertaken to determine how to improve a program or summative, undertaken to determine whether they are effective and efficient.
Program evaluation is not a technique or procedures. It is simply activities that use various analytical techniques and methods to evaluate programs and interventions.
Program evaluation is used extensively in all other sectors. Unlike performance measurement, program evaluation is concerned with cause and effect.
List the typical program evaluation questions and explain what kinds of analysis methods PE uses.
- To what extent did the program achieve its goals and objectives?
- What are the characteristics of the individuals and groups who participated in the program?
- For which individuals or groups was the program most effective?
- How enduring were the program’s effects?
- Which features of the program (ex) activities, settings, management strategies) were most effective?
- How applicable are the program’s objectives and activities to other participants in other settings?
- What are the relationships among the costs of the program and its effects?
- To what extent did social, political, and financial support influence the program’s outcomes and acceptability?
# Program evaluation adopt the following methods: Quantitative methods, cost-benefit analysis, qualitative research methods, interviews and focus groups
Define cost-benefit analysis (CBA)
CBA is a methodology grounded in welfare economics and is used to demonstrate whether a social investment will yield (prospective CBA) or has yielded (retrospective CBA) positive net benefits for society. CBA is used to determine whether an intervention generates net positive social value, and also to compare and rank alternative interventions with related but different outcomes.
CBA puts a value on the costs and benefits of a public investment and CBA does not intend to demonstrate economy and efficiency in common production processes.
CBA takes as given and assume that budgeted and actual costs embody economy in purchase of inputs, technical efficiency and the least costly mix of inputs.
Define the cost-effectiveness analysis
CEA can be thought of as a simplification of CBA. In CEA, monetary values are not calculated for interventions’ outcomes as they are in CBA. Instead, net social costs of interventions are divided by the quantitative results of the intervention, yielding a net cost per unit of outcome produced. In order for this measure to have evaluation significance, it can only be used in a comparative context, where net costs per unit of outcome produced are compared between interventions. For example, costs in a health intervention CEA would treatment costs and should also include all opportunity costs faced by patients that differ between alternative interventions.
CEA would compare 2 or more interventions to determine which can achieve a common outcome at the least cost per unit of outcome.
Define the cost-utility analysis (CUA)
CUS (Cost-utility analysis) is used for health interventions and is similar in its application to CEA. It can be used to compare interventions as is done in CEA, rather than to decide whether a single intervention produces net social benefits. In CUA, the outcome used in the denominator is a measure of quality of life referred to as quality-adjusted life years. (QALY), or a similar measure of utility, that is used to approximate the benefits of a health intervention in broad but non-monetary terms. Intervention costs are divided by QALY achieved to arrive at costs per QALY.
Define Best practice production frontier methods.
Best practice production frontier methods measure relative production inefficiency, employing information on the inputs and outputs of comparable decision-making units (DMUs). More specifically, a DMS’s degree of inefficiency is the extent to which its inputs could be reduced while maintaining its output, or equivalently, the extent to which its output can be increased while maintaining its inputs. In other words, best practice production frontier methods use the most efficient DMUs to benchmark less productive ones.
Define the performance measurement (Performance reporting)
PM is a management practice that identifies, measures, and reports on quantifiable program inputs, activities, outputs, workload or efficiency measures and outcomes in planning and reporting documents such as budgets and performance reports. Reported measures are typically called performance measures or performance indicators. Performance measurement aims to provide management and stakeholders with information that they can use to evaluate performance. However, PM does not systematically analyze the causal relationship between program, activities, and outcomes, which limits its usefulness.
Distinguish between effectiveness and efficiency in the context of program evaluation.
Effectiveness refers to how the policies/procedures of the government entity was successful in achieving its objectives/goals.
Efficiency means that the government policy is able to achieve its objectives and goals of given quality at the minimum costs. Efficiency can be categorized into technical efficiency and allocative efficiency. Technical efficiency is achieved when no resources are wasted in production; whereas, the allocative efficiency require the mix of inputs to achieve a certain level of output and be selected to minimize the costs given the relative prices of inputs. The least costs mix of inputs for a given level of output depends on relative input prices and relative marginal productivity.
Allocative efficiency relates to maximizing the social value of outcomes rather than choosing the mix of inputs that minimize the costs of outputs.
Distinguish between effectiveness and efficiency in the context of program evaluation.
A program is more effective if it is more successful at achieving its intended objectives. This differs and is less stringent than being efficient, where the focus is on achieving objectives at minimum cost (per unit of objective), or on maximizing social welfare. A program could be more effective at achieving objectives, yet inefficient compared to another less effective program because its cost is too high in relation to the magnitude of the objectives achieved.
Distinguish between effectiveness and efficiency in the context of program evaluation. Illustrate the difference with an example.
Consider two alternative medical treatments for an ailment. The first cures the ailment in 90% of cases and costs $100 per treatment. The second cures the ailment in 80% of cases and costs $50 per treatment. If each of treatments is administered to 100 people, the first will cure 90 people at an average cost of $111.11 per person cured ($10,000/90) and the second will cure 80 people at an average cost of $62.50 per person cured ($5,000/80). The first treatment is more effective, but the second treatment is more efficient because the cost per unit of objectives is lower. This is a simple hypothetical example to illustrate the concept and does not take into account a variety of complications that could occur in a real-life situation.
