Chapter 9 & Readings Flashcards
_____________ involves going beyond knowledge (real or estimated) of program impact and attempting to determine how cost effective the program is at what it does and how it compares (in efficiency) to other programs, and then, given this knowledge, deciding whether the program is worth its costs and/or if the program should be replaced with another more efficient program.
Efficiency Assessment
Efficiency Assessment is the approach typically taken by ______. It is very important, however, that all evaluators (including you!) understand how to think about efficiency assessment because of the importance of costs and budgets and resource allocation in every organization
economists
Who said this, “perhaps the greatest value of efficiency analysis is that it forces us to think in a disciplined fashion about both costs and benefits….Most other types of evaluation focus mainly on the benefits.”
Peter Rossi
What are the 2 major types of efficiency assesment?
Benefit Cost analysis and assessment
Cos-effectiveness analysis and assessment
In ___________,monetary (i.e., dollar) values are estimated for the resources used (i.e., costs) and for the program effects (i.e., benefits), and these two components ________ are then compared to determine the worth of a program.
Benefit-cost analysis or cost-benefit analysis
costs and benefits
Key Idea 1: Benefit-cost analysis can be carried out according to different perspectives, and you will need to _____________ when conducting a benefit-cost analysis.
select a perspective
Three common accounting perspectives:
- individual or target perspective, the costs and benefits are listed that relate to the individuals in the program.
- the program sponsor perspective, the costs and benefits are listed that relate to the program sponsor.
- communal or society or national government perspective, the costs and benefits are listed that relate to the community or society.
Key Idea 2: The evaluator must list all costs and benefits and then quantify these costs and benefits.
The costs and benefits need to be converted to dollar units, and any benefits or costs occurring in the future must be discounted (i.e., converting them to their present value).
present value in this context is the value of a future benefit or cost in present dollar values
benefit-cost ratio
the total dollar benefits divided by the total dollar costs, where any costs and benefits that may occur in the future are converted to their discounted or present values.
A benefit-cost ratio of 1 is the break even point; a benefit-cost ratio that is less than one means that the costs are greater than the benefits; a benefit-cost ratio that is greater than one means that the benefits are greater than the costs (i.e., the desired result).
A benefit-cost ratio of 1 is the b_______; a benefit-cost ratio that is less than one means that the costs are greater than the benefits; a benefit-cost ratio that is greater than one means that the benefits are greater than the costs (i.e., the desired result).
break even point
A second way to compare the costs and benefits is to subtract the total costs from the total benefits and determine the __________. If the _______ is positive, the benefits are greater than the costs,
Net benefits
A third way to compare costs and benefits is the __________is popular in business and it is a measure of the profit made by the program calculated as a percentage of money spent on the program.
return on investment index
(a) subtract total costs from the gross or total benefits and (b) divide the number resulting from step “a” by the total costs
prospective benefit-cost analysis (or ex ante analysis) is done before or during the planning or development of a program
Prospective benefit-cost analysis
Prospective analysis is important in helping decision makers to decide whether to develop a full-fledged version of a program idea or to abandon the program idea.
before spending money, one should consider the potential benefits and costs.
absolute benefit-cost analysis
- Select an accounting perspective to use for the analysis (e.g., select the participant perspective, the program sponsor perspective, or the communal perspective).
- List the relevant costs (including opportunity costs and externalities) and benefits of the program. Also make any relevant distributional considerations if taking the communal perspective (e.g., conduct separate analyses for different groups and use weights for costs or benefits that reduce inequality).
- Determine or estimate the dollar values of the costs and benefits of the program.
- Calculate the benefit-cost ratio (i.e., dollar benefits divided by dollar costs), net benefits (i.e., dollar benefits minus dollar costs) and/or the return on investment index.
- Make a decision about the usefulness or worth of the program (or prospective program) and make your evaluative judgment and recommendations.
absolute benefit-cost analysis
- Select an accounting perspective to use for the analysis (e.g., select the participant perspective, the program sponsor perspective, or the communal perspective).
- List the relevant costs (including opportunity costs and externalities) and benefits of the program. Also make any relevant distributional considerations if taking the communal perspective (e.g., conduct separate analyses for different groups and use weights for costs or benefits that reduce inequality).
- Determine or estimate the dollar values of the costs and benefits of the program.
- Calculate the benefit-cost ratio (i.e., dollar benefits divided by dollar costs), net benefits (i.e., dollar benefits minus dollar costs) and/or the return on investment index.
- Make a decision about the usefulness or worth of the program (or prospective program) and make your evaluative judgment and recommendations.
comparative benefit-cost analysis
- Select an accounting perspective to use for the analysis (e.g., select the participant perspective, the program sponsor perspective, or the communal perspective).
- List the relevant costs (including opportunity costs and externalities) and benefits for each of the programs being compared. Also make any relevant distributional considerations if taking the communal perspective (e.g., conduct separate analyses for different groups and use weights for costs or benefits that reduce inequality).
- Determine or estimate the dollar values of the costs and benefits for each of the programs being compared.
- Calculate the benefit-cost ratio (i.e., dollar benefits divided by dollar costs), net benefits (i.e., dollar benefits minus dollar costs), and/or the return on investment index for each of the programs being compared.
- Make a decision about the usefulness or worth of the program (or prospective program) in relation to the other comparable programs and make your evaluative judgment and recommendations.
comparative benefit-cost analysis
- Select an accounting perspective to use for the analysis (e.g., select the participant perspective, the program sponsor perspective, or the communal perspective).
- List the relevant costs (including opportunity costs and externalities) and benefits for each of the programs being compared. Also make any relevant distributional considerations if taking the communal perspective (e.g., conduct separate analyses for different groups and use weights for costs or benefits that reduce inequality).
- Determine or estimate the dollar values of the costs and benefits for each of the programs being compared.
- Calculate the benefit-cost ratio (i.e., dollar benefits divided by dollar costs), net benefits (i.e., dollar benefits minus dollar costs), and/or the return on investment index for each of the programs being compared.
- Make a decision about the usefulness or worth of the program (or prospective program) in relation to the other comparable programs and make your evaluative judgment and recommendations.
———————— is based on cost and effect data (rather than cost and benefit data as in benefit-cost analysis). In Peter Rossi’s words, in cost-effectiveness analysis the evaluator determines “the efficacy of a program in achieving given intervention outcomes in relation to the program costs.”
Cost-effectiveness analysis
STEPS FOR A COST EFFECTIVENESS ANALYSIS
- DEVELOP A LIST
- DETERMINE THE DOLLAR VALUE
- DETERMINE TOTAL COSTS
STEPS FOR A COST EFFECTIVENESS ANALYSIS
- DEVELOP A LIST
- DETERMINE THE DOLLAR VALUE
- DETERMINE TOTAL COSTS
Why would one want to conduct a cost-effectiveness analysis rather than a benefit-cost analysis?
The answer is because it is difficult to convert many kinds of benefits to dollar units and, therefore, cost-effectiveness analysis is more practical and less controversial.
What are the 2 categories that quantitative designs can be broken into?
- Those used to determine the effectiveness of an intervention
- those that are descriptive in nature such as surveys
_________ means that an evaluator has controlled the effects of variables other than treatment, in order to say with confidence that the results occurs because the participants experienced the program or activity that is the independent variable.
Internal Validity
__________ means that the sample is representative of the population, and therefore that if the treatment is applied with another group of people from that population under similar circumstances, it should be effective there as well.
External validity
Threats to Internal Validity
History Maturation Testing Statistical regression Instrumentation Differential selection experimental mortality experiemental treatment diffusion compensatory rivalry by the control group compensatory eqalization of treatments Resentful demoralization of the control group
Threats to External Validity
Lack of explicit description of the independent variable
Multiple treatment interference
The Hawthorne effect
Novelty and disruption effects
Experimenter effects
pretest or posttest sensitization
different types of measurment for dependent variables
different times of measurement for dependent variables
_____________ is a threat to validity when the research involves use of extreme groups such as participants are either the high or low end of the socres on the dependent variable measure prior to the intervention.
Statistical Regression
_____________ refers to the individuals who drop out during the study.
Experimental mortality
__________________ when the control group does not want to accept the the new way is better.
Compensatory Rivalry by the Control Group
________________________ is what happens when one group receives something and the other group receives nothing then any effects on the first group may be due to the fact that this group received something and not to the specifics of what it received.
Compensatory equalization of treatments
________________________ if the members of the control group know that the experiemental group is getting something, they become demoralized because they are not gettilng whatever that is.
Resentful demoralization of the control group
______________means that the evaluators need to provide detail so that others can know what needs to be included if they are going to implement the same intervention in another setting.
Lack of an explicit description of the independent variable
______________is an extrernal validity threat in which the participants experience several things as part of the treatment.
Multiple treatment interference
______________________ is an external validity threat that happens when the participants of the study are influenced because they think someone is watching them.
Hawthorne Effect
______________________is an external validity in which a person may embrace something because its new or are reluctant to embrace something new.
Novelty and disruption effects
__________ is a threat to external validity in which the results of the intervention maybe due to the researcher and cannot be reproduced.
Experiementer effects
_________is a threat to external validity in; which the person evaluated could be sensitized to issues because of a pretest or because they know they will be tested.
Pretest or Posttest sensitization
What are 2 other types of validity that could be important?
Treatment fidelity (not implemented as intended) Strength of treatment (the treatment was not given as strong enough dosage to justify the expectation of seeing change
What do the following symbols represent in experimental and quasi-experimental designs?
R
O
X
R- Random assignment
O-Observation
X-stands for independent variable if there is one
Describe posttest-only design.
No pretest is given but participants are randomly assigned to groups and a posttest is given.
R XO
R O
Describe single-factor multiple-treatment design.
Involves more than 2 groups R O X1 OOO (randomly assigned, pret, mentor postt, postt, postt) R O X2 OO (randomly assigned, pret, service program, postt, postt, postt) RO OOO (randomly assigned to no-treatment control, postt, postt, postt)
Describe solomon four-group design.
Developed to test the effect of pre-test sensitization of the dependent variable. R O X O (random assig., pretest, treatment, postt) R O O (Random, assign, pretest, no treatment, post t) R XO (random assig., no pret, treatment, post t) R O(random assig., no pret, no treatment, post t)
What design involves more than one independent variable and utilizes alphabetic symbols more commonly than Xs.
Factorial Design For ex.: if you have 2 variables with more than one variable A high school grad. A1 (has diploma) A2(does not have one) B Job Training B1 (part. in job training) B2(does not participate in training) A X B (interaction b/w the 2 variables)
_____________________ occurs when the participants can be divided into 2 or more groups, one of which does not get the treatment.
Quasi experimental
Static Group Comparison Design
X O
—–
O
Another example for the Bootcamp:
X O1 O2 O3 O4
———————–
O1 O2 O3 O4
__________________ quasi-experimental group design is similar to static-group comparison design, exept that evaluators use a pretest for both the experimental and control groups. OFten called a baseline.
Nonequivalent Control Group Design
O O
Describe the regression discontinuity design.
Used in evaluation studies in which the intervention and control groups are selected on the basis of scoring above or below a certain cutoff point on a test.
O C O
___________ occur when the evaluator does not have access to a control group which involved a pretest and posttest.
Single group design
_________ design includes an intervention and then multiple measures of the dependent variable at various time intervals.
Time Series Design
__________ involve in-depth exploration of a single case, such as an individual, a group of individuals, a classroom aschoo, a clinic, or even an event.
Case Studies
___________ are studies that ask questions about the social and cultural practices of groups of people. They focus on lived experiences, daily activities, and social context of everyday life from the perspective of the participants.
Ethnographic Designs
________________is a theory of language use within a social context.
Sociolinguistic Theory and ethnography
___________________ designs for evaluation are based on the belief that we can understand the meaning of events by engaging in reflection about the way we talk about them.
Narrative
___________________ states that the evaluator sets aside prior asuumptions and to understand on a deepr level as experienced in conscious and unconscious ways by the participants.
Phenomenological Studies
If qualitative and quantitiative designs are implemented as the same time in a study it is called ________design.
Concurrent mixed methods design
________________form of mixed method design, quantitative and qualitative designs are implemented fairly independently, perhaps by 2 teams.
Dialectical form
_______________________ is a design in which one data set is collected to support the larger data set in a study, although dialogue occurs between the 2 sets of data.
Embedded mixed methods design
________________ is a research design in which one type of data is collected first, and then new questions based on these findings are generated for collection of the other types of data,
Sequential mixed methods design
_____________ any research design that uses both quantiative and qualitative collection methods (either sequentially or concurrently) with the goal of prompting social change at levels ranging from the personal to the political.
Transformative mixed methods design
_________a subtype of trasnformative mixed method design in which an evaluation cycles through several phases.
Transformative cyclical mixed methods design
* particular responsiveness to diversity and cultural issues is prominent
Altschuld and Kumar present a model that involved three phases of a needs assessment. What are these three phases?
A. Planning, Conducting, and Assessing.
B. Preassessment, Assessment, and Postassessment
C. Attitudes, Learning, and Behavior
D. Input, Output, and Moderation
b
When is there a lack of treatment fidelity?
A. When the evaluator conducts more than one study at a time.
B. When the evaluator does not implement the program/study as intended
C. When the evaluator relies on an external auditor.
D. When the evaluator is part of the organization that is conducting the evaluation.
b
Cost Analysis is defined as an evaluation’s determination of whether a program’s effect was worth its cost.
A. True
B. False
A
The majority of evaluations are conducted for more than one purpose.
A. True
B. False
a
Which evaluation type or types are associated with the transformative paradigm? A. Outcome Evaluation B. Country-Led Evaluation C. Deliberative Democratic Evaluation D. B & C
D
In international development evaluation, how is the “gaining insight” purpose of evaluation typically labeled? A. Needs assessment B. Impact C. Relevance D. Responsiveness
c
The second major category of evaluation purposes is to figure out areas in need of improvement. What types of evaluations fall in this category?
A. Implementation, responsive, process, and developmental evaluation
B. Context, capacity, needs and assets assessment, and relevance evaluation
C. Impact, policy exportability, and sustainability evaluation
D. Gender analysis, deliberative democratic, and country-led evaluation
A
Which of the following is NOT a purpose for an evaluation?
A. To identify needed inputs, barriers, and facilitators to program development or implementation.
B. To determine inequities on the basis of gender, race, ethnicity, disability, and other relevant dimensions of diversity.
C. To determine how a program can be made to look positive despite its lack of impact/effect.
D. To demonstrate that accountability requirements are fulfilled.
c
When the researcher can draw clear causal conclusions from what went on in a study, this is called: A. Internal validity B. Internal reliability C. Internal consistency reliability D. External Validity
A
