Exam 2 Chapter 6

Question 1

Program Effect/Program Impact

Answer 1

Difference between outcomes that occurred with the implementation compared to the outcomes that would have occurred if the program had not been implemented

In other words, what outcomes and how much of those outcomes did the program cause?

Question 2

Unintended effects

Answer 2

Affects not intended by program (positive and negative)

Question 3

Variable program effects

Answer 3

Moderation analysis to determine if the program impacts subpopulations in different ways

Question 4

Dose-response analysis

Answer 4

It is expected that larger doses of the program will have stronger impacts. If this doesn’t happen the program may not be having any impact.

What is a sufficient dose of the program?

Question 5

Additional impact questions include

Answer 5

Relationship between fidelity and impacts

Question 6

Average effect

Answer 6

average difference attributable to the treatment

Question 7

This is a measure of what impacts: What works and what doesn’t work?

Answer 7

Intended

Question 8

This is a measure of what impacts : What works for whom under what circumstances and why?

Answer 8

unintended impacts, variable response, dose related response and fidelity

Question 9

When Is an Impact Evaluation Appropriate?

Answer 9

impact evaluation is appropriate for any program whose mission includes bringing about change in some set of identifiable outcomes

Question 10

Ways to test a new policy or program:

Answer 10

Efficacy Trial
Effectivness Trail
Proof of Concept

Question 11

Efficacy trial

Answer 11

Demonstrates that the program or policy can achieve desired outcomes

Program or policy is often carried out under ideal circumstances

Question 12

Proof of concept

Answer 12

For example – high-quality preschool programs can have life long effects

Doesn’t provide evidence that large scale effects will be achieved if program is implemented with lower fidelity and/or quality

Question 13

Effectiveness trial

Answer 13

For new program after they have done an efficacy trial

Can the program achieve its outcomes in real world not just ideal conditions?

Can the program be delivered with enough fidelity and quality to achieve outcomes?

Question 14

When is evaluation is especially appropriate for ongoing programs?

Answer 14

There may be a time when a program is modified and refined to enhance its effectiveness, accommodate revised program goals, or reduce costs.

Question 15

Counterfactuals – what would have happened without the program?

Answer 15

Outcomes in the absence of the program

Question 16

What is the best counterfactual?

Answer 16

Randomization

Can be difficult to impossible for many programs

Question 17

Fundamentals of Causal Inference

Answer 17

The Sample
Internal Validity
External Validity

Question 18

Fundamentals of Causal Inference: The Sample

Answer 18

Any program we evaluate is only delivered to a subset of the target population the program was designed to serve

Perry Preschool project only served a very small percentage of preschoolers at risk for academic underachievement in the US

Question 19

Fundamentals of Causal Inference: Internal Validity

Answer 19

Validity or accuracy of an effect estimate for the specific participant study sample used in the impact evaluation.

Internal validity increases with full participation of all program participant and use of reliable and valid measures

Question 20

Fundamentals of Causal Inference: External Validity

Answer 20

External validity

Refer to the extent to which the program effect estimates we measured for our evaluation sample will generalize to the entire target population

Would the entire population of preschoolers at risk for academic underachievement have the same positive outcomes as our evaluation sample if they attended Perry Preschool?

External validity is high if our sample is similar to the target population on characteristics that affect our outcomes

Question 21

The fundimental problem with causal inference:

Answer 21

when the outcomes for those exposed to the program are observed, their potential outcomes without program exposure must somehow be inferred in order to determine the program effect