Chapter 9 Flashcards
Data Analysis Issues in Program Evaluation
Evaluators must choose analysis techniques that will allow them to make judgments about practical and clinical significance
Consider the costs of both Type I and Type II errors
Design program monitoring systems that can be used appropriately by program staff
Statistical Significance in Program Evaluation
Statistical significance is the first and lowest threshold for determining that a program is having a meaningful effect
The .05 alpha level is a matter of convention
A more or less stringent alpha level can be set depending on circumstances
Minimum Detectable Effect Size (MDES)
Evaluators define the smallest effect size considered practically significant and design their studies to detect effects at or above this threshold
Decreasing Risk of Type II Error
Increasing sample size is the most common approach in research
Often not possible in evaluation research
Remove error variance by controlling for covariates in the analysis
Pretest scores
Demographic characteristics
Moderation analysis
Moderation analysis allows you to determine if the program is working equally well for all subgroups of participants
Variables used for moderation analysis
Program dose (dose response analysis)
Dose response most often tested by a correlation or regression between program exposure and outcomes
Demographic characteristics
Risk factors/Pretest outcome scores
Differential effects by demographics or risk factors in moderation analysis
Interaction between moderator and treatment status on outcome
Divide participants into subgroups and test for differences on outcome
Additional Moderation analysis information
In addition to determining if the program has differential impacts on subpopulations moderation analysis is more rarely used to test program theory
Drunk driving reduction intervention expected to have stronger impact on reducing accidents during Friday and Saturday nights than weekdays during the work hours.
Mediator Analysis (textbook defenition)
Focuses on causal pathways, identifying intermediate variables (e.g., knowledge gained) that explain how programs achieve their outcomes
What is mediator Analysis used for?
Used to test program theory
Does exposure to specific intervention lead to changes in proximal outcomes and do changes in proximal outcomes change more distal outcomes?
mediator Analysis (slides def)
Complex mediation tested by complex statistics usually structural equation modeling
Simple mediation process
Program activity
|
Proximal Outcome
|
Distal Outcome
.Tested by a series of simple regressions
Reporting Evaluation Results to steak holders
After you interpret your results taking significance level, effect size, moderation and mediation into account you must report them to stakeholders in an understandable way
Understanding reporting metrics
Understandable metrics
Direct report of differences on measures with practical meaning
Conversion of score into meaningful measure
Convert achievement test score into a grade level
Movement of individuals from one diagnostic category to another
more understandable metrics
Proportion of individuals who reached or exceeded a preset threshold
Poverty-level
Set by program staff
Odds ratios
Percentage of improvement
Comparison with normative group
Comparison with similar programs
