Lecture 6: Evaluation Flashcards
What is evaluation? And what does it involve?
- A process that critically examines a program
- It involves systematically collecting and analyzing information about activities, characteristics, and outcomes.
Why evaluate?
To determine the effect(s) of a program
- Is it succesfull in preventing and/ or reducing the health problem?
Accountability
- Is the money spend wisely?
Development
- To improve its effectiveness, implementation on larger scale and/ or inform programming decisions
Ethical aspects
- Unwanted side effects
Evaluation is often omitted, why? And what are known reasons?
Evaluation is often the last consideration in planning and implementing an intervention
Known reasons are:
- Money (evaluation is expensive)
- Threat (it can be threatening for employees in terms of unexpected outcome/ the intervention might not be effective)
- Time constraints
- Already proven effective
- Intervention is still developing
What is included in an evaluation plan?
- Relevant evaluation/ research question
- Appropriate design and methods
- A program/ intervention
- Outcomes
- Involvement of stakeholders
What types of evaluation exist?
- Effect evaluation
- Process evaluation
- Economic evaluation
- What question is answered in effect evaluation?
- What does it examine and what does it provide an answer to?
“How succesfull is the program/ intervention in preventing and/ or reducing the health problem?”
- It examines the effects (outcomes) of the program
- It provides an answer to the primary (research) question
The effect evaluation / program outcomes can be described for..
- Health
- Quality of Life
- Behavior
- Environment
- Etc.
What question is answered in process evaluation?
“Why is the program (not) succesful in preventing and/ or reducing the health problem?”
What is the purpose of process evaluation? And what does it examine?
(4 imporant aspects are examined)
To understand processes in order to strengthen or improve the programme being evaluated
Examines:
1. The delivery and content of a programme
2. The quality of its implementation
3. Barriers and facilitators of implementation
4. The organizational context (Staff, procedures, inputs and so on)
What question is answered in economic evaluation?
“Is the (preventive) health intervention cost-effective in preventing and/ or reducing the health program?”
Economic evaluation includes a comparison of..
.. the costs and outcomes of (preventive) health interventions
- Intervention costs
- Treatment costs (e.g., hospital, extra visit GP)
- Societal costs (e.g., absenteeism)
Basic evaluation questions refer to…
7 items
- Reach, integrity, acceptability
- Observed change
- Internal validity
- Effect explanation
- Cost-beneft assessment
- Applicability
- Generalizability
What does RE-AIM stand for?
Reach
Effectiveness
Adoption
Implementation
Maintenance
What is the RE-AIM framework used for?
To evaluate the impact of health promotion programs
What is meant by Reach in the RE-AIM framework?
- The percent and representiveness of individuals willing to participate in a program
- Explores characteristics of study participants
What is meant by Effectiveness in the RE-AIM framework?
- The impact of the intervention on targeted outcomes (e.g., health promotion outcomes, intermediate outcomes, health outcomes)
What is meant by Adoption in the RE-AIM Framework?
- The percent and representiveness of settings and intervention staff that agree to deliver a program
- Examines facilitators and barriers for adoption
What is meant by Implementation in the RE-AIM framework?
- The consistency and skill with which various program elements are delivered by various staff
What is meant by Maintenance in the RE-AIM framework?
- The extend to which individual participants maintain behavior change long term and at the setting level
- The degree to which the program is sustained over time within the organizations delivering it
What is GRADE used for?
Rating quality of evidence and strength of recommendations
What does GRADE stand for?
Grading of…
Recommendations
Assessment
Development
and
Evaluation
What is quality of evidence? and what levels are there?
Quality of evidence reflects the extend of our confidence that the estimates of the effect are correct
Levels:
- High
- Moderate
- Low
- Very low
What factors lower quality of evidence?
5 factors
- Study limitations (bias)
- Inconsistency of results
- Indirectness of evidence
- Publication bias
- Imprecision
What factors increase quality of evidence?
2 factors
- Large magnitude effect
- Dose-response gradient
Quality of evidence from high to low
9 in total
- Meta-analysis of RCT’s
- Systematic review of RCT’s
- RCT
- Non-RCT
- Pre-post comparison
- Correlational studies
- Expert committee reports
- Case studies
- Anecdotes
