Process evaluation Flashcards
why does process evaluation exist
asks the question: are we creating evidence-based interventions relevant to government policy?
Government response act 2011
UK house of lords inquiry into behaviour change 2010-2011
what was said by Baroness Julia Neuberger
- “Many of the goals to which governments aspire—such as bringing down levels of crime, reducing unemployment, increasing savings and meeting targets for carbon emissions—can be achieved only if people change their behaviour.”
“Many of the goals to which governments aspire—such as bringing down levels of crime, reducing unemployment, increasing savings and meeting targets for carbon emissions—can be achieved only if people change their behaviour.”
process evaluation looks into how we can be produce interventions which will actually cause individuals to change their behaviour
lack of..
population-level trial data on long term effectiveness of behaviour change interventions
i.e. lack of research looking at interventions for populations, rather than specific groups
due to their being a lack of evidence of population-level interventions
- lack of evidence of cost-effectiveness and long term impact
- barrier to the formulation of evidence based policies to change behaviour
importance of evaluation
- should be considered at the beginning of the policy design process
- external evaluation from experts should be sought
if you create a good intervention but don’t evaluate appropriately
not a very useful intervention
intervention mapping: design and evaluation framework
1) evaluation of needs assessment
2) logic model
3) program design
4) program production
5) implementation
6) evaluation
what are measurable change objectives
what will the intervention chaneg
identification change mechanisms (logic model)
determines process evaluation e.g. making it harder to get alcohol by making sure that shops and bars ID
logic models must be delivered in a way attractive to its target
identification of feasibility, attractiveness and sustainable delivery formats
PPI can be used to
identification of feasibility, attractiveness and sustainable delivery formats
–> avoiding later failures
co-creation of interventions e.g.
with those who will deliver and receive them e.g. a head teacher
planning evaluation
before materials are created
o. e. how are we going to measure if young people drink less?
- Otherwise we will never know if intervention is effective
Abraham et al (2016) names twelve broad characteristics of behaviour change interventions
1) specific behaviour changes targeted
2) modifiable process operating at different ;levels that regulates relevant behaviour patterns
3) change techniques known to alter identified regulatory process
4) modes of delivery to be used
5) intervention components i.e. the material needed
6) the setting in which the intervention will be delivered e.g. school place
7) the fit between intervention components and the cultural and practical context in which will be used
8) characteristics, qualifications and training of those delivering the intervention
9) intensity (contact time in each session
10) duration (number of sessions and overall period of intervention)
11) widely of delivery (were lessons delivered as designed)
12) evaluation, including outcome, process and economic evaluations
modes or formats of delivery i.e.
face to face meeting, telephone calls, innovative online programmes, leaflet distribution etc
setting of intervention e.g.
school, worksite etc
intensity
contact time in each session
duration
number of sessions and overall period of itnevrnetion
widely of delivery
were lessons delivered as designed
two main questions outcome evaluations ask
- DOES IT WORK?
- HOW DOES IT WORK?
Outcome evaluation: DOES IT WORK?
- effect size/ D value
- efficacy vs effectiveness
- observed differences and validity power
effect size also know as
d value
d value
is an effect size used to indicate the standardised difference between two mean
- it can be used to accompany reporting of t-test and ANOVA results
- its also widely used in meta-analysis
Cohen’s d
is an appropriate effect size for the comparison between two mean
