Process evaluation Flashcards
why does process evaluation exist
asks the question: are we creating evidence-based interventions relevant to government policy?
Government response act 2011
UK house of lords inquiry into behaviour change 2010-2011
what was said by Baroness Julia Neuberger
- “Many of the goals to which governments aspire—such as bringing down levels of crime, reducing unemployment, increasing savings and meeting targets for carbon emissions—can be achieved only if people change their behaviour.”
“Many of the goals to which governments aspire—such as bringing down levels of crime, reducing unemployment, increasing savings and meeting targets for carbon emissions—can be achieved only if people change their behaviour.”
process evaluation looks into how we can be produce interventions which will actually cause individuals to change their behaviour
lack of..
population-level trial data on long term effectiveness of behaviour change interventions
i.e. lack of research looking at interventions for populations, rather than specific groups
due to their being a lack of evidence of population-level interventions
- lack of evidence of cost-effectiveness and long term impact
- barrier to the formulation of evidence based policies to change behaviour
importance of evaluation
- should be considered at the beginning of the policy design process
- external evaluation from experts should be sought
if you create a good intervention but don’t evaluate appropriately
not a very useful intervention
intervention mapping: design and evaluation framework
1) evaluation of needs assessment
2) logic model
3) program design
4) program production
5) implementation
6) evaluation
what are measurable change objectives
what will the intervention chaneg
identification change mechanisms (logic model)
determines process evaluation e.g. making it harder to get alcohol by making sure that shops and bars ID
logic models must be delivered in a way attractive to its target
identification of feasibility, attractiveness and sustainable delivery formats
PPI can be used to
identification of feasibility, attractiveness and sustainable delivery formats
–> avoiding later failures
co-creation of interventions e.g.
with those who will deliver and receive them e.g. a head teacher
planning evaluation
before materials are created
o. e. how are we going to measure if young people drink less?
- Otherwise we will never know if intervention is effective
Abraham et al (2016) names twelve broad characteristics of behaviour change interventions
1) specific behaviour changes targeted
2) modifiable process operating at different ;levels that regulates relevant behaviour patterns
3) change techniques known to alter identified regulatory process
4) modes of delivery to be used
5) intervention components i.e. the material needed
6) the setting in which the intervention will be delivered e.g. school place
7) the fit between intervention components and the cultural and practical context in which will be used
8) characteristics, qualifications and training of those delivering the intervention
9) intensity (contact time in each session
10) duration (number of sessions and overall period of intervention)
11) widely of delivery (were lessons delivered as designed)
12) evaluation, including outcome, process and economic evaluations
modes or formats of delivery i.e.
face to face meeting, telephone calls, innovative online programmes, leaflet distribution etc
setting of intervention e.g.
school, worksite etc
intensity
contact time in each session
duration
number of sessions and overall period of itnevrnetion
widely of delivery
were lessons delivered as designed
two main questions outcome evaluations ask
- DOES IT WORK?
- HOW DOES IT WORK?
Outcome evaluation: DOES IT WORK?
- effect size/ D value
- efficacy vs effectiveness
- observed differences and validity power
effect size also know as
d value
d value
is an effect size used to indicate the standardised difference between two mean
- it can be used to accompany reporting of t-test and ANOVA results
- its also widely used in meta-analysis
Cohen’s d
is an appropriate effect size for the comparison between two mean
smaller the effect size
the larger the sample needed
outcome evaluations: HOW WELL DOES IT WORK?
effect size and clinical relevance
process evaluation asks
how does it work? the mechanisms and techniques
for whom does it work and in what setting? group setting- moderators
economic evaluation
how much doe sit cost- is it out effective
process evaluation definition
a study which aims to understand the functioning of an intervention by examining examining implementation, mechanisms of impact and contextual factors.
Process evaluation is complementary to, but not a substitute for high quality outcome evaluation
process evaluation takes place
within an RCT study usually
- evaluates the intervention
- collects the data
- explains how outcomes are reached by
how doe process evaluation explain how outcomes are reached
by examining:
- implementation
- mechanisms of change
- context
e. g. a condom intervention would be received very differently in e deter than south africa
process evaluation may be conducted using
mixed methods data collection
mixed methods data collection
e.g. qualitative analysis of interviews and quantitative analyses of questionnaires
complex interventions
an intervention comprising multiple components which interact to produce change
- complexity may also relate to the difficulty of behaviours targeted by interventions, the number of organisation levels targets or the range of outcomes
implementation
the process through which interventions are delivered and what is delivered in practice
3 factors should be considered when deciding how the intervention is delivered (3)
fidelity
dose
reach
fidelity
the consistency that is implemented with the planned intervention
- delivered oer protocol
measuring fidelity s a key part of making sure
the intervention is having the effect
-reduces risk of confounding
dose
how the intervention is delivered
reach
the extent to which a target audience comes into contact with the intervention
what counts as process evaluation. how much process does a study have to examine before it can be a process evaluation? how much does it have toe explain? In how much detail?
a minimal qualitative study (just looking at association between a couple of variables can be a simple process evaluation)
even just looking at association between a couple of variables
can be a simple process evaluation
to what extent should a process evaluation explain outcome?
at least partially
what is a theory
a system of interrelated proposition that should enable phenomena to be described, explained , predicted and controlled
quality of theory
- does it explain the phenomenon of interest?
- does the theory contain unambiguous concepts?
- are the relationships between and among the concepts clearly articulated?
- are the theoretical propositions empirically testable?
- has it been verified by data?
uses of theory in evaluation
proposes explanations of underlying causes or influences of phenomena
theories give researchers
different lenses through which to look at complicated problems and social issues, focusing heir attention on different aspects of the data and providing framework within which to conduct their analysis
during interpretation of data
theory development may be an output of the research
identifying mechanisms of change. Information, motivation, behaviour skills model:
a theoretical grouping mechanism
explain the info, motivation, behaviour skills model: a theoretical grouping of mechanisms
- information and motivation and related in they way that new information can act to motivate someone to develop behavioural skills which change behaviour.. Or motivation can cause individuals to look for information which change their behavioural skills
e. g. both info and motivation cause new behavioural skills to be developed and this impact behaviour - info and behaviour can also directly affect behaviour
example: helping motivated, overweight and obese women to lose weigh
- research question- does action and if-then planning help motivated people to lose weight
- sample:RCT of 45 women attending weight watchers classes
- intervention: single session add-on “ planning” session (including if- then planning)
- outcome: weight loss two months later
Barry and Kenny 1986 on mediators
In general, a given variable may be said to function as a mediator to the extent that it accounts for the relationship between the predictor and the criterion.
mediators explain
how external physical event take an internal psychological significance
mediators speak to how or why
such effect occur
intervention design should be based
on a theory of change or specified mechanism or programme which explains how an intervention is meant to work
process and evaluation int theory testing is useful in
distinguishing between ‘implementation theory and ‘theory of change’
implementation theory
how an intervention is delivered
theory of change
how an intervention brings about change in the target population
no theory is perfect
so the evaluator should allow for unexpected data and be prepared to refine theory later on
logic models:
representations of the theory of change
- only identifies factors which the intervention can change and that can be measured as outcomes
greater specificity
greater theoretical clarity
ideal logic models will
represent context, mechanisms and implementation factors
-and identifies how they are related to each other
logic model is designed for
the intervention rather than just applying the standard template
a simple logic model example
1) IV intervention- causes?
2) planning- causes?
3) DV weight loss
the results of the intervention may suggest
that the theory of change (and therefore logic model) was inadequate and needs to be changed/ developed
- such insight allows process evaluation to drive theory of development
