W5-T3 Implementation Outcomes Flashcards
define implementation outcomes
outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda
- serve as indicators of implementation success
- proximal indicators of implementation processes
- key intermediate outcomes
recognize Proctor et al.’s (2011) taxonomy of implementation outcomes
acceptability – perception amongst stakeholders that new intervention is agreeable
adoption – the intention to apply or the application of a new intervention
appropriateness – perceived relevance of intervention of setting, audience or problem
feasibility – the extent to which an intervention can be applied
fidelity – the extent to which an intervention gets applied as originally designed/intended
implementation costs – costs of the delivery strategy, including the costs of the intervention itself
coverage/reach – the extent to which eligible patients/population actually receive intervention
sustainability – the extent to which a new intervention becomes routinely available/is maintained post-introduction
suitability of different methods for assessing implementation outcomes
understand the importance of validated and pragmatic quantitative measures
know where to identify validated implementation outcome instruments
define three types of outcomes
- implementation outcomes
(acceptability, adoption, appropriateness, costs, feasibility, fidelity, reach, sustainability) - service outcomes (i.e. A&E visits)
(efficiency, safety, equity, and timeliness) - patient/client outcomes (i.e. alcohol consumptions)
(Functions, symptoms)
what are other frameworks (beside Proctor) that can be used to identify the implementation outcomes
Consolidated Framework for Implementation Research (CFIR)
RE-AIM
how CFIR works in identifying implementation outcomes
assesses 39 constructs over 5 domain
1. intervention characteristics
2. outer setting
3. inner setting
4. characteristics of individuals
5. process of implementation
how RE-AIM works in identifying implementation outcomes
assesses 5 dimensions across individual, organisation and community levels:
1. reach
2. effectiveness
3. adoption
4. implementation (i.e. fidelity)
5. maintenance
define how implementation outcomes are measured
- qualitative interviews or focus group
- observation
- survey or questionnaires
- routinely collected data
how is the level of analysis and implementation stage considered and way to measure
Outcomes: adaptability
Level of analysis: individual provider: consumer provider
Implement stage: early/mid/late
measurement: survey/qualitative interviews/admin data
outcomes: adoption
level of analysis: individual provider: organisation or setting
Implementation stage: early to mid
measurement: admin data, observation, qualitative interview, survey
outcomes: sustainability
level of analysis: administrators: organisation or setting
implementation stage: late
measurement: case audit, qualitative interviews, questionnaires, checklists
describe how implementation outcome is used in implementation study
implementation study: improving the uptake of apps that support people quitting smoking or
reducing their alcohol intake
Implementation strategy: a text message sent to hospital patients identified as smokers or risky drinkers via the electronic health record
implementation outcomes:
acceptability: explored before sending the text messages, via focus groups with patients and staff to help develop the message content and refine the process
feasibility: explored using the hospital’s electronic health – data collection: assess how many patients
were recorded as smokers and drinkers at risky levels and whether they had mobile phone numbers.
adoption: recording the proportion of patients who access the apps: patients that had received the link to the apps via text message.
why it is important to validate implementation outcome instruments
lack of consensus on which instruments should be used for measuring the same outcome
inconsistencies in the outcomes reported and difficulties in comparing these outcomes in systematic reviews and meta-analyses.
There is variability in the quality in terms of reliability and validity of instruments, and it’s not clear if the best instrument is being used for a given outcome
define the concepts of reliability and validity
a measure has to be reliable before it can be valid, but reliability does not infer validity.
define measurement properties
reliability (reliability, measurement error, internal consistency)
validity (content validity, criterion validity, construct validity)
responsiveness (the ability of a measure to detect the change in an individual over time. )
define pragmatic measure and why we need them
to usefully inform the assessment of implementation determinants, mechanisms, processes, strategies, and outcomes. measures must be both psychometrically sound and pragmatic
define pragmatic measure construct
toward criteria for pragmatic measurement in implementation research and practice: a stakeholder-driven approach using concept mapping
what is concept mapping exercise
- asked the stakeholders to conceptualize the domains that comprise the pragmatic measure construct
- a systematic review found 47 criteria identified to be grouped into four categories: acceptable, compatible, easy and useful
define pragmatic rating scale
pragmatic measures for implementation research: development of the psychometric and pragmatic evidence scale
– 11 items or questions belonging to these four categories
– 6 point rating system assigned to each question
define three brief validated instruments
Outcome: Acceptability of Intervention Measure (AIM)
Items: … meet my approval, … is appealing to me, I like…, I welcome…
Intervention Appropriateness Measure (IAM)
Items: … seems fitting, …seems suitable, …seems applicable,…seems like a good match
Feasibility of Intervention Measure (FIM)
Items: …seems implementable, …seems possible, …seems doable, …seems easy to use
define three options for online repositories of implementation outcome instruments
Society of Implementation Research Collaboration (SIRC)
— They use the findings of their systematic reviews of implementation outcome instruments used in mental
health settings, to populate the repository
— They identify implementation outcome instruments that assess all of the 39 consolidated frameworks for implementation research constructs
– require membership
Grid Enabled Measures (GEM)
— online repository of implementation outcome instruments
— relies on crowd-sourcing, where instrument developers proactively add their publication to the repository
— Any measure can be added to this repository without any validation dataset
KCL and UEA (University of East Anglia)
— based on the findings of the systematic review of
implementation of outcome instruments in physical health settings
— allows you to search for instruments to assess the implementation outcome used in Procter’s taxonomy
— To view a summary of the instrument; the
number of items; the country of application; and the level of analysis (i.e. patient, provider, organisation)
— to consider the methodological quality of the
psychometric studies included in the repository
— to consider the instrument quality
— to view the usability rating of the instrument
— Where permission is granted, the repository provides access to both the psychometric study and the published instrument.
