Fidelity & Adaptation Flashcards
Implementation fidelity: definition(s)
Defining implementation fidelity:
o The rigor of sticking to the original intervention as it was designed and tested; degree to which program implemented as intended by developers
o Implementation fidelity = adherence = integrity = determination of how well a program is being implemented in comparison with the original program designed (Mihalic, 2002)
o Importance increasingly acknowledged since the early 2000s
5 components of implementation fidelity (Mihalic, 2002):
o (1) Adherence = whether intervention is being delivered as designed, with all core components delivered to appropriate population by trained staff using the right protocols and materials in locations prescribed
o (2) Exposure/Dosage = number, frequency, length initially programmed for the intervention to work
o (3) Quality of program delivery = manner in which person intervening delivers a program
o (4) Participant responsiveness = extent to which participants are engaged by and involved in the activities and content of the program
o (5) Program differentiation = is defined as “identifying unique features of different components or programs”, and identifying “which elements of…programmes are essential”, without which the programme will not have its intended effect (as cited in Carroll, 2007)
Cultural adaptation: definition(s)
Culture is a complex and multidimensional construct
o Over 100 varied definitions of culture (Baldwin and Lindsley, 1994)
o Can be differentiated from “race” and “ethnicity” (but sometimes terms are used interchangeably)
o Race tends to assume a biological basis can be applied to classifying humans: yet there is more genetic variation within races than between races; race more of a social construct than a biological one
o Ethnicity attempts to further differentiate racial groups according to lineage (e.g. country of origin, location, religion, language); like race, it carries its own historical, political and social baggage
o Knowing someone’s ethnic identity or national origin does not reliably predict beliefs and attitudes
Culture is a general system of behaviors, beliefs, values, symbols shared by a group of people, and learned and transmitted through participation in group life
o Everyone has a culture: it is not just the group a person is born into; it can evolve and involve multiple spheres
o Cultures are not static: they grow and evolve in response to new circumstances, challenges and opportunities
o Culture is not deterministic: different people take on and respond to the same cultural expectations in different ways
o Cultural “differences” are complicated by differences in status and power between (and within) cultures
Defining cultural adaptation:
o Not only how society categorizes an individual but also how one categorizes/identifies themselves
o Cultural adaptation: “The systematic modification of an evidence-based treatment or intervention protocol to consider language, culture and context in such a way that it is compatible with the client’s cultural patterns, meanings and values” (Bernal et al., 2009)
o Cultural adaptation doesn’t entail completely discarding programme theory / theory of change—instead, uses theory and procedures from initial efficacy trials as a base, but informs that knowledge with input from the subcultural group being targeted (Barrera et al., 2013)
Implementation fidelity: arguments for
PRESERVATION OF CAUSAL MECHANISM
“Using implementation fidelity information in the analysis of intervention effectiveness is important because fidelity outcomes (a) are related to the internal validity of an intervention study, (b) allow increased confidence in attributing improvements to the intervention, and (c) may increase statistical power by controlling for error associated with diminished implementation quality” (Breitenstein, 2010)
“It is only by making an appropriate evaluation of the fidelity with which an intervention has been implemented that a viable assessment can be made of its contribution to outcomes, i.e., its effect on performance” (Carroll, 2007)
EXAMPLE: Welsh National Exercise Referral Scheme (NERS)
o In the study of the Welsh National Exercise Referral Scheme (NERS), Moore et al. (2013) provided evidence on how the poor implementation of the exercise referral intervention resulted in ambiguous process evaluation outcomes
o The NERS aimed to increase the physical and mental health of the at-risk population by improving their involvement in a group physical activity (Moore et al., 2013)
o HOWEVER, the NERS program developers overlooked the importance of maintaining a reliable monitoring system throughout the implementation [risk to implementation fidelity]
o The absence of the monitoring system resulted in the limited delivery of the motivational interviews (MI) and one-to-one consultations between the patients and the exercise professionals
o Thus, the lack of implementation fidelity negatively affected the final process evaluation that, in turn, could not adequately assess the effectiveness of the whole intervention (Moore et al., 2013)
Promotes correct utilization of core components—only way of really gauging whether ineffectiveness is due to poor implementation or inadequacies of the programme itself (see: Type III error)
Essential for replicability:
o “Research indicates that the diffusion of effective interventions typically yields diminishing returns as the process enfolds” (Durlak, 2008)
More fidelity, greater behavior change (Mihalic, 2002)
The greater the number of modifications present, the greater the likelihood that key components linked to effectiveness are altered/removed (Mihalic, 2002, p. 11); danger of Type III error
Internal validity
o Still space for adaptation so long as core components remain untouched
o If low internal validity, cannot adapt
o If robust internal validity, and core components can be identified, adaptation is possible—core/non-core components must be identified at the outset
o If many elements exist in an intervention, and we don’t know the exact nature of how they interact with one another to produce the desired outcome(s), changing things slightly may result in harm/ineffectiveness
o Internal validity not solely concerned with adherence, but also dosage, exposure, quality of delivery, and participant responsiveness
Practicality: time, cost, roll-out of intervention
Ethics: assumption that in an original intervention, harm is mitigated and considered, but cannot guarantee the same with an untested/unpiloted adaptation—’first do no harm’ principle; equipoise
o Any adaptation increases the potential for unpredictable outcomes and, as such, increases risk of harm and wasted resources
Cultural adaptation may increase effectiveness for certain populations, but it is not the key driver of actual impact
Cultural adaptation: arguments for
Transportabilty of an intervention
Reaching/tailoring to marginalized, vulnerable or traditionally overlooked groups
More participatory; accounts more for target populations and creates formal space for their input
o Can build greater trust with the community
o One specific form of cultural adaptation is “constituent-involving”; i.e., drawing on cultural knowledge and experience (Barrera et al., 2013)
Strong external validity
o Griner and Smith (2006) meta-analysis of culturally adapted mental health interventions: effect sizes (d = 0.45) suggest a moderately strong benefit of culturally adapted interventions
o Interventions targeted to a specific cultural group were found to be more effective than those for mixed groups
“Whereas the ‘gold standard’ for a replication had been strict fidelity to the content and procedures of the original project, we now understand that slavish [uncompromising] fidelity may result in an intervention that is faithful to the form, but not the essence, of the original” (Morrison et al., 2009)
Implementation fidelity: arguments against
Rigidity of implementation fidelity reduces the applicability of an intervention—not only about research for research’s sake, but also about achieving practical change and impact (i.e., external validity)
Run the risk of ignoring or further marginalizing certain traditionally overlooked groups
Hard to identify core components in different contexts
No standardized means to measure fidelity
The study of implementation process has been limited to date (Backer, 2001): o Durlak (1997) noted that less than 5 percent of more than 1200 published prevention studies provide data on program implementation o Durlak and Wells’s (1998) meta-analysis of prevention programs found that 68.5 percent were described too broadly to be replicated and few included measurement of fidelity o In reviewing school-based prevention programs, Gresham, Gansle, Noell, Cohen, and Rosenbaum (1993), found that only 14.9 percent measured implementation integrity o Dane and Schneider (1998) assert that “promoting” integrity (e.g., use of a manual, providing training, etc.) is not the same as “verifying” integrity (gathering research data that these interventions were effective in high-quality implementation)
Cultural adaptation: arguments against
May result in an entirely new intervention, with a different programme theory / theory of change than the initial intervention
Some studies hint at higher retention but lower effectiveness of cultural adaptations: o Gottfredson (2002) SFP trial in DC: adaptations helped increase engagement, but intervention was ultimately not effective o Kumpfer et al. (2002): found increased retention of up to 40% for culturally adapted family-based interventions, but positive outcomes were simultaneously reduced
Risks further segmentation of cultural groups
Impossible to adapt to each culture
Costly, time-consuming:
o Parents in the intervention group of Baumann, Rodríguez, Amador, Forgatch, and Parra-Cardona’s (2014) Parent Management Training Oregon (PMTO) study in Mexico were engaged in activities such as training and coaching by local PMTO mentors
o HOWEVER, it took 18 months to train the local PMTO coaches at the professional PMTO and the fidelity implementation (the FIMP) training before the intervention (Baumann et al., 2014, p. 34)
More piloting required
Potentially difficult to directly compare cultural adaptation to original intervention—comparison to SoC [standard of care] is the norm, but that might be an inaccurate comparison (Barrera et al., 2013)
o Using finite resources for adaptation, while not knowing if the intervention will work/be beneficial, risks increasing harm and/or ineffectiveness
