chapter 8; implementation and evaluation Flashcards
a soft launch is when the program planners do not call a lot of attention to initiating the intervention. one or several sites may be used to test out procedures and ensure that all processes are running smoothly before expanding the program to the intended scale
yes
testing out the intervention in this manner is called pilot testing
regardless of the decision of whether to run a pilot test, most programs set a hard launch data (after the pilot, if one will be used) and hold a partner kickoff meeting, news event, town hall, or other activity designed to generate interest and enthusiasm in the program
shortly after the launch, program managers will monitor all aspects of the intervention to ensure they are functioning as planned
these monitoring activities constitute process evaluation. in operational settings, corrections - or continuous quality improvement - can still take place
yes
in experiments (e.g. the effectiveness trials of interventions), protocols would need to be resubmitted to institutional review boards before making major changes.
yes
throughout the programs implementation, program managers will want to collect and review some key metrics with stake holders to see whether objectives are being reached
after the intervention has run for long enough to have reached a sufficiently large audience and potentially made an impact, an evaluation of outcomes can then be conducted
the evaluation design depends heavily on the logic model for the intervention, which must take opportunities and threats into consideration.
two of the key tools used to build a program, are the logic model and SWOTE analysis, are also used in its evaluation
SWOTE stands for “strengths, weaknesses, opportunities, threats, and ethics”.
to see if there has been a change, it is essential to collect what are called baseline data before launching the program
therefore, even though evaluation often comes at the end of an intervention it must be included in the design from the beginning
logic model, “a picture of how you organization does its work- the theory and assumptions underlying the program”
or
a logic model displays the connections between resources, activities and outcomes. as such it is the basis for developing a more detailed management plan. during the course of implementation, a logic model is used to explain, track and monitor operations, processes and functions. it serves as a management tool as well as a framework to monitor fidelity to the plan
preparing logic model; resources/inputs range from tangible items, such as money, paid staff or volunteer hours, facilities, and shipment, to more intangible items, or the involvement of collaborators at the state, national , or global level.
preparing logic model; activities is a description of what the program will do . a communication program often includes the following activities; [..] activities, outputs, outcomes, impact
- mass media such as paid advertising, public service announcements, entertainment education events (plays, television program scripts, radio soap operas), social media, or student classroom materials
- patient decision aids such as computer animations, decision software, pencil and paper flipbooks, kits, or worksheets
- training workshops for healthcare providers, teachers, coaches, or clergy to use media effectively
- outreach and education activities such as health fairs, speaking engagements, trade shows, or grocery store or drugstore promotional activities
numbers are associated with activities in terms of what is anticipated and budgeted; for example,
conduct 10 teacher training workshops” or “distribute 10,000 patient decision aids”
logic model; outputs
outputs describe whether the activities were delivered as planned; for example ; for mass media
- mass media ; if you planned to deliver 10k student workbooks, how many students actually received workbooks. if you bought radio and television air time, when and where did you spend air? what was the measured audience. if you ran a website, how many pages, downloads, or forms were completed?
- for patients; how many health facilities, practices, or individual healthcare providers have agreed to use the patient decision aids? how many patients have received them ?
- for intermediaries (people using the health communication materials to work with others): how many workshops were implemented? how many participants completed pre and posttests? how many materials were ordered post training?
- for outreach: how many public appearances, speaking engagements or health fairs were held? where and when ? Who came?
assessing whether your outputs were generated according to plan is an essential prelude to outcomes
logic model; outcomes
are normally divided into immediate, intermediate, and long term ranges.
- immediate outcomes for mass media might include the follow (next day recall of message, awareness of issue, change in attitude or motivation to try something). responses would be assesses through survey research.
- intermediate outcomes; generally include changes in individual behaviors, enactment of policies, or uptake of technology or strategies by organizations.
- Long term outcomes may be identical to the impact, although the former refers more to individuals 9e.g. an individual quits smoking and their life is extended) and the latter to population (e.g. the death rate from tobacco goes down)
logic model; impact is generally measured in terms of population level health or socioeconomic improvements. this would be reflected in reductions in age specific mortality rates, prevalence of disease, and disease specific mortality rates. population measures of quality of life would improve
logic model; putting the model together. should fit on a one printed page (it can be a big page)
logical model; backwards planning with logic models
taking your logic model in hand, works backwards and ask;
1. what needs to happen in the medium term for these objectives to be met?
2. before that, what needs to happen in the near term?
3. step back again, and ask: what activities must be conducted, and which what audiences, to achieve these short term outcomes
4. what resources are needed to conduct these activities
a SWOT analysis is the standard business tool for assessment of strengths, weaknesses, opportunities, and threats.
new addition oof an E to SWOT for ethical assessment which we feel is an indispensable piece for health communication programs
SWOT;
strength and weaknesses refer to conditions that are within your control, or at least inherent to your organization’s ability to implement the program .
SWOT;
opportunities and threats are outside of your control. you may plan for or around them, but they will happen on their own.
SWOT;
ethical considerations may reside inside your organization, but it is important to assess whether intentionally, or inadvertently, your program could harm someone, limit someones rights, or promise something that cannot be delivered
- strengths are the attributes within the organization. these include personnel capabilities, experiences, material resources, organizational commitment, time allowed, and budget
- strengths are the attributes of the intervention/communciation campaign. these include the positioning of a product or service, its cost, its attractiveness or reputation, and so forth. if you have a great product to service to work with (e.g. a free pizza nigh on a college campus), that is a program strength. if you have na unpopular idea to sell (e.g. an increase in tuition), this can be a weakness
weaknesses (SWOT);
- gaps within the organization; these include a lack of knowledge, skills, experience, or material resources. weaknesses may also be less tangible, such as a lack of leader commitment to the intervention or a nonexistent or poor reputation in the community, or within a government or other bureaucracy. of course, insufficient funding or a rushed timelines are also program weaknesses.
- weaknesses of the actual intervention; these may include delayed or limited availability (such as seasonal flu vaccine in some years), costs to produce (if more than the market price to sell), or distribution hurdles, or it may be that the service is unattractive to consumers in some way (e.g. think about having to promote a stool blood test or colonoscopy for colon cancer screening
opportunities , SWOT;
positive facts related to happenings at the time and in the place you have planned for your intervention. partners and stakeholders may provide critical insights into opportunities (as well as threats), which you may not know exist:
a favorable political climate, funding, technology development and innovation, seasonal and style trends, big events that draw a lot of attention - celebrity endorsements
- political; leaders can play an important role in supporting or thwarting health promotion activities within a state or congressional district
- funding; coalitions that come together solely in response to the availability of funding often have a hard time maintaining their collaboration
- technology; enormous changes In terms of what is possible, as well as reductions in cost, to reach many people with more information
- trends; there are always trends and seasonal reasons that may support (or threaten) an intervention.
big events; rather a person can become a spokesperson and do much more for the health communication program before or after the event itself (e.g. Super Bowl commercials) - celebrity; if a person is well known, can be see as a positive role model, efforts are made to attract the celebrity as a spokesperson
SWOT; threats
threats are factors that could potentially delay or prevent you from achieving your program objectives; again, they are outside your immediate control
- political instability, environmental catastrophe, activities linked to risky funding or dependent on personalities
- political; International work is frequently threatened by clashing political parties, including uprisings, strikes, and localized conflicts. conflict and war have to be listed as the biggest threats, not only to the work of public health practitioners, but also obviously to the entire
- environment; powerful weather that occurs seasonally or randomly, and agricultural conditions all jeopardize health communication efforts.
- funding; organizations, academic institutions, or even small groups can all suffer externally created losses in funding or other resources (the actions of one individual may threaten the reputation of an entire organization or institution) celebrities risky.
ethical dimension;
utilitarianism , deontological principles, the golden rule, other rights and privileges
- utilitarianism; defined as the “greatest food for the greatest number of people”. it is central to public health, and much of public policy. requires forecasting results and presents the possibility that the “ends justify the means”
- deontological principles; duty based principles. follow “stick to honorable principles, and the outcomes will take care of themselves”. many believe you cannot achieve a just outcome (or end) through unjust means. uses rules
- golden role; “love thy neighbor as thyself” also known as “do unto others as you would have them do unto you.”
- other rights and privileges; how far an individuals right to liberty in the pursuit of happiness may go is often described, in common speech, as far as the end of someone else’s nose. (decoration of independence phrase)
ethical dimensions are impossible to apply simultaneously
ye
SWOTE is in terms of when and how It might affect the process
final strategy analysis;
strength and weaknesses that are most likely to affect the programs inputs;
- is the program based on the organizational strengths and those of the intervention? - if not, how can these strengths be featured more prominently in the intervention? - how can the program fix each weakness?
- do the strengths and weaknesses of the partners balance each other out? this can be a critical question when deciding on partner arrangements
- is the proposed program too far away from the core business of the organization (e.g. If the mission of the university is to educate students, may you conduct a health communication campaign in the community?)
- do you need to rethink the pram before too much is invented ?
final strategy analysis;
opportunities and threats
- how can the program exploit each opportunity? - what must be changed to exploit an opportunity? - what does it maximize: achievement of outputs or outcomes?
- how can the program defend against each threat? - how realistic are the threats, and how great a risk do they pose? - at what point do you need to account for the threats: between inputs and outputs or between outputs and outcomes?
ethics; how can the program/organization be fair and conduct the intervention in the most ethical manner possible?
- what are the bases of these decisions?
- again, what needs to be changed, if anything, to prioritize human rights, gender equity, or other ethical issues over short term programmatic gains ?
- do you need to add partners to accomplish these changes?
analysis is best done with your partners and stakeholders
compare the strengths and opportunities for your program against the weaknesses and threats, you will make a “go/no-go” decision
production and dissemination factors;
- presentability; quality saves money, more attention, possibly keep
-expandability; trained counselors working with clients on behavior change, interpersonal communication. but, quality becomes more of an issue
- sustainability; a strategic health communication program matches partner organizations as intended audiences and spreads the costs of campaign broadly (broader reach + can afford)
- cost effectiveness; strategic communication use resources creatively and to best advantage based on audience research and process evaluation
implementation plan;
- what will be achieved?
- who will do the work?
- what are the expected roles and responsibilities of partners and allies?
- how much will all this cost?
- what is a realistic timeline ?
- what? ; defines the activities to take place according to a schedule and sequencing. need to account for external events (school, holidays that could compete for audience time)
- who? ; activities can be done by professionals or volunteers, provided they have the right mix of skills
- partner/ally roles; potential partners with whom to work. can determine the success or failure of a program
- how cost? you will need to trade off time for money - and must create budget. personnel salaries for anyone with expertise that you cannot acquire for free.
- timeline; time to plan and produce, including conducting formative research, developing and testing material and varying everything for dissemination.
implementation plan types of costs;
- (direct costs are the part of the budget that contributes to a programs outputs),
- (nonpersonnel costs; concept to program outputs that are provided by vendors) (travel, equipment)
- (indirect costs; what it costs agency to exist, but they are not tied directly to creating the programs outputs) (water, heat, air conditioning)
- ( in kind contributions; in the nonprofit world in which public health often operates, organizations frequently consider their time, space, use of equipment, and other in house resources as in kind contributions to a project budget, not factored into the direct or indirect costs of the budget)
- (total budget) ; is the sum of direct and indirect costs, including in kind contributions and additional donations
the implementation, advocacy; if you project is meant to garner support for a local issue, such as funding for an under five health or education program, you might want to hold a local town hall meeting to launch your activities.
get the press involved and engaged (free publicity = raise awareness, allows to leverage the coverage with other media involvement)
two ways to interact with members of the media are through the use oof a press advisory (sometimes also called a media advisory)
press relaese
the press or media advisory is a short, one page alert that lets media known about upcoming events
think of it like an invitation that provides the who, what, where , and why what you are doing
a press release, is written like an article you might find in a newspaper or news website. fast about the health issue and quotes from key stakeholders to stimulate pickup y news outlet to generate print and broadcast coverage
in smaller media markets, press releases may be published in their entirety, which ensures all the information you are trying to get out to your audience is presented
manage your social media; the implementation plan
manage, spend time creating, curating, and releasing your posts
implementation plan, monitoring and evaluating your program; c
collect data at each stage of planning and implementation
determine;
- whether activities are functioning as planned: (called process evaluation interchangeably used with monitoring) = collection of data from the same sources over set periods of time, whereas process evaluation is often done only one or twice at the outset of a program
- contribution of each activity to the overall effect; based on programs logic model and requires collecting data about the effectiveness of delivery
- whether program is effective at accomplishing its goals; is it reducing barriers that prevent change and increasing desired behaviors?
- whether unintended consequences are taking place, so that the program can be modified as needed: e.g. health communication program aimed at increasing partner testing may have the unintended consequence of increasing gender based violence
- which actives should be continued and/or scaled up ?
process evaluation ; monitoring and evaluating your program
tells whether your project is functioning according to plan. in logic model, this step measures your inputs and outputs. depending on program, you may measure dissemination, reach, training results, c customer satisfaction with services or interventions, and so on
process evaluation;
measure dissemination, reach, training results, c customer satisfaction with services or interventions, and so on
- dissemination; tell you what went “out” from your organization. might measure; - by media channel, number of impressions or placements of information (magazines, billboards) - number of materials disseminated to intermediaries, such as teachers,
- where and when your social media posted
implementation and evaluation ; reach
measures what was actually delivered or received. “next day recall” type survey (usually delivered by telephone) to assess whether broadcast media were attended to . social media provide more easily accessed data
guidelines for reach evaluation;
- establish a baseline
- create a multichannel dashboard (allow for comparing channels against each other)
- identify the top channels with google analytics,
- use SMART goals for social media (includes measuring the number of visits, leads, and “customers” within specific time frames)
- drill down on individual posts (consider each post as a message)
- identify your influencers (when someone retweets or resins your stuff, thank them)
- use hostages (#) to your advantage
training preparation; if using personnel in project, prepare them to deliver the intervention in a consistent manner. pretest of knowledge, attitudes, and confidence conducted before training, and then a similar assessment afterwards
when interested in assessing how the target audience perceives the quality of an interaction, can provide a customer satisfaction survey.
In 2016, more than 4400 general hospitals partic-ipated. (Pediatric, psychiatric, and specialty hospitals were excluded.) More than 99% of the hospitals used an approved survey vendor who contacted patients by mail (60%) or telephone (40%). BOX 8-11 shows an abstract of the questions used in the HCAHPS in English. It is also available in Spanish, Russian, Chi-nese, Vietnamese, and Portuguese, as of this writing.
In 2016, more than 4400 general hospitals partic-ipated. (Pediatric, psychiatric, and specialty hospitals were excluded.) More than 99% of the hospitals used an approved survey vendor who contacted patients by mail (60%) or telephone (40%).
outcome evaluation; the starting point for deciding to do an evaluation is the Neds of the program managers and stakholders. the evaluation demonstrates the effectiveness of the intervention.
need to document impact by measuring whether pgroam objectives have been met, barriers have been reduced, favorable attitudes, and perceptions have come about in the target audience, and particular desired behaviors have increased
description; might want to use both quantitative methods ( counting, measuring) and qualitative methods (interviewing participants about their experience) to answer these questions
causality;
- did the change take place after the health communication program began (that is, the change cannot be attributed easily to something contemporaneous
- can you detect a degree of response against a “dose” of the itneventoin ?
- can you control for confounding variables
- was there a comparison group
- what was the likelihood that those person more likely to adopt the behavior were those who participated in the first place
it can be extremely difficult to attribute change to a communcaiton intervention in community settings . (change of exposure is limited)
people share information all the time , so maintaing clear boundaries on where an intervention takes place is nearly impossible
- entities might be communicating about the same thing, without you being aware of it
judgement ; are the outcomes good enough? most often, if a measurable outcome can be quantified, the next question will pertain to the costs of achieving it. this is called a cost benefit analysis
indicators for measuring change; assuming a baseline survey or some other data collection has taken place (again, the time to think about evaluation is not at the end of the project), the theoretical structure and logic model you have designed for the intervention will lead to electing an indicator
managing an evaluation; manage, define, fram, describe, understand causes, synthesize, report, and support use
getting the word out; when you get to the end of the evaluation, you need to write up results and share the with stakeholders or the public . the point of doing an evaluation is to use the results
