Program Evaluation Flashcards
program evaluation
the systematic collection of info about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future program development (CDC)
for ex: policy - to close the gap in college attendance between low SES students and mid/high SES students; possible programs - a scholarship program with specific income eligibility criteria, a free college-readiness program provided directly lower SES districts; subsidies for standardized testing for low SES students; school choice programs - so that students can select a school (in or outside of their district)
theory of action
describes how a project or a program is designed and set up; it articulates the mechanisms through which the activities are being delivered (for ex: through which actors - such as NGOs, govt. or markets - and following which processes - such as grants to NGOs disbursed from a challenge fund, provision of technical assistance, advocacy activities, or the establishment of partnerships); it is important to articulate a Theory of Action at a program’s design stage b/c it affects the Theory of Change and the program’s delivery; a program theory simply does not “exist” w/out a Theory of Action, and its rationale cannot be tested unless the program logic specifies how the “change logic” is going to be implemented
how to measure success
data collection (archival or current), interpret data: you want to see if they actually accomplish what they want to accomplish; head count: how many people attended? testing: pre and post tests to see how many more correct answers attendees could get after participating in your program; data analysis: what % of people who attended your program graduated, increased grades, became employed, etc.; comparison: how many came to the events before your outreach project; how many came after your outreach project
how accountability, expectations and actions lead to a high level of performance (desired effects)
accountability - why we are holding the program evaluation, holding the organization accountable for whatever program they create
why would you want to evaluate a program
to see if it’s effective; it is a way to evaluate the specific projects and activities community groups may take part in, rather than to evaluate an entire organization or comprehensive community initiative; it could be w/ an existing program or the implementation of a new program; you want to 1st determine the criteria you will use to evaluate a program; to engage those who have a stake in the program (stakeholders), you want to describe the existing program protocol - the expectations, activities and current resources; program evaluation is useful to improve how things get done, see if there are any ways to redistribute the workload of the program or to determine other ways of gathering useful data about the overall organization
consider practical limitations of evaluation processes
data is hard to access or doesn’t exist; barriers to change, employees do not want to participate, “its fine as it is” mentality; recruiting participants - using a formal subject pool (people who have already agreed to participate), volunteer subject; standardizing the procedure - easy to introduce extraneous variables (variables you don’t care to look at), all of this add noise, standardizing reduces; record keeping - good record keeping; manipulation check - check to confirm that the IV was successfully manipulated; pilot testing - a small scale study to make sure a new procedure works as planned
what methods can you use to collect evaluation data
archival records, survey, interview, observation; you could retrieve this data not only from the program participants, but through employees of the organization as well, this will help to understand the impact of the implementation of the program - how does it impact their workload? does it make things simpler or more difficult? are there ways to streamline the process?; can collect data from - employees, customers, clients, groups of customers or clients and employees together, program documentation, etc.
why is it important to evaluate a program’s effectiveness
to see if the program is worth the investment of time, money, and resources; goal to improve, change, or get rid of a program/replace
program stakeholders
stakeholders include - those implementing the program (funding agencies, administrators, employees of the program), those targeted by the program (who directly receive/participate in the activities) (individual beneficiaries, representatives of groups such as coalitions or agencies), those otherwise impacted by the program (community members, taxpayers, employees, investors), and those who are “interested” in the work (other evaluators, scholars, policy-makers, law-makers)
outputs
typically the units of work done, for ex. - number of clients served, number of training hours; aka they are the activities of the program completed or the process of the work done
outcomes
the short, medium, or long term goals for the program or the intended effect the program has had on the participants; outcomes/effectiveness measures program effects in the target population by assessing the progress in the outcomes or outcome objectives that the program is to achieve
inputs
the resources needed for the program or organizational components like money, people, equipment/technology, facilities
logic model
provides a way of visually representing the essentials of a program, policy, or even a whole organization, as an open system - categorizes key activities and outlines the intended flow of these activities, linking outputs to outcomes - depicts networks of constructs (ideas/concepts) that can indicate intended cause-and-effect linkages
needs assessment
used to systematically describe and diagnose social needs; define needs, assess the extent of the needs, prioritize the needs, and determine the best solution - provide evidence/justification for programs; part technical, part sociological, and part ethical, it is dangerous to plan a needs assessment without considering all three of these domains; needs can be thought of as gaps in services outcomes, or the state of an individual or group (such as health or education) could mention Maslow’s Hierarchy of Needs; phase 1: pre-assessment - getting organized and establishing what is known; phase 2: assessment - conduct the needs assessment, develop a deeper understanding of the needs of the target population, service deliverers, and organizations responsible, identify likely causes of needs; phase 3: post-assessment - take action to resolve the underlying need
formative
these types of assessments are quizzes and tests that evaluate how someone is learning material through a course; some evaluations are focused on guiding the program towards effectiveness - the evaluation will provide feedback that identifies which goals are being well-met and where there is room for improvement
