Program Evaluation Flashcards
1
Q
Program
A
- More of less coherent system of activities that have been organized to address specific needs and resolve particular problems of a given population
- A bunch of activities that are interrelated so we put them together because they make sense to be together, they’ve been organized to meet specific needs
2
Q
Program Evaluation
A
- It’s in many fields, but it’s especially important in community psych
- Is the program doing what it says it’s doing?
- We need to make sure programs are doing the same thing as others and we need to make sure they are productive
- We don’t want to have programs that are wasting us money or have redundant movies
- Kind of like research but like not really
3
Q
Components of a program and of a program logic model
A
- Program mission
- Program processes
- Program outcomes
4
Q
Program processes - implementation/program evaluations
A
- Inputs: what you need, staff/supplies, how much money…
- Activities: What are you actually doing/outreach, intake, actual activities, program exits, etc.
- Outputs: Numbers that prove we’re doing something, how many people are coming or learning, how are the following the curriculum
- This step answers what our program is actually doing
4
Q
Theory of change
A
- A statement about how a program’s activities are intended to produce particular outcomes
- How do you think you will make a difference? Or how do think the program will make a difference
4
Q
Program logic models
A
- A depiction of the relationships among program resources, program activities, outputs, and outcomes
- Can be in a table/text/graphic forms
- Important for understanding a program’s theory of change and the program’s feasibility
- How does this actually run/ how is it running or organized/ how do all the activities work together and do all these things make sense
- Theory of change
4
Q
Program mission/needs assessment
A
- What is the program claiming to due
- Ex: Ending hunger in Ottawa
- What are the needs that drive the program
- What values and principles drives the way we go about things
- Ex: homeless people may not like many systems so you have to find ways to get them off the streets and into the shelters
4
Q
Program outcomes - Impact/Outcome evaluations
A
- Initial outcomes
- Intermediate outcomes
- Long term outcomes
4
Q
Possible constraints to programs
A
- Could be things like taxes, or finding good people to work, etc.
5
Q
Main types of program evaluation
A
- Needs assessment
- Process (implementation) evaluation
- Outcome (impact) evaluation
5
Q
Potential users of program evaluation findings
A
- Program clients
- Program staff
- Program management
- Program funders
- Policy makers and decision makers
- We don’t just want to grade the program we want to give them helpful feedback
5
Q
What is a needs assessment
A
- Assessment of scope and nature of problem
- Evaluation of services in place to address problem
- Examination of service alternatives
- Development of plan for new program
- Usually do this when starting a program but can also be done with existing programs
- Ex: homelessness in Ottawa has changed a lot in the last 20 years, but the people running the programs haven’t, so we do these evaluations to almost revamp the program
6
Q
Needs assessment sample questions
A
- What is the prevalence of persons who are homeless in Ottawa?
- What services are available in Ottawa for persons who are homeless?
- How well are current services serving persons who are homeless?
7
Q
Needs assessments methods primary data sources
A
- Primary data sources (you collect it as the evaluator)
- Key informant surveys
- Focus groups
- Community forums
- Community surveys
8
Q
Needs assessment methods secondary data sources
A
- Secondary data source (data that already exists from someone else)
- Social indicators
- Service utilization data
- Published studies
9
Q
Process or implementation evaluation
A
- How is the program actually running and what is it actually doing
- If the activities are being delivered to the intended recipients in the intended amount at the intended level of quality
- What we want to do comes from the program logic model
10
Q
Why must we do a program logic evaluation before a process evaluation
A
- We don’t know why people are benefiting from the program
- We don’t know if it is poorly implemented theory
11
Q
Types of program failure
A
- Implementation failure: Can be corrected if we improve how we operate
- Theory failure: Your theory/assumptions is just wrong. No evidence saying what you want to do is going to change
12
Q
Key questions in a process/implementation evaluation
A
- Is the program reaching the intended population?
- Is the program being implemented as planned?
- How or why has be the actual program differed from the intended program?
13
Q
Common methods of implementation evaluation
A
- Chart reviews (secondary)
- Observation of service delivery (primary)
- Staff interviews (primary)
- Client interviews (primary)
- Daily logs of services delivered (secondary)
14
Q
Outcome/Impact evaluation
A
- Assessment of whether interventions have actually produced their intended effects
- Ex: produced planned client benefits
15
Q
How do we get evidence that people have actually benefited from programs
A
- Self reports
- Comparison groups (is it ethical?)
- Gets more complicated and more expensive
16
Q
Outcome evaluations sample questions
A
- Are clients of a community support program actually experiencing the intended improvement in functioning as a result of their participation in the program?
- Is service intensity associated with client outcomes?
17
Q
Rigor and practical considerations in program evaluations
A
- We want it to be the best it can be but a lot of our research methods aren’t going to work for practical, cost, ethical, etc. reasons
- We don’t always have the best evidence, because we don’t have the best design, and that makes us ask more questions
18
Type of designs for an outcome evaluation
- Experimental approach
- Quasi experimental
- Non experimental approach
- A lot of qualitative because that's all we can really do
