PSYC*2360 Chapter 13: Program Evaluation Flashcards
What does a program evaluation involve?
Involves using the scientific method to assess whether an organized activity is achieving its intended objectives
What are grants?
Monetary gifts to help organizations meet a specific goal or objective
T or F: Program evaluations are helpful, but not often essential to the grant writing process.
False. They are often essential.
What are three types of program evaluations?
- A needs evaluation
- A process evaluation
- An outcomes evaluation
Which type of program evaluation involves an assessment of which features of a program are most valuable and who they benefit?
A needs evaluation
What are the two main goals of a needs evaluation?
- Identify features to continue or discontinue
- Determine potential components to add
Which type of program evaluation is particularly helpful for program planning?
A needs evaluation
Which type of program evaluation involves an assessment of a general program operation, including whom the program serves and how it delivers services to that population?
A process evaluation
What is the main goal of a process evaluation?
Determine ways to improve program implementation and delivery
Which type of program evaluation seeks to ensure a match between program goals and those serviced?
A process evaluation
Which type of program evaluation involves an assessment of whether a program effectively produces outcomes that are consistent with stated objectives or goals?
An outcome evaluation
What are the two main goals of an outcome evaluation?
- Identify unmet goals and outcomes that can be improved
- Establish evidence of program effectiveness
What are four potential benefits of program evaluations?
- Forces members to clarify the program’s purpose and intended outcomes
- Provides the organization with evidence of the program’s effectiveness and that the program is using resources efficiently
- Allows the organization to evaluate a new approach or resource
- Encourages organizations to identify areas for improvement
What is the main goal of a program evaluation?
To provide actionable information that can have immediate implications to benefit a specific program
T or F: In a program evaluation, there is often less control over extraneous variables that may bias findings.
True
What is one way to avoid potential bias during a program evaluation?
Use an outside consultant to conduct the evaluation
T or F: A program’s outcomes are often immediately obvious.
False
T or F: Program evaluations seek to help improve a program, therefore no additional attention must be paid to potential consequences.
False
T or F: If a program evaluation is too intrusive, researchers might undermine client’s trust or limit the program’s effectiveness.
True
What are the three phases of a program evaluation?
- The planning phase
- The execution phase
- Communication of results
What are the four steps involved in the planning phase of a program evaluation?
- Identify key stakeholders
- Describe the program
- Clarify the evaluation’s goals
- Create an evaluation plan
What are the two steps involved in the execution phase of a program evaluation?
- Gather data
- Analyze data
What are the three steps involved in the communication of results phase of a program evaluation?
- Form conclusions
- Make recommendations
- Report findings
Which method of data collection involves assembling several individuals to discuss a specific topic?
Focus groups
Ideally, how many participants should be included in a focus group?
6-12
Ideally, should participants in focus groups know each other beforehand?
No
Is the fact that in a focus group, participants have the opportunity to react to each other’s contributions most often considered an advantage or a disadvantage?
An advantage because it provides richer information
T or F: Focus groups provide a high volume of insightful content using participants’ own words in a quick and efficient manner.
True
In focus groups, are facilitators typically allowed to ask follow-up questions to clarify meaning?
Yes
What is the most notable disadvantage of focus groups?
The inability to generalize to other populations
What is the role of a moderator in a focus group?
To ask the questions and facilitate discussion
What are the four characteristics of an ideal moderator?
- Has a thorough understanding of the focus group’s topic
- Has a strong foundation in research methods
- Understands group dynamics (the behaviours and psychological processes of social groups)
- Remains as neutral and bias-free as possible during discussions
In a program evaluation, information from focus groups can be used to develop what additional method of data collection/measurement?
Surveys
T or F: Although focus groups provide quantitative information about the program’s effectiveness, surveys enable qualitative information to be gathered.
False. Focus groups provide QUALITATIVE data and surveys provide QUANTITATIVE data.
What is a single-item indicator?
When only one item or question is being used to measure a variable
What are two concerns when using single-item indicators?
- Poor reliability
- The variable is only being measured in a general sense
What is a single sample t-test used to evaluate?
Whether a sample mean statistically differs from a specific value
T or F: Even though program evaluations aren’t intended to expand generalizable knowledge, IRB approval is still always required.
False. IRB approval is often not required.
Rather than having participants sign an informed consent form, it can be noted at the beginning of a survey that returning it implies consent to be part of a study. This is known as what?
Implied consent
T or F: A program evaluation report should present findings in a clear and succinct way, avoid jargon, and be tailored to the audience.
True
What are word clouds?
Visual representations of the frequency that certain words are used in qualitative assessments
In a word cloud, what do larger words indicate?
Higher frequency of use
What can be used to visually represent a study’s findings that quickly conveys a lot of information in an engaging and easy-to-understand format?
Infographics
The “conclusions and program recommendations” section of a program evaluation report is the equivalent to which section of an APA style report?
The “general discussion” section
What are the three things that a “conclusion and program recommendations” section of a program evaluation report focuses on?
- How the outcome findings match with the program goals
- What the findings indicate about the program’s strengths and weaknesses
- What steps the program should take for improvement
When discussing findings that support a program, stakeholders often exhibit hindsight bias, therefore it is important to remind them of what?
The benefits of having data to support assumption
When discussing findings of a program evaluation that disagree with prevailing opinions, what it is important to remind stakeholders of?
The natural flaws in thinking and the importance of data to support claims
What type of research involves a systematic and planned sequence of related studies, where subsequent studies build directly from a previous study’s findings?
Programmatic research
