Module 04 Flashcards
Chapter 12 and 13
1
Q
What is a process evaluation?
A
A process evaluation examines how a program operates and delivers its services to clients.
2
Q
When should process evaluations occur?
A
Process evaluations should occur before or at the same time as outcome evaluations.
3
Q
What are the two major categories of program processes?
A
- Client service delivery system
- Administrative support systems
4
Q
What is the primary aim of a process evaluation?
A
To monitor a program’s services and assess client satisfaction.
5
Q
True or False: Process evaluations focus solely on program outcomes.
A
False
6
Q
What are the three main purposes of conducting a process evaluation?
A
- Improving a program’s operations
- Generating knowledge
- Estimating cost-efficiency
7
Q
How can process evaluations improve program operations?
A
By fine-tuning the services delivered to clients and ensuring administrative support is effective.
8
Q
What does generating knowledge through process evaluations typically inform?
A
Decisions about the further development of the program’s services.
9
Q
What are the eight questions that can focus a process evaluation?
A
- What is the program’s background?
- What is the program’s client profile?
- What is the program’s staff profile?
- What is the amount of service provided to clients?
- What are the program’s interventions and activities?
- What administrative supports are in place?
- How satisfied are the program’s stakeholders?
- How efficient is the program?
10
Q
Fill in the blank: Process evaluations are sometimes referred to as ______.
A
[formative evaluations]
11
Q
What is a common challenge in describing program processes?
A
Achieving a level of precision in using consistent language across different disciplines.
12
Q
What does monitoring interventions and activities imply?
A
That there are labels and definitions for what is done with clients, which increases communication.
13
Q
What is meant by the term ‘black box’ in relation to client service delivery?
A
It reflects the notion that clients enter and exit a program with no clear understanding of the processes involved.
14
Q
What is the significance of a program’s client profile?
A
It informs how the processes within the program are operationalized and monitored.
15
Q
What type of data is important to gather in a client profile?
A
- Age
- Gender
- Income
- Education
- Race
- Socioeconomic status
16
Q
What can the program’s history reveal?
A
Critical insights into its day-to-day operations and the political and social context it operates within.
17
Q
What role does stakeholder satisfaction play in a process evaluation?
A
It helps assess the effectiveness and reception of the program’s services.
18
Q
What is the significance of worker qualifications in a program?
A
Monitoring worker qualifications helps establish minimum-level qualifications for job advertisements and provides insight into the effectiveness of service delivery.
19
Q
How do educational backgrounds influence program approaches?
A
Different educational backgrounds can lead to varying philosophical approaches in programs addressing the same social issue, such as teenage pregnancy.
20
Q
What sociodemographic data is typically used to describe workers?
A
Age, gender, and marital status are commonly used sociodemographic features.
21
Q
In what ways can worker time be categorized?
A
Worker time can be divided into face-to-face contact, telephone contact, report writing, advocacy, supervision, and consultation.
22
Q
What is the importance of tracking service intensity?
A
Tracking service intensity helps determine how much time is spent on various client interactions and can guide program decisions.
23
Q
True or False: Programs always have clear-cut intake and termination dates.
A
False
24
Q
What is a process evaluation?
A
A process evaluation assesses whether actual services delivered match the original design and logic model of a program.
25
What are administrative supports in a program?
Administrative supports include fixed conditions of employment and operations designed to assist workers in delivering client services.
26
What factors can affect worker effectiveness in a program?
Worker pay, caseloads, availability of support staff, and working conditions can significantly impact effectiveness.
27
Fill in the blank: The primary objective of a literacy program visit is to _______.
[increase literacy skills of children]
28
What are the two main objectives of the Rural Family Literacy Program?
* To increase literacy skills of children
* To increase parents' abilities to assist their children in developing literacy skills
29
What is the role of stakeholder satisfaction in process evaluation?
Stakeholder satisfaction assesses how well the program meets the needs and expectations of those involved.
30
What can be inferred from a program's efficiency?
Efficiency reflects the amount of resources expended versus the achievement of program objectives.
31
How can data from home visits influence program decisions?
Data can provide insights into the effectiveness of interventions and help adjust strategies to enhance client engagement.
32
What is one example of how administrative decisions can change service delivery?
Switching from group care settings to providing interventions in less intrusive environments, like clients' homes.
33
True or False: Individual client success rates should be used to evaluate social workers.
False
34
What data collection method was used by workers in the Rural Family Literacy Program?
Workers completed a Daily Family Visit Log to monitor fidelity and effectiveness of their visits.
35
What impacts can high caseloads have on social workers?
High caseloads can lead to decreased effectiveness in responding to clients' needs.
36
What key aspect must be monitored to assess program fidelity?
The actual services delivered must align with the original program design and objectives.
37
What is the purpose of stakeholder satisfaction in program evaluation?
Stakeholder satisfaction assesses the programs services from the stakeholders' perspective.
## Footnote
It is crucial for understanding how well the program meets the needs of its clients.
38
What is a common method for collecting client satisfaction data?
Client satisfaction surveys conducted at program termination.
## Footnote
These surveys can provide valuable insights into client experiences and perceptions.
39
True or False: Client satisfaction data is relevant to outcome evaluations.
False.
## Footnote
Client satisfaction is specifically relevant to process evaluations.
40
What are the four issues to address when developing data-collection instruments for process evaluation?
* Easy to use
* Flow with a program's operations
* Designed with user input
* Comprehensive data-collection system
## Footnote
These criteria ensure the effectiveness and efficiency of data collection.
41
Fill in the blank: Data-collection instruments for process evaluation should be _______.
easy to use.
42
Why is it important for data-collection instruments to flow with a program's operations?
They need to fit within the context of the program and provide useful data for improving client service delivery.
## Footnote
This alignment ensures that data collection does not disrupt the program's workflow.
43
What should be included in a client intake form?
* Client characteristics
* Reasons for referral
* Service history
## Footnote
This information is essential for both case-level and program-level evaluations.
44
What is the role of user input in developing data-collection instruments?
User input, especially from line-level workers, ensures the relevance and accuracy of the data collected.
## Footnote
Workers are more likely to record accurate data if they see the relevance of the instruments.
45
What are the three considerations for developing a data-collection system for process evaluation?
* Determining the number of cases to include
* Determining times to collect data
* Selecting a data-collection method
## Footnote
These considerations help in structuring the evaluation process effectively.
46
How can a program ensure random sampling for client satisfaction surveys?
By randomly selecting clients to participate as close to the intake meeting as possible.
## Footnote
This method helps in generalizing results to all clients within the program.
47
What is a potential issue with lengthy client intake forms?
They may become overly detailed and burdensome for both clients and workers.
## Footnote
Short and long forms can be developed to balance the depth of data collected.
48
What is meant by 'case-level interventions' in the context of data collection?
Specific actions taken by workers to address individual client needs.
## Footnote
These interventions can be tracked to assess client progress.
49
What are some examples of data that might be collected at different times during the evaluation process?
* Basic client demographics at intake
* Sensitive information after rapport is established
## Footnote
Collecting data at appropriate times can improve the quality of the information gathered.
50
Why is it important to monitor process data for reliability and validity?
To ensure that the data collected accurately reflects the program's operations and client experiences.
## Footnote
Poor data quality can lead to incorrect conclusions about program effectiveness.
51
Fill in the blank: Data collection should balance _______ and _______.
breadth; depth.
52
What type of data should be collected from all clients for case-level evaluations?
Intake forms and assessment data.
## Footnote
These forms are critical for planning client treatment interventions.
53
What is the primary purpose of recording workers' activities?
To document and evaluate worker-client interactions and interventions
## Footnote
This process is often time-consuming and may raise questions about the reliability of self-reported data.
54
What are some fixed client characteristics that can be recorded?
* Race
* Gender
* Service history
* Problem history
55
Why might the reliability of data collected by workers be questioned?
Because the data is often self-reported by the workers themselves.
56
What is a recommended practice for supervisors regarding worker-client interactions?
Supervisors should observe worker-client interactions to assess the reliability of self-reports.
57
What is interrater reliability?
The extent of agreement between the workers' perceptions and the supervisors' perceptions of worker-client interactions.
58
What issue can arise when workers administer client satisfaction questionnaires?
Social desirability bias, leading to less honest ratings from clients.
59
What should be done to ensure client responses to satisfaction questions are confidential?
Clients should be informed that their responses are confidential and will be pooled with others’ responses.
60
What is the importance of summarizing process evaluation data?
To ensure that the conclusions drawn from the evaluation are directly based on the gathered data.
61
What can happen if data is collected inconsistently?
It will be difficult to summarize and may produce inaccurate information.
62
What is one potential outcome of a backlog in data summarization?
It may indicate that too much data is being collected.
63
How does the timing of data recording impact the evaluation process?
Initial diligence may wane, leading to incomplete data that misrepresents the program's effectiveness.
64
What key questions can be asked regarding outcome data collected?
* Is the time spent with a family related to success?
* What interventions are associated with successful outcomes?
65
Why is it important to communicate findings from process evaluations?
To ensure results are used for program improvement and to foster discussions among workers.
66
What is a program audit sheet?
A tool that lists all necessary data to be recorded for each client.
67
What should be included in program development meetings following a process evaluation?
Data summaries and discussions on individual worker feedback.
68
What is the black box in the context of program evaluations?
The unknown aspects of how workers function on a day-to-day basis.
69
Fill in the blank: The final step in a process evaluation is the _______.
[dissemination and communication of findings]
70
What components should a process evaluation consider?
* Preparation
* Intake
* Screening
* Assessment
* Termination
* Follow-up
71
What type of data can process evaluations provide regarding interventions?
Clues about which interventions work with specific client problems.
72
True or False: Process evaluations only focus on client outcomes.
False. They also focus on the inner workings of a program.
73
What is the role of program administrators in process evaluations?
To review client records and ensure necessary data is captured.
74
What should be done if clients are hesitant to provide honest feedback?
Use a neutral person to administer satisfaction questionnaires.
75
What is the significance of using visual data in evaluations?
It provides a forum for discussion and helps illustrate program effectiveness.
76
What should be tracked to assess client participation in counseling sessions?
Attendance and quality of participation.
77
What documents are typically involved in the intake process?
* Screening Form
* Intake Form
* Social History
* Data Entry Forms
78
What is the purpose of the Individual Rehabilitation Plan (IRP)?
To assist clients in developing a plan to meet individual and program objectives.
79
What is the purpose of process evaluations?
To improve services to clients
## Footnote
Process evaluations focus on collecting data to make informed decisions regarding program operations.
80
What must program staff decide when designing a process evaluation?
They must decide on the following:
* Questions to ask
* Data collection methods
* Responsibilities for monitoring data
* Data analysis methods
* Dissemination of results
## Footnote
These decisions are crucial for effective evaluation of program processes.
81
What does IRP stand for?
Individual Rehabilitation Plan
## Footnote
An IRP outlines the goals and objectives for an individual in a rehabilitation program.
82
What is the first step for a client seeking job placement?
Client meets with a job placement counselor
## Footnote
This meeting is to identify available job slots that fit with the client's training.
83
What document is used for job placement referrals?
Job Placement Referral Form
## Footnote
This form is essential for tracking job placement efforts.
84
What is the objective of the IRP that clients must achieve?
Clients must demonstrate readiness to function independently in the community
## Footnote
This involves planning for termination from programs like Safe Haven.
85
What is involved in the follow-up process after a client achieves their IRP objective?
Case manager makes telephone contacts at agreed-on times
## Footnote
Follow-up is critical for ensuring continued support and assessment.
86
What happens during the exit phase for a client?
Follow-up contacts end by mutual agreement
## Footnote
This indicates a formal closure of the case.
87
What forms are used during the client termination process?
The following forms are used:
* Victims to Victors Termination Form
* Safe Haven Termination Form
* Data Entry Form
## Footnote
These forms help document the termination process and client outcomes.
88
Fill in the blank: Data is collected on many program dimensions to make _______ decisions about a program's operations.
informed
## Footnote
Informed decisions are based on accurate and relevant data collected through evaluations.
89
What is the main purpose of outcome evaluations?
To demonstrate the nature of client change after receiving services.
## Footnote
Outcome evaluations help determine if clients show improvement due to program interventions.
90
What must we have a clear sense of when conducting an outcome evaluation?
What expected changes (the program's outcomes) we hope to see.
## Footnote
Clear objectives are crucial for assessing the effectiveness of a program.
91
How many major steps are involved in conducting an outcome evaluation?
Six major steps.
## Footnote
These steps are illustrated in the referenced figure in Chapter 3.
92
What are the five purposes of conducting outcome evaluations?
* Demonstrate the nature of client change
* Provide feedback to stakeholders
* Select the best evidence-based interventions
* Provide accountability
* Generate knowledge for the profession
## Footnote
These purposes guide the evaluation process and enhance program effectiveness.
93
True or False: An outcome evaluation can indicate how well a program is working.
False.
## Footnote
An outcome evaluation indicates if program objectives are met but does not explain how or why.
94
What does an outcome evaluation primarily evaluate?
Whether the program is meeting its program objectives.
## Footnote
This assessment focuses on specific outcomes related to client change.
95
Fill in the blank: A program outcome evaluation is designed for a __________ program.
[specific]
## Footnote
Each evaluation is tailored to the unique context of the program being assessed.
96
What is a critical aspect of an outcome evaluation regarding client objectives?
To assess only one small component of a complex social problem.
## Footnote
This limitation emphasizes the complexity of social issues addressed by programs.
97
What is an example of a question that an outcome evaluation can answer regarding client outcomes?
Was the client outcome achieved?
## Footnote
This question assesses the effectiveness of the program in achieving its goals.
98
What type of data is collected to determine if the program caused changes in clients?
Data collected through more complex evaluation designs.
## Footnote
Such designs help establish causality between program interventions and client outcomes.
99
What can complicate the follow-up data collection for outcome evaluations?
Clients receiving services from other programs during the follow-up period.
## Footnote
This overlap can obscure the evaluation of the original program's effectiveness.
100
What aspect of client change might be assessed in an outcome evaluation?
The longevity of changes made by clients.
## Footnote
This assessment determines if positive changes are maintained over time.
101
What is a potential limitation of evaluating a single program objective at a time?
Limited knowledge gained from evaluating only one component.
## Footnote
Cumulative evaluations over time provide more confidence in the results.
102
What should be done if an evaluation of a program's objectives turns out to be poor?
Investigate why this is so by doing a process evaluation.
## Footnote
Process evaluations can help understand the underlying issues affecting program outcomes.
103
What does the term 'evidence-based interventions' refer to?
Interventions selected based on their proven effectiveness in creating positive client change.
## Footnote
These interventions are crucial for achieving program objectives.
104
True or False: Outcome evaluations provide a complete picture of a program's efficiency.
False.
## Footnote
Outcome evaluations focus on effectiveness, not efficiency.
105
What is the first step in conducting an outcome evaluation?
Engage Stakeholders
## Footnote
Engaging stakeholders involves involving all relevant parties in the evaluation process.
106
Why is it critical to clearly specify a program's objectives?
It defines how we understand our overall program in concrete terms
## Footnote
Clear objectives lead to better program evaluation and effectiveness.
107
What is the second step in conducting an outcome evaluation?
Describe the Program
## Footnote
This includes using theory of change and logic models.
108
What should follow-up data collection intervals ideally be?
3, 6, or 12 months after clients exit a program
## Footnote
This timing allows for assessing the sustainability of program effects.
109
True or False: Performance objectives and client outcome objectives are the same.
False
## Footnote
Performance objectives focus on outputs, while client outcome objectives focus on actual client changes.
110
What are the two steps involved in focusing the evaluation?
Selecting program objectives and measuring program objectives
## Footnote
Both steps are key to ensure the evaluation is targeted and meaningful.
111
Fill in the blank: The difficulty in measuring changes in a client's self-esteem may lead programs to focus on _______.
performance objectives
## Footnote
This can misguide the evaluation process.
112
What are the three levels of outcomes identified in the evaluation process?
* Initial Outcomes
* Intermediate Outcomes
* Long-Term Outcomes
## Footnote
Each level helps to categorize the objectives for better assessment.
113
What is the third step in conducting an outcome evaluation?
Focus the Evaluation
## Footnote
This requires careful selection of objectives and measurement criteria.
114
What does gathering credible evidence for an outcome evaluation include?
* Selecting samples or data sources
* Pilot-testing measuring instruments
* Administering measuring instruments
## Footnote
These steps ensure the reliability and validity of the evaluation.
115
How can we measure program objectives effectively?
Using standardized measuring instruments with high validity and reliability
## Footnote
This ensures accurate assessment of outcomes.
116
What is a potential challenge in collecting follow-up data?
Difficulty in locating clients after they leave a program
## Footnote
This can be particularly challenging with transient or underserved populations.
117
What is the significance of using a logic model in program evaluation?
It helps clearly display program objectives and their relationships
## Footnote
Logic models guide the evaluation process by providing a visual representation of program components.
118
What is an example of an initial outcome indicator for outpatient mental health services?
Number of consumers who received outpatient services during the quarter
## Footnote
This measures initial awareness and access to services.
119
What is the importance of client follow-up data?
It assesses the longevity of changes made by clients
## Footnote
Follow-up data helps determine if the program effects are sustained over time.
120
What is the purpose of assessing consumers every 6 months?
To evaluate improvements in well-being/life satisfaction
121
What is Indicator 10?
Number and percentage of consumers who report an increase in well-being (life satisfaction)
122
What is the ideal sample size for each subgroup in an outcome evaluation?
At least thirty clients
123
What is the main issue affecting sample size in outcome evaluations?
Whether program resources exist to collect data from all clients
124
What is the goal of random sampling in program evaluations?
Each client has an equal chance of being included in the study
125
True or False: Social workers should evaluate their own performance in outcome evaluations.
False
126
What is a critical aspect of random selection in outcome evaluations?
The decision to include clients is made without bias
127
Fill in the blank: Quality data collection requires several explicit ______ that need to be laid out and strictly followed.
[procedures]
128
What should be done when clients decline to participate in an evaluation?
Explore the reasons for their refusal
129
What is the purpose of pilot-testing measuring instruments?
To ascertain whether the instrument produces the desired data
130
What should be done if a self-report measuring instrument is used?
Check the accuracy of the data using multiple data sources
131
What is the first step in administering measuring instruments?
Decide which questions the outcome evaluation will answer
132
What is the significance of aggregating data in outcome evaluations?
To provide an overview on client outcomes
133
What must conclusions drawn from outcome evaluations directly come from?
The data gathered during the evaluation
134
What are normative data useful for in outcome evaluations?
Interpreting client data when measurement occurs at program exit
135
What should be included in reports of outcome data for stakeholders?
Concrete and objective results
136
Fill in the blank: When analyzing data in subgroups, we can gain additional ______ for program decision-makers.
[information]
137
What is the primary purpose of outcome evaluations?
To determine whether client changes have occurred as a result of intervention efforts.
## Footnote
Outcome evaluations provide valid and reliable data for decision-making.
138
What percentage of families with toddlers showed improvement in problem-solving skills according to the analysis?
Seventy-five percent.
## Footnote
This contrasts with families that have teens, where almost no improvement was observed.
139
What should be measured to evaluate program outcomes realistically?
The amount of average improvement and the number of clients expected to show success.
## Footnote
This helps educate stakeholders about client populations.
140
True or False: 100% success in deterring social issues like drug addiction is a realistic expectation for any program.
False.
## Footnote
In some cases, a 50/50 chance of improvement is expected.
141
What is one method to report outcome data over time?
Presenting client outcomes from one year to the next to show program trends.
## Footnote
This method helps stakeholders understand program effectiveness.
142
What types of stakeholders should receive outcome results?
Key stakeholders, including funders and policymakers.
## Footnote
Routine sharing of outcome data is essential for informed decision-making.
143
Fill in the blank: The likelihood of having evaluation results used is increased when results are presented in a _______.
straightforward manner.
144
What is a common obstacle to putting evaluation results into practice?
Failing to remember the law of parsimony when presenting the final report.
## Footnote
Reports should be clear and concise for the intended audience.
145
What happens when evaluation results contradict strong predetermined beliefs?
There may be resistance to using the findings.
## Footnote
For example, social workers may believe their efforts are always helpful.
146
What is the role of confidentiality in outcome evaluations?
To protect client identities by reporting data in aggregate forms.
## Footnote
Summarizing data helps avoid singling out any one client.
147
What does program outcome assessment evaluate?
The degree to which the program is meeting its overall objectives.
## Footnote
This usually means assessing the effectiveness of interventions.
148
When are outcome evaluations typically conducted?
Before or at the same time as efficiency evaluations.
## Footnote
Efficiency evaluations focus on the program's operational effectiveness.
149
What is the significance of analyzing data in subgroups during evaluations?
It provides important detail for program decision-makers.
## Footnote
This analysis helps identify program strengths and weaknesses.
150
From chapter 12 slide, Resources are known as
Inputs
151
From chapter 12 slide, Activities are known as
What the program does
152
From chapter 12 slide, Outputs are known as
The services that are delivered
153
From chapter 12 slide, Outcomes are
For clients
154
From chapter 12 slide, Impacts are
For the community or society
155
From chapter 12 slide, Process Evaluations are focused on what?
Examining the activities of a program
156
From chapter 12 slide, What is identified during a needs assessment?
A needs assessment identifies the need for a program.
157
From chapter 12 slide, What is the focus of a formative evaluation?
The initial program and contains information to help “form” and stabilize
the program
158
From chapter 12 slide, What elements are included in process evaluation?
Program description
Program monitoring
Quality assurance
159
From chapter 12 slide, What is the purpose of outcome evaluation?
The purpose of outcome evaluation is to determine if the program works and if it impacts the target population’s problem as identified in the needs assessment.
160
From chapter 12 slide, What is the primary purpose of a formative evaluation?
A formative evaluation is used to guide and direct programs, assess whether a new program was implemented as planned, and adjust and enhance interventions.
161
From chapter 12 slide, When is process evaluation commonly employed?
Process evaluation is commonly employed by new programs.
162
From chapter 12 slide, What does process evaluation assess?
Process evaluation assesses whether a new program was implemented as planned and what was learned during program implementation.
163
From chapter 12 slide, At what stage of a program can process evaluation be employed?
Process evaluation can be employed at any time during a program’s developmental stage.
164
From chapter 12 slide, What can process evaluation help determine regarding program failure?
Process evaluation helps determine whether the failure of the program was due to a poor program design or poor implementation.
165
From chapter 12 slide, What elements can be included in a process evaluation?
Process evaluation can include program description, program monitoring, and quality assurance.
166
From chapter 12 slide, What can process evaluations help fine-tune?
Process evaluations can fine-tune the service delivery process.
167
From chapter 12 slide, How can process evaluations help identify interventions?
Process evaluations help identify which interventions work best for whom.
168
From chapter 12 slide, What kind of profiles can process evaluations provide?
Process evaluations provide a clear client and staff profile.
169
From chapter 12 slide, What aspect of a program does process evaluation assess regarding its implementation?
Process evaluation assesses program fidelity, ensuring the program is being implemented as intended.
170
From chapter 12 slide, What is the purpose of program description in process evaluation?
Program description documents the operations of a program and provides necessary data to judge the intensity and reliability with which services are delivered.
171
From chapter 12 slide, How is program monitoring important in process evaluation?
Program monitoring helps understand what happened in a program and to whom, ensuring that the program is serving those for whom it was designed, and it tracks progress toward meeting expectations.
172
From chapter 12 slide, What types of data does program monitoring rely on?
Program monitoring relies heavily on data captured by agencies, including face-to-face and telephone interviews, surveys, key informant interviews, focus groups, organization record analysis, program documentation analysis, observations, and case studies.
173
From chapter 12 slide, What is the purpose of quality assurance in process evaluation?
Quality assurance evaluates compliance with a set of standards, often focusing on the process of treatment, identifying and correcting deficiencies, and ensuring adherence to guidelines and accountability.
174
From chapter 12 slide, How does quality assurance differ from program evaluation?
Quality assurance focuses on the process of treatment, rather than outcomes, and is often driven by legislative mandates to promote consistency and treatment fidelity.
175
From chapter 12 slide, What is Step 1 in the process evaluation process?
Step 1 is deciding what questions to ask.
176
From chapter 12 slide, What is Step 2 in the process evaluation process?
Step 2 is developing data collection instruments.
177
From chapter 12 slide, What is Step 3 in the process evaluation process?
Step 3 is developing a data collection monitoring system.
178
From chapter 12 slide, What is Step 4 in the process evaluation process?
Step 4 is scoring and analyzing data.
179
From chapter 12 slide, What is Step 5 in the process evaluation process?
Step 5 is developing a feedback system.
180
From chapter 12 slide,
What is Step 6 in the process evaluation process?
Step 6 is reporting and using the findings, which involves summarizing the results, interpreting the data, and sharing the findings with stakeholders to inform decisions and improvements for the program.
181
From chapter 12 slide,
What are some key questions to ask when deciding what to evaluate in process evaluation?
Key questions include:
What is the program background?
What is the client background?
What is the staff profile?
How much service is provided to the client?
What are the program interventions and activities?
What administrative supports are in place?
What is stakeholder satisfaction?
182
From chapter 12 slide, How do process evaluation questions differ from needs assessment and outcome questions?
Process evaluation questions focus on program implementation (e.g., service delivery, client background, staff profile), while needs assessment questions identify the need for a program, and outcome questions assess the effectiveness of the program in addressing the identified problem.
183
From chapter 12 slide, Provide an example of a process-oriented question you could ask about your program.
An example of a process-oriented question could be: "How effectively are the interventions being delivered to the clients, and are they receiving the intended amount of service?"
184
From chapter 12 slide, What should be considered when monitoring stakeholder participation in process evaluation?
Consider who is being monitored and how, particularly accounting for barriers to participation, especially for vulnerable or underprivileged groups.
185
From chapter 12 slide, How are data collection procedures developed in process evaluation?
Data collection procedures are developed with input from key stakeholders, considering who has a say in what data is collected and how.
186
From chapter 12 slide, What is the concept of saturation in data collection?
Saturation refers to determining how many people need to be engaged to answer evaluation questions, often achieved through periodic analysis during data collection.
187
From chapter 12 slide, What is data fatigue, and why should it be considered in process evaluation?
Data fatigue occurs when participants are overtaxed with data collection or monitoring, which can lead to reduced participation or accuracy in responses.
188
From chapter 12 slide, What should be considered when developing data collection instruments and monitoring systems?
Factors to consider include ensuring consistency in data collection, addressing sampling concerns, and maintaining reflexivity, or awareness of implicit biases that could affect the analysis.
189
From chapter 12 slide, What is reflexivity in the context of analyzing data?
Reflexivity refers to being aware of implicit biases during data analysis and developing strategies to mitigate those biases.
190
From chapter 12 slide, What factors should be considered when disseminating and communicating results?
Factors to consider include:
How to share the results
When to share the results
Where to share the results
To whom the results should be shared
191
From chapter 12 slide, Why is it important to share process data?
Process data is only helpful if shared, as it enables stakeholders to understand and act on the findings.
192
From Chapter 13 slide, What is the focus of outcome evaluations?
Outcome evaluations are focused on the outcomes, specifically what is expected to be seen as a result of the program's activities.
193
From Chapter 13 slide, What is one purpose of outcome evaluation?
One purpose of outcome evaluation is to improve program services to clients.
194
From Chapter 13 slide, How does outcome evaluation assist in decision-making?
Outcome evaluation provides feedback for decision-making, helping to guide future program actions and adjustments.
195
From Chapter 13 slide, What knowledge does outcome evaluation generate?
Outcome evaluation generates knowledge for the profession, contributing to broader insights and practices.
196
From Chapter 13 slide, What is Step 1 in outcome evaluation?
Step 1 is operationalizing program objectives.
197
From Chapter 13 slide, What is Step 2 in outcome evaluation?
Step 2 is operationalizing variables and stating the outcomes.
198
From Chapter 13 slide, What is Step 3 in outcome evaluation?
Step 3 is designing a monitoring system.
199
From Chapter 13 slide, What is Step 4 in outcome evaluation?
Step 4 is analyzing and displaying data.
200
From Chapter 13 slide, What is Step 5 in outcome evaluation?
Step 5 is developing a feedback system.
201
From Chapter 13 slide, What is Step 6 in outcome evaluation?
Step 6 is disseminating and communicating results.
202
From Chapter 13 slide, What is a theoretical theory in the context of program evaluation?
A theoretical theory is a reasoned set of propositions, derived from and supported by established data, which serves to explain a group of phenomena.
203
From Chapter 13 slide, What is a conceptual theory of change?
A conceptual theory of change is a representation of how you believe change will occur within your program.
204
From Chapter 13 slide, What is an operational logic model?
An operational logic model is a systematic and visual way to present the perceived relationships among the resources available, the activities planned, and the changes or results hoped to be achieved.
205
From Chapter 13 slide, How is evaluation ideally linked to practice?
Evaluation is ideally wrapped up with practice, ensuring that the evaluation process informs and improves program implementation.
206
From Chapter 13 slide, What is operationalization in the context of program evaluation?
Operationalization is the explicit specification of a program’s objectives in a way that makes measurement of each objective possible.
207
From Chapter 13 slide, What is an operational definition?
An operational definition is a clear and specific description of an objective that allows for measurable assessment.
208
From Chapter 13 slide, What is an indicator in program evaluation?
An indicator is a specific, observable, and measurable characteristic or change that shows the progress a program is making towards a specified outcome.
209
From Chapter 13 slide, Why is sampling often necessary in program evaluation?
Sampling is often necessary due to the size of the population or participants' choice, as ideally, everyone would be included.
210
From Chapter 13 slide, How does the timing of data collection affect the questions that can be answered?
The timing of data collection affects the types of questions that can be answered:
Pre and post-program data are needed for questions on change.
Follow-up data is necessary for questions on sustained outcomes.
Longitudinal data is required for questions on causation
211
From Chapter 13 slide, What are some factors to consider when deciding when data will be collected?
Factors to consider include:
What method of data collection will be used (e.g., phone, in person)?
Who will collect the data and whether they are impartial.
212
From Chapter 13 slide, What is important when deciding how data will be collected?
It is important to determine the method of collection (e.g., phone, in person), who will collect the data, and whether the person collecting the data is impartial.
213
From Chapter 13 slide, How does the timing of data collection affect the types of questions that can be answered?
The timing of data collection affects the types of questions that can be answered:
Questions on change require pre and post-program data.
Questions on sustained outcomes require follow-up data.
Questions on causation require longitudinal data.
214
From Chapter 13 slide, What is an example of a question on change?
"To what extent do participants in the Life Remodeled Youth Immersion Program learn about race relations in Phoenix during their participation in the program?" (Requires pre and post-program data)
215
From Chapter 13 slide, What is an example of a question on sustained outcomes?
"To what extent do Atlantic Impact participants use the information learned in high school programming during their first year of college?" (Requires follow-up data)
216
From Chapter 13 slide, What is an example of a question on causation?
"How does the experience of participating in Atlantic Impact inform students’ matriculation through higher education?" (Requires longitudinal data)
217
What are the differences between process and outcome evaluations?
Process evaluations assess how services are delivered and whether they meet expectations, while outcome evaluations measure the effectiveness of the program in achieving long-term changes or improvements in clients' conditions.
218
What are the differences between process and outcome evaluations?
Process evaluations assess how services are delivered and whether they meet expectations, while outcome evaluations measure the effectiveness of the program in achieving long-term changes or improvements in clients' conditions.
219
220
How do process and outcome evaluations relate to a logic model?
In a logic model, process evaluations correspond to the activities and outputs columns, focusing on service delivery, while outcome evaluations relate to the outcomes and impacts columns, focusing on client changes and program impact.
