Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

SWG 623 Program Evaluation Finals > Module 05 > Flashcards

Module 05 Flashcards

Chapter 15, 16, 17, and 18

1
Q

What are the five types of measuring instruments discussed for evaluation measurement needs?

A
  • Journals and diaries
  • Logs
  • Inventories
  • Checklists
  • Summative instruments

These instruments are practical for various evaluation contexts.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

What is the primary use of journals and diaries in evaluations?

A

Data collection in interpretive studies that collect data in the form of words

They are not typically used in positivistic studies focused on numerical data.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What is a key consideration regarding the reliability of journals?

A

A journal is reliable if the same experience evokes the same written response

However, fatigue or changes in perspective can affect this reliability.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

True or False: Logs are more detailed than journals.

A

False

Logs are structured and generally less detailed than journals.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

What does an inventory typically consist of in evaluations?

A

A list completed by evaluation participants

For example, an inventory designed to measure depression may ask participants to list things that make them feel depressed.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

What is the purpose of a checklist in evaluation?

A

A list prepared by the evaluator to measure specific criteria

For example, a checklist for measuring depression may include feelings experienced over a specific timeframe.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

What is the main function of summative instruments?

A

To obtain data from one question or multiple questions about program objectives and combine responses into a single score

They provide a composite score indicating the individual’s position on the objective being measured.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What are the response categories commonly used in summated scales?

A
  • Strongly agree
  • Agree
  • Neutral
  • Disagree
  • Strongly disagree

This format allows for varied degrees of agreement or disagreement.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

What is the difference between unidimensional and multidimensional summative measuring instruments?

A

Unidimensional measures one variable; multidimensional measures multiple related subvariables

Multidimensional instruments combine several unidimensional ones.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

What are standardized measuring instruments known for?

A

Being extensively tested and providing information on their testing results

They typically include details about purpose, description, norms, scoring, reliability, and validity.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Fill in the blank: A standardized measuring instrument measuring client satisfaction is called the _______.

A

Client Satisfaction Inventory (CSI)

This instrument assesses how clients feel about the services they have received.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

What does the scoring process of the Client Satisfaction Inventory involve?

A
  • Adding the value of valid responses (SUM)
  • Determining the number of valid responses (N)
  • Subtracting N from SUM
  • Multiplying the result by 100
  • Dividing by (N x 6)

This method calculates a score reflecting client satisfaction.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

What has been the impact of the social service agency on the individual?

A

Positive change and feeling of being understood

The individual feels they can talk openly and that the help received is better than expected.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

How does the individual perceive the social workers?

A

Some are helpful, while others seem only concerned with payment

There is a mix of feelings regarding the social workers’ effectiveness and intentions.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

What scale is used to measure social service satisfaction?

A

A scale from one to five

The scale includes options from ‘Strongly agree’ to ‘Strongly disagree’.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

What does the social worker’s attitude affect according to the individual?

A

Feelings of embarrassment and trust

Some social workers ask embarrassing questions, affecting the individual’s comfort level.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

What is the purpose of the Self-Esteem Index (SEI)?

A

To measure problems with self-esteem

The SEI is a 25-item scale focusing on self-concept evaluative components.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

What are the scoring criteria for the Self-Esteem Index?

A

Scores above 30 indicate significant problems; below 30 indicates no problems

The SEI is designed to assess self-esteem issues.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

What is the reliability of the Self-Esteem Index?

A

Mean alpha of 0.93 indicating excellent internal consistency

The SEI has excellent stability with a high test-retest correlation.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

What demographic groups were included in the SEI study?

A

Single and married individuals, clinical and nonclinical populations, various ethnicities

The study included Caucasians, Japanese, Chinese Americans, and others.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

What are the key factors to evaluate when assessing standardized measuring instruments?

A
  • Sample representativeness
  • Validity of the instrument
  • Reliability of the instrument
  • Practicality of application

These factors help determine the accuracy and applicability of the instrument.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

What are some advantages of standardized measuring instruments?

A
  • Readily available and easy to access
  • Established reliability and validity
  • Norms available for comparison
  • Often free of charge

Standardized instruments are beneficial for systematic evaluations.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

What are some disadvantages of standardized measuring instruments?

A
  • Language may be difficult
  • Tone may not fit program philosophy
  • Target population may not understand the instrument
  • Scoring procedures may be complex

Disadvantages can affect the effectiveness of measurement in certain contexts.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

True or False: The Self-Esteem Index can be used with children under the age of 12.

A

False

The SEI is not recommended for use with children under 12.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Q

Fill in the blank: The SEI is designed to measure the _______ of a problem the client has with self-esteem.

A

[degree, severity, or magnitude]

The SEI evaluates how significant self-esteem issues are for clients.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
26
Q

What is the consequence of using a measuring instrument with a population it wasn’t tested on?

A

It may yield inaccurate results

It’s crucial to match the instrument with the characteristics of the population being assessed.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
27
Q

What is an important consideration regarding the validity of an instrument?

A

The content domain must be clearly defined

Validity ensures the instrument measures what it is intended to measure.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
28
Q

What does the term ‘instrument bias’ refer to?

A

When standardized tests do not accurately reflect the abilities of minority populations

Instrument bias can lead to underestimation of abilities.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
29
Q

What is the primary concern regarding standardized intelligence tests for ethnic minority children?

A

Scores may underestimate their actual abilities due to lack of representation in standardization samples.

The concern is that these tests require proficiency in the European American culture.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
30
Q

What does validity refer to in the context of measuring instruments?

A

Validity addresses the extent to which a measuring instrument achieves what it claims to measure.

Validity is particularly questioned when ethnic minorities are not included in the development of the instruments.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
31
Q

What is a potential misuse of measurement related to cultural values?

A

Assuming that all groups value variables equally can lead to misrepresentation and the assertion of superiority of one group’s values over another’s.

For example, spirituality may be more important to Native Americans than material possessions.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
32
Q

How can language create measurement issues in research with ethnic minorities?

A

Some ethnic minorities may lack proficiency in English, leading to potential misinterpretation of results from instruments designed only in English.

Translations of instruments may not be equivalent, affecting the validity of the findings.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
33
Q

What is a one-group post test-only design?

A

A design that measures the success of participants after they have undergone an intervention without comparison to another group.

It is often used to evaluate program objectives.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
34
Q

What are the five basic types of one-group evaluation designs?

A
  • One-group post test-only design
  • Cross-sectional survey design
  • Longitudinal designs (trend, cohort, panel)
  • One-group pretest-post test design
  • Interrupted time-series design.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
35
Q

What is a cross-sectional survey design?

A

A design that surveys a cross-section of a population only once to gather data.

It is commonly used in needs assessment studies.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
36
Q

What is the difference between trend studies and cohort studies in longitudinal designs?

A
  • Trend studies take different samples at different times.
  • Cohort studies follow a specific group over time.
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
37
Q

True or False: Two-group designs compare an experimental group against a control group.

A

True.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
38
Q

What issue arises from using culturally insensitive instruments with ethnic minorities?

A

It leads to misrepresentation and poor understanding of ethnic minorities.

The validity of studies using such instruments is often questioned.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
39
Q

Fill in the blank: The lack of __________ of measuring instruments with ethnic minority populations has been well documented.

A

sensitivity

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
40
Q

What is a trend study?

A

A trend study takes different samples of people who share a similar characteristic at different points in time.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
41
Q

How does a trend study differ from other types of studies?

A

A trend study samples different groups of people at different points in time from the same population.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
42
Q

What is the purpose of Antonia’s trend study?

A

To determine whether parents of second-grade children are becoming more receptive to child abuse prevention education.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
43
Q

What is a cohort study?

A

A cohort study takes place when evaluation participants who have a certain condition and/or receive a particular treatment are sampled over time.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
44
Q

In a cohort study, what is followed over time?

A

A particular cohort of people who have shared a similar experience.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
45
Q

What is a panel study?

A

In a panel study, the same individuals are followed over a period of time.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
46
Q

What is the primary advantage of panel studies?

A

Panel studies can reveal both net change and gross change for the same individuals.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
47
Q

What is the one-group pretest-posttest design?

A

A design that includes a pretest of the program objective, which can be used as a basis of comparison with the posttest results.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
48
Q

What does the pretest-posttest design help determine?

A

It helps determine how the intervention affects a particular group.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
49
Q

What is the interrupted time-series design?

A

A design that conducts a series of pretests and posttests on a group over time, before and after an independent variable is introduced.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
50
Q

What is the main characteristic of trend studies?

A

Data are collected from the population at more than one point in time.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
51
Q

True or False: In trend studies, there is experimental manipulation of variables.

A

False.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
52
Q

What does a cohort analysis attempt to identify?

A

Cohort effects and whether changes in the dependent variable are due to aging or other factors.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
53
Q

Fill in the blank: A cohort is any group of individuals who are linked in some way or who have experienced the same _______.

A

[significant life event].

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
54
Q

What type of studies are often used with public opinion polls?

A

Trend studies.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
55
Q

What can cohort studies tell us about populations?

A

What circumstances in early life are associated with the population’s characteristics in later life.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
56
Q

What is the significance of measuring the same individuals over time in a panel study?

A

It allows for the determination of changes in attitudes and behaviors that might go unnoticed in other research approaches.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
57
Q

What is one potential outcome indicator for evaluating a child abuse prevention program?

A

A reduction in parents’ risk for abusive and neglecting parenting behaviors.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
58
Q

What is a common use for cohort studies?

A

To follow a group linked by a shared condition or treatment over time.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
59
Q

What is the main purpose of panel studies?

A

To measure the same sample of respondents at different points in time

Panel studies can reveal both net change and gross change in the dependent variable for the same people.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
60
Q

What are the two types of panel studies?

A
  • Continuous panel
  • Interval panel
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
61
Q

What is internal validity?

A

The approximate certainty about inferences regarding cause-effect or causal relationships.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
62
Q

What does a one-group pretest-posttest design attempt to establish?

A

A relationship between the intervention and the program objective.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
63
Q

What is the effect of history on internal validity?

A

Any outside event that may affect the program objective and is not taken into account in the evaluation’s design.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
64
Q

Fill in the blank: The higher the internal validity, the greater the extent to which _______ can be controlled.

A

rival hypotheses

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
65
Q

What is maturation in the context of internal validity?

A

Changes, both physical and psychological, that take place in evaluation participants over time.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
66
Q

True or False: Maturation can be controlled by using a control or comparison group.

A

True

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
67
Q

What is the testing effect?

A

The effect that taking a pretest might have on posttest scores.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
68
Q

What is a potential threat to internal validity related to instrumentation?

A

Errors that occur when measuring instruments change over time.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
69
Q

List three threats to internal validity.

A
  • History
  • Maturation
  • Testing
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
70
Q

What is the purpose of establishing a baseline in a time-series design?

A

To ensure that normal fluctuations are not confused with the results of the policy or intervention.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
71
Q

Fill in the blank: Panel data are particularly useful in predicting _______.

A

long-term or cumulative effects

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
72
Q

What is a one-group pretest-posttest design?

A

A design where measurements of a program objective are taken before and after an intervention.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
73
Q

What is the role of a control group in a study?

A

To help control for extraneous variables like history.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
74
Q

True or False: Internal validity is irrelevant in studies that do not attempt to establish causal relationships.

A

True

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
75
Q

What does the term ‘reactive effects’ refer to in research?

A

Changes in participants’ behavior due to their awareness of being studied.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
76
Q

What are alternative explanations in the context of internal validity?

A

Other factors that could account for observed changes in the dependent variable.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
77
Q

What are the three steps of the one-group pretest-posttest design?

A
  1. Measuring some program objective
  2. Initiating a program to change that variable
  3. Measuring the program objective again at the conclusion of the program
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
78
Q

What is the testing effect?

A

The effect that taking a pretest might have on posttest scores, potentially influencing participants’ responses due to pretest exposure

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
79
Q

True or False: Testing effects are a threat to internal validity.

A

True

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
80
Q

What can cause a participant to score worse on a posttest?

A

Anxiety induced by the pretest or boredom from responding to the same questions again

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
81
Q

What is instrumentation error?

A

Weaknesses of a measuring instrument itself, such as invalidity, unreliability, improper administration, or mechanical breakdowns

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
82
Q

Fill in the blank: Instrumentation error refers to the weaknesses of a measuring instrument itself, such as _______.

A

invalidity, unreliability, improper administration, mechanical breakdowns

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
83
Q

What does statistical regression refer to?

A

The tendency of extremely low and extremely high scores to regress, or move toward the average score over time

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
84
Q

True or False: Statistical regression can be mistaken for the effects of an intervention.

A

True

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
85
Q

What is differential selection of evaluation participants?

A

The potential lack of equivalency among preformed groups of evaluation participants

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
86
Q

Why is mortality a threat to internal validity?

A

Participants dropping out may be different from those who stay, affecting the study’s findings

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
87
Q

Fill in the blank: Mortality refers to the loss of _______ through normal attrition over time in evaluation designs.

A

evaluation participants

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
88
Q

What are reactive effects?

A

Changes in behaviors or feelings of research participants caused by their reaction to the novelty of the situation or knowledge of being part of a study

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
89
Q

What historical study is associated with reactive effects?

A

Studies at the Hawthorne plant of the Western Electric Company

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
90
Q

What do interaction effects refer to?

A

The effects produced by the combination of two or more threats to internal validity

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
91
Q

What is the relationship between experimental and control groups concerning internal validity threats?

A

They can experience diffusion of treatments, compensatory equalization, compensatory rivalry, and demoralization

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
92
Q

What is diffusion of treatments?

A

When members of the experimental and control groups talk about the study, potentially invalidating the findings

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
93
Q

What occurs during compensatory equalization of treatment?

A

When researchers attempt to compensate control group participants who are not receiving the intervention

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
94
Q

What is compensatory rivalry?

A

When the control group becomes motivated to compete with the experimental group

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
95
Q

Fill in the blank: Demoralization refers to feelings of deprivation among the control group that may cause them to _______.

A

give up and drop out of the study

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
96
Q

What is demoralization in the context of control groups?

A

Feelings of deprivation among the control group that may cause them to give up and drop out of the study

This effect can also be referred to as mortality.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
97
Q

What is the primary focus of two-group designs in research?

A

To minimize threats to internal validity and provide data that approaches proving cause-effect relationships.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
98
Q

What is the comparison group pretest-post test design?

A

A design that includes a comparison group that receives both pretest and post test but does not receive the intervention.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
99
Q

True or False: In the comparison group pretest-post test design, random assignment to groups is used.

A

False

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
100
Q

What statistical technique can be used if pretest differences are not statistically significant but still affect the posttest?

A

Analysis of covariance.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
101
Q

What does the comparison group post test-only design improve upon?

A

It improves on the one-group post test-only design by introducing a comparison group that does not receive the intervention.

102
Q

In classical experimental design, what method is used to create experimental and control groups?

A

Random assignment method.

103
Q

What is the main advantage of the classical experimental design?

A

It controls for many threats to internal validity due to random assignment.

104
Q

Describe the randomized post test-only control group design.

A

Participants are randomly assigned to either an experimental group that receives the intervention or a control group that only takes the post test.

105
Q

What is external validity?

A

The degree to which the results of a specific study are generalizable to another population, setting, or time.

106
Q

List the four major threats to external validity relevant to program evaluations.

A
  • Selection-treatment interaction
  • Specificity of variables
  • Multiple-treatment interference
  • Researcher bias
107
Q

What does selection-treatment interaction refer to?

A

It occurs when an evaluation design cannot provide for random selection of participants from a population.

108
Q

Fill in the blank: Specificity of variables relates to the generalizability of results based on the __________ of the study sample.

A

[specific group of people]

109
Q

What is multiple-treatment interference?

A

It occurs when an evaluation participant is given two or more interventions in succession, affecting the results of each.

110
Q

How can researcher bias affect study outcomes?

A

Researchers may unconsciously manipulate a study to see results that align with their expectations.

111
Q

What is one method to control for researcher bias?

A

Conducting a double-blind experiment.

112
Q

What is the significance of pretests in evaluation designs?

A

Pretests help ensure equivalence between control and experimental groups.

113
Q

True or False: The randomized post test-only control group design requires a pretest.

A

False

114
Q

What should be allowed between two interventions to minimize multiple-treatment interference?

A

Sufficient time.

115
Q

What is the final takeaway regarding evaluation designs?

A

Each design has its advantages and disadvantages, and familiarity with them helps in selecting the most appropriate one for a specific evaluative effort.

116
Q

What are the three basic procedures needed for program evaluation?

A
  1. Selecting a data source
  2. Selecting a sample from your data source
  3. Collecting data from your sample
117
Q

What can serve as data sources in evaluations?

A

Data sources can be:
* People
* Existing data

118
Q

Who are some examples of people that can be data sources?

A

Examples include:
* Federal and state personnel (politicians, government officials)
* Program workers (therapists, caseworkers)
* Clients (individuals, families, groups, communities)

119
Q

What is a key characteristic of the best data sources?

A

The best data sources provide firsthand or direct knowledge about the experience being evaluated.

120
Q

What is the difference between firsthand and secondhand data?

A

Firsthand data comes directly from individuals with experience, while secondhand data is provided by individuals with indirect knowledge.

121
Q

What are existing data sources?

A

Existing data sources are previously recorded documents or artifacts relevant to current evaluation questions.

122
Q

Name three general areas where existing data can be found.

A
  1. Public data (census data, government documents)
  2. Client data (client records, service plans)
  3. Program data (evaluation reports, program contracts)
123
Q

What is a sampling frame?

A

A sampling frame is a comprehensive list of every unit (people, documents, artifacts) from which a sample is drawn.

124
Q

What are the two main types of sampling methods?

A
  1. Probability sampling
  2. Nonprobability sampling
125
Q

What is the benefit of probability sampling?

A

Probability sampling produces samples considered representative of the larger sampling frame.

126
Q

What is simple random sampling?

A

Simple random sampling involves selecting each unit using a chance procedure (e.g., rolling dice).

127
Q

How does systematic random sampling work?

A

Systematic random sampling involves determining the total number of units, the desired sample size, calculating an interval, and randomly selecting a starting point.

128
Q

What is stratified random sampling?

A

Stratified random sampling involves identifying relevant strata, determining their proportions in the population, and using random sampling within each stratum.

129
Q

What is cluster sampling?

A

Cluster sampling involves selecting clusters randomly and then sampling units within those clusters.

130
Q

What is convenience or availability sampling?

A

Convenience or availability sampling includes the nearest or most accessible units.

131
Q

Fill in the blank: The best data sources are those that provide _______ knowledge regarding the experience that is the subject of your evaluation.

A

firsthand

132
Q

True or False: Nonprobability sampling ensures that each unit in a sampling frame has an equal chance of being selected.

A

False

133
Q

What is purposive sampling?

A

Purposive sampling includes units known or presumed to provide good data.

134
Q

What is nonprobability sampling?

A

Nonprobability sampling methods do not give each unit in a sampling frame an equal chance of being picked for an evaluation study.

135
Q

Name the four types of nonprobability sampling.

A
  • Convenience or Availability Sampling
  • Purposive Sampling
  • Quota Sampling
  • Snowball Sampling
136
Q

What is Convenience or Availability Sampling?

A

Include the nearest or most available units.

137
Q

What is Purposive Sampling?

A

Include units known or used to be good data sources based on some theoretical criteria.

138
Q

What is Quota Sampling?

A

Identify variables relevant to the evaluation, combine them into discrete categories, determine the percentage of each category, calculate quotas, and select data sources until each quota is filled.

139
Q

What is Snowball Sampling?

A

Locate a small number of data sources, then ask them to identify others in the population, continuing until the desired sample size is obtained.

140
Q

Why are nonprobability sampling methods used?

A

They are used in situations where it’s desirable to limit or pick data sources based on some unique characteristic.

141
Q

What is the aim of nonprobability sampling strategies?

A

To produce quality, firsthand data from sources that share something in common.

142
Q

When is it necessary to use sampling strategies in an evaluation plan?

A

When the sampling frame is large, previous efforts to include all units have failed, only unique data sources are desired, program resources are limited, or multiple data sources are needed.

143
Q

What are the advantages of Simple Random Sampling?

A
  • High representativeness if all subjects participate
  • Ideal sampling method
144
Q

What are the disadvantages of Simple Random Sampling?

A
  • Not possible without a complete list of population members
  • Potentially uneconomical to achieve
145
Q

What is Stratified Random Sampling?

A

Random sample from identifiable groups (strata), ensuring specific group representation.

146
Q

What is Cluster Sampling?

A

Random samples of successive clusters of subjects, allowing random selection when no single list of population members exists.

147
Q

What is Purposive Sampling?

A

Hand-pick subjects based on specific characteristics.

148
Q

What is Quota Sampling?

A

Select individuals as they come to fill a quota by characteristics proportional to populations.

149
Q

What is Accidental Sampling?

A

Includes volunteers or subjects who happen to be available.

150
Q

What is Participant Observation?

A

The evaluator has no formal role but observes as a participant.

151
Q

What are the main data collection methods mentioned?

A
  • One-to-One Interview (General)
  • One-to-One Interview (Unstructured)
  • One-to-One Interview (Semi-structured)
  • One-to-One Interview (Structured)
  • Focus Group
  • Phone Interview
  • Participant Observation
152
Q

What are the advantages of One-to-One Interviews?

A
  • Provide in-depth information
  • Allow for clarification and probing
153
Q

What are the disadvantages of Focus Groups?

A
  • Requires great skill from the interviewer
  • Data can be challenging to analyze
154
Q

Fill in the blank: Nonprobability sampling methods are used when it’s desirable to limit or pick data sources based on _______.

A

[unique characteristic]

155
Q

True or False: Nonprobability sampling allows for generalization of characteristics to the larger population.

A

False

156
Q

What should be clear when selecting data collection methods?

A

The evaluation question should guide the selection of data sources and data-collection methods.

157
Q

What is a critical factor to ensure when collecting data?

A

Develop protocols that yield credible data.

158
Q

What is the role of the evaluator in participant observation?

A

The evaluator has no formal role as a participant; they act as a silent observer.

159
Q

What is a major advantage of the evaluator’s role in participant observation?

A

The evaluator retains beliefs of the participant without ethical issues at stake.

160
Q

What is a disadvantage of participant observation?

A

It is difficult to maintain distinct roles; the observer’s presence can change the nature of interactions.

161
Q

What are the two main types of data discussed?

A
  • Existing data
  • New data
162
Q

What are examples of existing data?

A
  • Documents and reports
  • Datasets
163
Q

What is the purpose of using existing data?

A

To profile recent and past characteristics or patterns that describe communities, clients, workers, or program services.

164
Q

What are the challenges of gathering existing data?

A

Old documents may not be easily accessible and there may be no existing data recorded.

165
Q

What is the significance of reviewing existing documents?

A

It provides insights from previously analyzed and summarized data.

166
Q

What types of data are included in existing statistical data?

A

Numbers and figures calculated from original raw data.

167
Q

What are datasets?

A

Datasets store existing raw or original data and organize them to connect all data elements to their source.

168
Q

What is the disadvantage of census data?

A

Census data can become outdated quickly and provide only a general picture of a population.

169
Q

What types of data do client datasets include?

A
  • Demographics
  • Treatment progress
  • Intake forms
  • Assessments
170
Q

What are the two main problems associated with client and program datasets?

A
  • Data may be incomplete or inconsistently recorded
  • Data apply to a specific point in time
171
Q

What methods can be used to obtain new data?

A
  • Individual interviews
  • Surveys
  • Group interviews
  • Observations
172
Q

What is a key characteristic of individual interviews?

A

They can produce new, original data about social needs, program processes, or outcomes.

173
Q

Fill in the blank: The evaluator’s presence can change the nature of the _______.

A

interactions being observed.

174
Q

True or False: Reviewing existing documents can save time and help avoid reinventing the wheel.

A

True

175
Q

What are examples of existing documents used in evaluations?

A
  • Published research studies
  • Government documents
  • Client reports
176
Q

What is the main goal of face-to-face interviews in needs assessment?

A

To ask questions that permit open-ended responses.

177
Q

What type of interviews are used when prior knowledge of the topic exists?

A

Structured interviews.

178
Q

When is it appropriate to use informal unstructured interviews?

A

When very little is known about the problem area.

179
Q

What is a key characteristic of informal interviews?

A

They allow for a free-flowing discussion.

180
Q

What do surveys aim to gather?

A

Opinions from numerous people to describe them as a group.

181
Q

How do survey questions differ from structured and unstructured interview questions?

A

Survey questions are narrower and yield shorter responses.

182
Q

What should be considered when creating survey questions?

A

They should yield valid and reliable responses.

183
Q

What is one disadvantage of mail surveys?

A

Low response rate.

184
Q

List three common methods of administering surveys.

A
  • Mail
  • Telephone interview
  • In-person interviews
185
Q

What is one strategy to increase the number of survey respondents?

A

Include a cover letter stating the purpose of the evaluation.

186
Q

What are the three strategies for conducting group interviews?

A
  • Open forums
  • Focus groups
  • Nominal groups
187
Q

What is the purpose of open forums?

A

To address general evaluation questions and gather stakeholder responses.

188
Q

What is a focus group designed to do?

A

Gather perceptions about a predetermined topic of interest.

189
Q

What role does a facilitator play in a focus group?

A

Guides discussion and ensures balanced participation.

190
Q

What is the nominal group technique?

A

A structured method for collecting data from individuals with limited interaction.

191
Q

What is the main difference between structured observations and participant observations?

A

Structured observations are objective, while participant observations involve the observer’s involvement.

192
Q

What is an essential aspect of structured observations?

A

They occur under controlled conditions and aim to collect precise data.

193
Q

What is the role of the observer in structured observations?

A

To remain impartial and focus on specific interactions.

194
Q

What is a key advantage of using the nominal group technique?

A

Efficient data collection from numerous sources.

195
Q

Fill in the blank: The main goal of surveys is to gather ______ from numerous people.

A

[opinions]

196
Q

True or False: Surveys rely heavily on interviewer skills to separate responses.

A

False.

197
Q

What should be done after administering a survey?

A

Enter and tabulate survey information.

198
Q

What is a common disadvantage of open forums?

A

They tend to draw a select group of people with strong opinions.

199
Q

What is the purpose of follow-up letters in survey administration?

A

To prompt nonrespondents to complete the survey.

200
Q

What should be included in the initial orientation for focus group participants?

A

An introduction to the topic of focus.

201
Q

In a nominal group, what is the process for ranking responses?

A

Individually rank the top five responses.

202
Q

What should observers in structured observations record?

A

Only observable facts related to the predefined dimensions.

203
Q

What is a significant factor in ensuring the validity of structured observations?

A

Using trained observers and precise protocols.

204
Q

What should be the characteristics of reports for outside organizations?

A

Reports should be clear and understandable.

205
Q

How can participant observation differ from structured observation?

A
  1. The observer is not impartial.
  2. The rules for observation are far more flexible.
206
Q

What is a challenge for participant-observers?

A

To balance their dual roles so that data are based on fact and not personal impressions.

207
Q

What are the basic tasks in implementing regular trained observer measurements?

A
  1. Identify specific data needed.
  2. Develop the trained observer rating guide.
  3. Test the guide with raters.
  4. Decide when ratings will be made and reported.
  5. Select and train observers.
  6. Assign staff to oversee the process.
208
Q

True or False: Participant-observers interact with the people they are watching.

A

True.

209
Q

What is the benefit of participant observation?

A

Members of the group can pick up subtle or cultural nuances that may be obscure to an impartial viewer.

210
Q

What is a data-collection plan?

A

A structured approach to gather and evaluate data for program objectives.

211
Q

What should the data-collection plan specify?

A
  1. Program objectives to be measured.
  2. Measurement methods for each objective.
  3. Data sources.
  4. Data-collection methods.
  5. Time frame for data collection.
  6. Location for data collection.
  7. Who will collect the data.
212
Q

Fill in the blank: The variable for self-esteem can be measured using the _______.

A

Rosenberg Self-Esteem Scale.

213
Q

What is essential for data collectors to ensure high-quality data?

A

Training and supervision.

214
Q

What should training for data collectors aim to achieve?

A
  1. Consistent application of standards and procedures.
  2. Understanding of how data will be used.
  3. Clarity of roles and responsibilities.
  4. Preparedness for handling events that may arise.
215
Q

What types of training methods can be used for data collectors?

A
  1. Written instructions.
  2. Verbal instructions.
  3. Meetings.
  4. Data-sharing agreements.
  5. Train-the-trainers approach.
  6. Formal training.
216
Q

What is the purpose of a Memorandum of Understanding in data collection?

A

To establish formal agreements on data access, usage, and conditions.

217
Q

What should be included in formal training for data collectors?

A

Background material about the data being collected, including type, source, and purpose.

218
Q

What factors influence the selection of training methods for data collectors?

A

Audience, training needs, resources, and personal style.

219
Q

What is the purpose of providing background material in data-collection training?

A

To clarify the type of data being collected, from whom, and for what purpose

Background material helps data collectors feel more confident, motivates them to obtain high-quality data, and aids in troubleshooting.

220
Q

What should data-collection instructions cover?

A

Every aspect of data collection, including:
* Identifying or locating appropriate respondents or records
* Processing the collected data
* Specific roles and responsibilities

221
Q

True or False: Data collectors need to know their own roles and responsibilities.

A

True

Understanding their roles helps ensure proper data collection and management.

222
Q

Fill in the blank: An evaluation overview statement can be developed and used to provide _______.

A

[information about the evaluation purpose and data usage]

223
Q

What is a common training topic related to data collection logistics?

A

Identifying appropriate respondents/records

Understanding the importance of adhering to data-collection protocols preserves data quality.

224
Q

What is the importance of recruiting participants effectively?

A

To achieve high response rates and protect respondents’ rights to refuse participation.

225
Q

What should data collectors be trained to do when gaining access to records?

A

Know what to say to gain admittance and request records.

226
Q

What does accurate recording of data ensure?

A

Meaningful comparison and interpretation of the data.

227
Q

What should data collectors do if they encounter an emotionally upset respondent?

A

Terminate or reschedule the interview.

228
Q

What is a critical consideration regarding data confidentiality and security?

A

Understanding who is allowed access and what to do in case of a breach.

229
Q

What is the role of supervision and monitoring in data collection?

A

To ensure data are collected appropriately and resolve issues as they arise.

230
Q

What should training on data collection include regarding feedback?

A

Methods for routinely gathering feedback from data collectors.

231
Q

True or False: Data collectors should be aware of special considerations when working with different target audiences.

A

True

This includes cultural or religious customs and disabilities that need to be accommodated.

232
Q

What is one strategy for ensuring data collectors’ safety during fieldwork?

A

Pairing them to work together in a ‘buddy’ system.

233
Q

What should be emphasized in team training for data collectors?

A

How to work together and how their roles complement one another.

234
Q

Fill in the blank: Conducting practice sessions allows data collectors to _______.

A

[practice all aspects of the data-collection protocol]

235
Q

What is the significance of conducting a pilot test of data-collection methods?

A

To identify potential issues and refine the data-collection protocol.

236
Q

What is a key tip for successful data-collection training?

A

Always conduct some type of data-collection training.

237
Q

Why is it important to avoid assuming data-collection procedures are intuitive?

A

To prevent misunderstandings that can result in unusable data.

238
Q

What should be included in a handbook for data collectors?

A

Protocols, measuring instruments, instructions, contact numbers, and supplementary materials.

239
Q

What is the primary purpose of data-collection training?

A

To ensure all data-collection activities in your evaluation are conducted properly and effectively

Data-collection training, whether formal or informal, is crucial to avoid misunderstandings and ensure data usefulness.

240
Q

True or False: Experienced data collectors do not need training.

A

False

Even experienced data collectors benefit from training to understand specific procedures for each evaluation.

241
Q

What is a key factor when selecting trainers for data-collection teams?

A

Use high-quality trainers

Recruiting the best supervisors and trainers can improve the performance of data collectors.

242
Q

Why is it important to ensure respondent comfort?

A

Respondents need to feel comfortable with data collectors

This may involve selecting data collectors who share similar backgrounds with respondents.

243
Q

Fill in the blank: Training needs should be considered _______.

A

broadly

Consider all procedures needed to access and use data, even from secondary sources.

244
Q

What should data collectors be encouraged to report?

A

Problems and observations

Data collectors’ observations are invaluable as they are closest to the evaluation implementation.

245
Q

What is the importance of documentation in data collection?

A

Ensures approaches are well documented and others can take over if necessary

Documentation also serves as a historical record of evaluation methods.

246
Q

What does ongoing monitoring of the data-collection process help identify?

A

Whether data collection is proceeding as planned

It allows for intervention or additional training if needed.

247
Q

List three didactic approaches for training.

A
  • Overview of the evaluation
  • Understanding evaluation standards
  • Review of data collection instruments

Didactic approaches ensure important content is conveyed in a structured way.

248
Q

What is an example of a hands-on training approach?

A

Role-playing

Role-playing simulates actual data-collection situations allowing practice and feedback.

249
Q

What checklist items should be included in data-collector training?

A
  • Background information on the program
  • Clear instructions on data collection
  • Expectations regarding professional evaluation standards
  • Contact information for questions
  • Discussion on data custody and safeguarding

This checklist ensures comprehensive training for data collectors.

250
Q

What is the summary conclusion regarding data collector training?

A

Data collectors must be highly trained and supervised to collect high-quality data

Proper training ensures data meets standards of utility, accuracy, and propriety.

SWG 623 Program Evaluation Finals (4 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2025 Bold Learning Solutions. Terms and Conditions