Final Flashcards
An evaluation that uses the data collected before a program begins. Examples include pilot projects, baseline surveys, and feasibility studies.
Design evaluation
Coefficient of stability: Evaluator administers the same instrument twice, separated by a short period of time. Results are compared, using a statistic such as a correlation coefficient
Reliability coefficient/ test-retest reliability
Evaluator administers two equivalent versions of the same instrument (parallel forms) to the same group of people. Results are compared, using a coefficient of stability.
Alternate-form coefficient
A nonexperimental research method relying on questionnaires or interview protocols
Survey
The extent to which researchers can make this statement with confidence
Internal validity
Study conducted at a single time period, and data are collected from multiple groups
Cross-sectional study
Data are collected at two or more points in time
Longitudinal study
Design that combines cross-sectional and longitudinal elements by following two or more age groups over time
Cohort-sequential design
Two observers’ data are compared to see whether they are consistently recording the same behaviors when they view the same events. Statistical procedures such as correlation or percentage of agreement can be used to establish this type of reliability.
Interrater reliability
Interpretive research approach relying on multiple types of subjective data and investigation of people in particular situations in their natural environment
Qualitative research
Use of multiple data sources, research methods, investigators, and/or theories/ perspectives to cross-check and corroborate research data and conclusions
Triangulation
used to determine whether a single observer is consistently recording data over a period of time.
Intrarater reliability
To what degree does all accumulated evidence support the intended interpretation of scores for the proposed purpose?
construct validity
commonly used data collection instruments or procedures designed to measure personality, aptitude, achievement, and performance.
Tests
Items on the test represent content covered in the program (e.g., did the teacher teach the children that blue on the map means water?). Evaluators can work with content specialists to list the content that is part of the program, and can compare the test items to see whether they correspond.
Content-related evidence
a self-report data collection instrument that is filled out by research participants.
Questionnaire
The researcher takes independent samples from a general population over time and the same questions are asked.
Trend study
a situation where the interviewer asks the interviewee a series of questions
Interview
indicates that the measure actually reflects current or future behaviors or dispositions.
Criterion-related evidence
a situation where a focus group moderator keeps a small and homogeneous group (of 6–12 people) focused on the discussion of a research topic or issue.
Focus group
Evaluators need to be aware of the consequences of using data, especially with regard to the potential to worsen inequities.
Consequential evidence
Researcher watches and records events or
behavioral patterns of people
Observation
Observation conducted in real-world situations
Naturalistic observation
Observation conducted in lab setting set up by the researcher
Laboratory observation
Evaluators need to stay on site for sufficient time to “get the story right.” If a study is conducted over too short a time or interviews are conducted with too few people, it is possible that an evaluator will reach “premature closure”
Prolonged and substantial engagement
the ability to generalize the results of a study to the population from which the research population was drawn.
External validity
A word that produces an emotionally charged reaction
Loaded term
A question that suggests how the participants should
answer
leading question
Asking about two or more issues in a single question
Double-barreled question
Nonoverlapping response categories
Mutually exclusive categories
Response categories that cover the full range of possible responses
Exhaustive categories
An ordered set of response choices, such as a 5-point rating scale, measuring the direction and strength of an attitude
Rating scale
Descriptors placed on points on a rating scale
Anchor
Participant must select from the two response choices provided with an item
Binary forced-choice approach
A scaling technique that is used to measure the meaning that participants give to attitudinal objects or concepts and to produce semantic profiles
Semantic differential - a scaling technique that is used to measure the meaning that participants give to attitudinal objects or concepts and to produce semantic profiles
An item directing the participant to different follow-up questions depending on the initial response
Contingency question
Tendency for a participant to respond in a
particular way to a set of items
response
Evaluators need to be aware of their assumptions, hypotheses, and understandings, and of how these change over the period of the study.
Progressive subjectivity
those incurred to provide a treatment or a service
direct costs
resources lost due to the disorder.
indirect costs
Events that occur during the study other than the experimental treatment effect that can influence the results
History threat
describing design that is intended to simplify life for everyone by making products, communications, and the built environment more usable by as many people as possible at little or no extra cost
Universal design
naturally occurring physical or psychological changes
Maturation threat
A monitoring system that would involve community members in determining what indicators of change are important to them
Most Significant Change Method
Christian Commission for Development in Bangladesh and Rick Davies
Administration of a pretest that affects participants’ performance on a posttest (i.e., participants become “testwise”).
testing threat
Having pretests and posttests that differ in terms of difficulty; this can lead to seeming changes that are really due to the difference in the tests.
Instrumentation
Having extreme groups in a study (i.e., people very high or very low on a particular characteristic); it is possible that changes will be seen on the dependent variable because of this threat.
statistical regression
Differences between the experimental and control groups on important characteristics, other than receipt of the intervention.
differential selection
Differential dropouts of participants from either the experimental or control groups.
experimental mortality
treatment was not implemented as intended
treatment fidelity
monetary (i.e., dollar) values are estimated for the resources used (i.e., costs) and for the program effects (i.e., benefits), and these two components (costs and benefits) are then compared to determine the worth of a program.
cost-benefit analysis
based on cost and effect data (rather than cost and benefit data as in benefit-cost analysis).
cost-effectiveness analysis
What a concept means in abstract or theoretical terms
Conceptual definition of a sample
What you will observe and/or measure; links the concept you want to sample to the real world
Operational definition of a sample
Group to which you wish to generalize your results
target
List of people who match the conceptual definition
Experimentally accessible population
List of people in the experimentally accessible population
Sampling frame
Match between accessible population and target population
population validity
voluntary consent without threat or undue inducement; it includes knowing what a reasonable person would want to know before giving consent (informed), and explicitly agreeing to participate (consent).
informed consent
collecting, analyzing, storing, and reporting data in such a way that the data cannot be traced back to the individual who provides them.
confidentiality
involves the selection of a sample from a population in a way that allows for an estimation of the amount of possible bias and sampling error.
Probability-based sampling
Those in which every member of a population has a known nonzero probability of being included in the sample.
Random samples
Every member of the population has an equal and independent chance of being selected.
Simple random sampling
A robust description of the context in which qualitative research is conducted
Thick description
Selection of easily obtainable participants for sample group and usually the cheapest and fastest way of obtaining a sample group; interviewers happen to be available at a program site.
Convenience sampling
When will you have a poor strength of treatment?
A. When the participants are sensitized to the posttest.
B. When the evaluator uses different types of measurement for dependent variables.
C. When the treatment “dose” was not strong enough to produce the expected change.
D. When the evaluator systematically conducts an intervention at different times of the day.
C
In evaluation, a limitation of traditional definitions of needs assessment (according to your textbook) is that they focus only on needs and lack any attention to assets?
A. True
B. False
True
The assumption that all people within a particular subgroup are similar to each other in terms of their other background characteristics, or similar enough where differences are not identified.
Myth of homogeneity
Which of the following is(are) needed to infer causality?
A. X precedes Y
B. X is related to Y
C. Confounding variables can be ruled out as causes
D. All of the above are required.
D
When an extraneous variable systematically varies with the independent variable and influences the dependent variable, it is called: A. Another dependent variable B. A confounding variable C. A moderating variable D. An unreliable variable
B