Chapter 3 Flashcards
How can you identify learning in Kirkpatrick’s model?
Examine pretest to posttest change on learning scores
What might be a disadvantage of using an experimental approach in evaluation?
A. The variables and interactions between variables are complex to be worked out in advance.
B. The variables and interactions between variables are complex to be worked out in advance.
C. Ensuring that treatment and control groups are separate during the study
You can use Kirkpatrick’s evaluation model for any kind of program.
False
Positivists and postpositivists believe that ___________ the object of study reduces bias.
A. distance from
B. limiting contact with
Which of the following adds some justification of the methodological choice of a randomized design?
A. When you want to establish a cause-and-effect-relationship
B. When random assignment is not problematic.
C. When there is uncertainty about the effect of an intervention.
What are the four levels (in order) of the Kirkpatrick model of evaluation?
C. reaction, learning, behavior, results
If a randomized experimental approach is not feasible, what are some other alternatives?
A. In-depth case studies
B. Statistical analysis of observational data
C. Quasi-experimental comparison group studies
One of Donald Campbell’s major contributions to the field of evaluation is his work on experimental and quasi-experimental designs.
True
The methodological assumption(s) in the postpositivist-and-methods-branch of evaluation includes which of the following?
Scientific method, quantitative methods, and hypothesis
Ralph Tyler
1930s- Educational evaluation-established education, object, then determine whether those objectives had been met
Donald Campbell
The use of experimental and quasi-experimental designs and their role in controlling for extraneous variables in determining causal relationships
Gary Henry & Melvin Mark
Use of causal modelling for evaluation with attention to ethical issues.
Emergent Realist Evaluation
philosophy of neorealism in that a reality exists independent of the observer & regularities in the pattern of events can be explained by generative mechanisms
What are 5 environments that affect transfer of training?
- Preventing-supervisor doesn’t allow
- Discouraging-supervisor discourages use of new training
- Encouraging-supervisor encourages new use of skills
- Neutral-supervisor doesn’t acknowledge new training
- Requiring-supervisor monitors and requires use of training
National Comm. for the pRotection of Human Subjects of Biomedical and Behavioral Research (1979( identifies 3 ethical principles, what are they?
- Beneficence (unnecessary risk or harm)
- Respect
- Justice (those who bear risk are the benefactors)
NCforProt of Human Subjects identifies 6 norms. what are they?
- valid research design
- research must be competent
- Consequences must be identified.
- Sample must be appropriate for the study, rep. the population
- Volunteer signs informed consent
- Researcher must inform the participant whether harm will be compensated.
What are Kirkhart’s 5 justifications for considering validity from a multicultural perspective?
- Interpersonal-quality of interactions b/w part. in eval. process
- Consequential-
- Experiential
- Theoretical
- Methodological
You will need to ___________ the evaluation questions. Then ___________________. Give high priority to evaluation questions that are likely to lead to the ________.
prioritize, eliminate decreased priority questions for your final evaluation plan, utilization of the evaluation.
THeory Failure
Program is implemented but the theory is faulty hence the expected results do not occur.
Implementation Failure
the planned program services are not delivered hence the expected results do not occur
Backward Mapping
Identifying desirable endpoints or outcomes and then determining what must be done to get there
What are the 5 major evaluation domains in the RLF evaluation model (Rossi, Lipsey, freeman)
1. Needs Assessment 2 .Theory Assessment 3. IMplementation Assessment 4. IMpact Assessment 5. Efficiency Assessment
Independent Variable
Dependent Variable
the program, policy or process that is implemented in hopes of serving a change in knowledge, beh. attitude (the dependent variable).
When are quasi-experimental used?
when evaluators are not able to assign participants randomly to treatment groups
case studies
stat analysis of observ. data
Theory Based Evaluation
Chen & Rossi, lipsey, Donaldson, an approach that focuses on the theories people have about what it takes to have a successful program
Positivists hold the belief ontologically that_____.
One reality exists that is independent of the observer, bias in the research can be reduced by distance from the object of study, and limited interactions with object of study
What might be a disadvantage of using an experimental approach in evaluation?
A. The variables and interactions between variables are complex to be worked out in advance.
B. Inability to control exposure to the intervention.
C. Ensuring that treatment and control groups are separate during the study
You can use Kirkpatrick’s evaluation model for any kind of program. T or F
False
Which of the following adds some justification of the methodological choice of a randomized design?
A. When you want to establish a cause-and-effect-relationship
B. When random assignment is not problematic.
C. When there is uncertainty about the effect of an intervention.
D. All of the above
If a randomized experimental approach is not feasible, what are some other alternatives?
A. In-depth case studies
B. Statistical analysis of observational data
C. Quasi-experimental comparison group studies
D. All of the above might be an alternative
One of Donald Campbell’s major contributions to the field of evaluation is his work on experimental and quasi-experimental designs. T or F
True
Which of the following names is NOT associated with theory-based evaluation?
A. Huey-Tsyh Chen
B. David Fetterman
C. Mark Lipsey
D. Stewart Donaldson
The methodological assumption(s) in the postpositivist-and-methods-branch of evaluation includes which of the following?
Scientific method, quantitative methods, and hypothesis
RLF Model
Rossi, Lipsey, Freeman, 2004
Each evaluation should be tailored to fit local needs, resources, and type of program.
Eval. questions are the core.
Black-box evaluation
focus only on desired outcomes and not the processes between input and output
Method driven evaluation
prioritized research methods and focuses only on impact
A _____ theory includes and action model and change model.
Program
The _____ is a program’s systematic plan for organizing resources, settings, staff, and support organizations to deliver intervention services and to teach the target pouplation.
Action model
The ____ is the program’s description of the causal processes and outcomes required to resolve the identified problems.
change model
Brinkerhoff’s Success case method
Finding out what about a training or other organizationl intervention worked
Examine successful cases and compare them to other cases
__________ involves interviewing or having trainees and their supervisors, peers, and subordinates fill out questionnaires several weeks and months after the training event to measure their perceptions about whether the trainees are appllying what they learned.
Retrospective Survey Design
Included in Level 3-Behavior or Transfer of Training in Kirkpatrick’s 4 levels of evaluation
_____evaluation focuses on improving the evaluation object.
Formative
_____evaluation focuses on determining the overall effectiveness , usefulness, or worth of the evaluation object.
Summative
What is “Law enforcement training in southeast Asia-a theory driven evaluation about?”
Johnson, Young, Foster, & shamblen
Assessed an international law enforcement training progaram designed to aid law enforcement officials in combating international crime
Example of Kirkpatrick’s 4 Levels of Eval.
___________ means that one can listen to and use more than one paradigm in the same evaluation study
dialetical pluralism
“Evaluation of the Suicide Prevention Program in Kaohsiung City Taiwan”
HO, Chen, Ho, Lee, Chen, Chou
CIPP example
Evaluate effectiveness of the Kaohsiung Suicide Prevention Center
The most improtant purpose is not to prove, to improve an idea
Underreporting of suicide, financial constraints, Universal strategies
