Lesson 11 Flashcards
What is Evaluation Design?
AKA: Research Design. It is the plan established for data collection or hypothesis testing
What is the Evaluation Design attempting to address?
Causality
What are 3 design characteristics to improving causality:
- Comparison/control group similar to the
experimental/exposed group - Measurement of outcome variables before and after the intervention
- Minimal threats to internal and external validity
R- represents
Random assignment of an evaluation study participant (or unit) to a group
E- represents
Experimental (intervention or treatment) group
E1, E2, E3, …., En indicates planned exposure of the group to different intervention procedures
C- represents
Control (equivalent) group established only by random assignment
Indicates no exposure to an intervention or exposure to a minimum or standard intervention procedures
C underlined -represents
Comparison group established through any method other than randomization
X represents
Intervention procedures applied to an E (experimental) group
X1, X2, X3,…., Xn indicates an intervention consisting of multiple, different procedures
O represents
An observation or measurement to collect data. Including:
Tests, Interviews, Visual or audio ratings, or Record reviews
O1, O2, O3,…., On indicates multiple measurements at different times
What is a Non-Experimental Design?
Includes one experimental group (E) but no control group (C) or comparison group (C). Weakest of the three research designs
In regards to internal validity and causation
What is a Quasi-Experimental Design?
Includes an experimental group (E) and a comparison group (C) created by methods other than random assignment
Includes observations of both groups, both prior to and after application of intervention (X) procedures
Multiple ‘waves’ of data collection May yield interpretable and supportive evidence of behavioral impact Asserts varying degrees of control over several biases
But usually does not account for all biases of internal validity
What is an Experimental Design?
Includes random assignment of experimental (E) and a control (C) group
Observations of both groups, prior to and after application of the intervention (X) procedures
Yields the most interpretable and defensible evidence of behavioral impact
Strongest (Most rigorous, ‘Gold standard’)
Asserts the greatest degree of control over biasing factors that compromise internal validity of results
What are the most common Research and Evaluation Designs?
One-Shot Case Study Retrospective Pretest One-group Pretest-Posttest Design Time Series Design Pretest-Posttest Control-Group Design Case Study Design
Explain a One-Shot Case Study
X——— 0
Survey of program participants after participation in the program
Absolute simplest form of outcome evaluation
What are the Primary Weaknesses of a One-Shot Case Study?
Does not demonstrate change during or as a result of the program
Best if used to answer descriptive research questions Ex: Effects of a tornado on a community
Interviews will provide good information, but hard to establish observations are due to the disaster
Explain a RETROSPECTIVE PRETEST
X——— 0
Survey of program participants after participation in the program
However, asks respondents to recall information before and after the intervention
What is a weakness of a Retrospective Pretest?
Recall bias
Relies on memory
Explain the One Group Pretest-Posttest Design
01—–X—–02
Basic in design
Evaluator collects data before and after the program is conducted
What is the primary weakness of the One Group Pretest-Posttest Design?
Potential selection biases
Attrition
Potential measurement bias
Poor validity Poor reliability
What is INTERNAL VALIDITY?
Validity is synonymous with accuracy
Degree to which measured change can be attributed to the program
What is EXTERNAL VALIDITY?
Generalizability.
Extent to which the program can produce similar effects in other populations
What are two Non-program related sources of bias?
Maturation Historical Bias
What are three Program related sources of bias?
Attrition Selection Bias Measurement Bias
How can we increase internal validity?
Observe participants at additional times before and after a program (multiple waves) Was change random or systematic?
Observe additional natural groups who have no exposure to the program (control /comparison group)
In other words, use a quasi-experimental or experimental design
How to demonstrate causality?
- The cause precedes the supposed effect in time
- The cause covaries with the effect
More of the cause = More of the effect. Vice versa - No viable explanation of the effect can be found except for the assumed cause
Explain the TIME SERIES DESIGN
Evaluator gathers data at multiple times before and after the program is delivered
When is the Time Series Design useful?
Useful in examining the effects of a new health- related policy Minimum Legal Drinking Age Seatbelt law Raising cigarette taxes Mandatory to wear motorcycle helmets
Explain the MULTIPLE TIME SERIES DESIGN
The addition of the C group may strengthen the program’s control over the primary three biases
What are limitations of the multiple time series design?
Need the same multiple observations of C group
before, during, and after the program
A lot of people and a lot of observations Time and $$ intensive
NON-EQUIVALENT CONTROL-GROUP DESIGN
Pretest & Posttest Comparison Group Design:
01—–X—–02
01—– —–02
Quasi Experimental Design
ADD R and becomes Experimental
What are several things to remember when implementing a Non-Equivalent Control Group Design?
Standardized assessments at baseline and follow- up are conducted of all E and C / C group participants
Two groups should not be significantly different at baseline for any IVs or DVs that are predictors of impact
Assuming program fidelity, this design produces good control over three major biases
Isolation of the effects of the program from other plausible interpretations the Non Equivalent Control Group Design by?
Maturation: Groups tested at the same time (same amount of time
to mature)
Historical Bias: Groups are equally exposed affected by historical events
Testing Effects: Both groups were tested twice (both become more familiar)
Attrition: Can compare to see if both rates are similar?
