SCC Chapter 5 – Designs That Use Both Control Groups and Pretests Flashcards
Justification for use of pretest
- smaller differences on pretest = less likelihood of strong initial selection biases (though without random assignment we don’t know that confounds are unrelated to outcome
- helps with stat analyses, especially if measures are reliable
Untreated Control Group Design With Pretest and Posttest Samples
- nonequivalent comparison group design
* most common of all quasi-experiments
Design Map of Untreated Control Group Design With Pretest and Posttest Samples
Diagram = NR O1 X O2
NR O1 O2
Selection-instrumentation threat
difficulty measuring certain points in a scale precisely, or having some items weighted more heavily than others): more acute if groups are unequal
Selection-regression
groups selected are not equally matched due to performance (gifted children matched with non-gifted children in other group will bias results
Selection-history
an event occurred between pre- and post- treatment that biases results
Outcome 1: Both Groups Grow Apart in the Same Direction: selection-maturation
“fan-spread model” – standardizing scores makes “fan” disappear because the variance is equalized. These effects may be spurious if groups are unequal
Outcome 2: No Change in the Control Group
treatment group may be older or other maturation issues
Outcome 3: Initial Pretest Difference Favoring the Treatment Group That Diminish Over Time
superiority of tx group is diminished or eliminated at posttest
Outcome 4: Initial Pretest Differences Favoring the Control Group Diminish Over Time
this is desirable, for instance, if a school implements a tx program for underachieving students. Outcome is subject to scaling and history threats
Outcome 5: Outcomes that Cross Over in the Direction of Relationships
amenable to causal interpretation. Power to detect a statistically reliable interaction in this type of study is low. Should not rely on research design to obtain this type of result
5 Outcome Patterns that are Plausible for Different Result Scenarios
Outcome 1: Both Groups Grow Apart in the Same Direction: selection-maturation
Outcome 2: No Change in the Control Group
Outcome 3: Initial Pretest Difference Favoring the Treatment Group That Diminish Over Time
Outcome 4: Initial Pretest Differences Favoring the Control Group Diminish Over Time
Outcome 5: Outcomes that Cross Over in the Direction of Relationships
Ways to Improve the Untreated Control Group Design With Dependent Pretest and Posttest Samples
- Using a Double Pretest
- Using Switching Replications
- Using a Reversed-Treatment Control Group
- Matching Through Cohort Controls
- Matching Through Cohort Controls by Adding Pretests
- Improving Cohort Designs With a Nonequivalent DV
- Combining Switching Replications With a Nonequivalent Control Group Design
Using a Double Pretest
understand biases pre-treatment (from pretest1 to pretest 2). Also permits assessment of a selection-maturation threat on the assumption that the rates between the 2 pretests will continue between the second pretest and posttest. This is testable ONLY for the untreated group. It can be difficult to get a 2nd pretest together
Using Switching Replications
Researcher administers tx at a later date to the group that initially served as a no treatment control. The second phase is not an exact replication. Obvious contextual differences between the groups (one receives treatment before the other, but they both receive O2 at same time). Problem: you can’t remove tx from the 1st group at O3
Using a Reversed-Treatment Control Group
X+ intended to produce an effect in one direction, while X- is intended to produce the opposite. Assumes that little historical or motivational change is taking place.
Matching Through Cohort Controls
Cohort = the successive groups that go through processes such as graduating from school, etc. cohorts can be useful as control groups if 1) one cohort experiences a given tx and ealier or later cohorts do no; 2) cohorts differ only in minor ways; 3) organizations insist that tx be given to everyone (making control impossible; 4) an org’s records can be used for constructing and comparing cohorts
Cohorts will never be as comparable as groups that are randomly assigned
Improving Cohort Controls by Adding Pretests
Compare cohort pretest means to assess for nonequivalence; Pretest increases statistical power by allowing use of within-subject error terms. Enables assessment of maturation and regression and enters into better statistical adjustment for group nonequivalence. History is a salient internal validity threat in this design
Improving Cohort Designs With a Nonequivalent DV
improve a study with a specific measure of DV
Combining Switching Replications With a Nonequivalent Control Group Design
introduce tx to part of the original control group, with other controls remaining untreated over this later time period (requires more than 2 groups). Can also reintroduce tx a second time to some of the original tx group to evaluate the benefits of additional treatment
