Ch11 Flashcards
Six threats to internal validity in one-group pretest/posttest designs
- Maturation threats
- History threats
- Regression threats
- Attrition threats
- Testing threats
- Instrumentation threats
(also combined threats)
Maturation threats + prevention
change in behavior that happens spontaneously over time
To prevent:
add comparison group-
if treatment group improves significantly more than comparison group, subtract the effect of maturation to see results of the treatment alone
History threats + prevention
when an external, “historical,” event affects members of the treatment group at the same time as the treatment
To prevent: comparison group-
if both groups experienced the same external circumstances, the treatment group should still improve more than the comparison group
Regression threat (regression to the mean) + prevention
- statistical threat in which an extreme mean(avg) at time one regresses back to a more avg mean at time 2
- Only occur when there’s an extreme pretest score
- To prevent: use comparison groups-
regression might be present if one group starts off with extreme scores, or if both start with extreme scores and end with moderate scores
Attrition threat + prevention
Attrition threat: when there’s a reduction in # of participants between the pretest and posttest (people drop out)
- Only a problem if systematic- depends on what the scores are- closer to the mean is better
Prevent by:
- removing the participants who dropped out from pretest avg
- Look at pretest score of dropouts- if their scores are extreme, more likely to threaten internal validity
Testing threat + prevention
type of order effect- change in participants as a result of experiencing the dependent variable ( a test) more than once
To prevent:
- Posttest only design
- Different versions of test for pretest and posttest
- Comparison group- if larger change in treatment group, then testing threat can be ruled out
Instrumentation threats (instrumentation decay) + prevention
Instrumentation threats (instrumentation decay): when a measuring instrument changes over time
- can be observer/coder
Prevent:
- Posttest only
- Make sure pretest and posttest forms are equivalent
Retrain observers throughout study (use clear coding manuals) - Counterbalance pretest and posttest forms
combined threats
Selection-history threats: an outside event or factor systematically affects participants at one level of the IV
Selection-attrition threat: participants in one experimental group experience attrition
3 Potential Internal validity threats to any study-
Observer bias
Demand characteristics
Placebo effect
observer bias + prevention
researcher’s expectations influence their interpretation of results
Prevent by: double blind study or masked design
demand characteristics
participants change their behavior after figuring out the study’s purpose
Prevent by: double blind study or masked design
Placebo effects + prevention
when people receive a treatment (inert or real) and improve only because they believe it’s working
- Double-blind placebo control study: patients and those treating don’t know who receives the real treatment
prevention:
Introduce third group (comparison group) that receives no treatment or placebo, if there’s a placebo effect, the placebo group should improve more than the no treatment group
Null effects
independent variable didn’t change the dependent variable, no covariance
types of null effects
- Not enough btwn groups difference
- Within-groups variability obscured group differences
- No difference
null effect: Not enough btwn-groups difference types
- Weak manipulations
- insensitive measures
- Ceiling and floor effects
- Reverse design confound
Weak manipulations
type of null effect: Not enough btwn-groups difference
not enough of an increase in levels of IV to make a difference (in the dependent variable)
Insensitive measures
type of null effect: Not enough btwn-groups difference
Insensitive measures: dependent variable not operationalized with enough sensitivity
Needs detailed, quantitative increments instead of 2-3 levels
Ceiling and Floor effects:
When participants scores on the dependent variable are clustered on either the high or low end
-Can be result of problematic independent variable (weak manipulation), or a problematic dependent variable (insensitive measures)
Design Confounds acting in reverse
- Want to be sure if there aren’t changes, it’s really because there isn’t an effect of the IV on the dependent variable
-Can counteract true effects of an IV
noise (error variance, unsystematic variance)
- unsystematic variability within each group
- obscures differences btwn each group b/c there’s more overlap btwn the groups
- A Statistical validity problem- the more overlap btwn groups, the smaller the effect size, less likely to be statistically significant
Types of within-group differences
- Measurement error
- Individual differences
- Situation noise
Measurement error + prevention
Measurement error: a human or instrument factor can inflate or deflate a person’s true score on the dependent variable
- DV score = participants true score +/- random error of measurement
To reduce:
- Use reliable (internal, interrater, test-retest) , precise measurement
- Measure more instances
(Measure larger sample- random errors will cancel each other out)
Individual differences + prevention
spread out scores within each group, obscuring btwn-group differences
- Can be a problem in independent-groups design
To reduce:
- Change to within-groups (or matched groups)
-Add more participants
Situation noise + prevention
any kind of external distraction that causes variability within-groups that obscure btwn-group differences
Minimize by controlling surrounding of an experiment
power + what kind of validity
likelihood that a study has statistically significant results when the IV actually has an effect on the DV
- aspect of statistical validity
What kind of studies are more likely to detect true differences
Studies with a lot of power are more likely to detect true differences
- Can detect even small effects
-Strong manipulation is similar to having large effects
