Validity Flashcards
Questions addressed by internal validity
To what extent can the intervention, rather than extraneous influences, be considered to account for the results, changes, or group differences
Questions addressed by external validity
To what extent can the results be generalized or extended to ppl, settings, times, measures, characteristics other than those in this particular experimental arrangement
Questions addressed by Construct validity
what specific aspects of the intervention or arrangement was the causal agent - what is the conceptual basis (construct) underlying the effect?
Questions addressed by Statistical conclusion validity
To what extent is a relation shown, demonstrated, or evident, and how well can the investigation detect effects if they exist
Threats to internal validity
History Maturation Testing Instrumentation Statistical regression Selection bias Attrition Combinations of selection and other threats Diffusion or imitation of treatment Special treatment or reactions of controls
History
Threat to internal validity
effects of events common to all subjects that might alter performance and be mistaken for the effects resulting from the intervention
Usually outside of the experiment eg natural disaster, 911, etc.
Within experiment eg fire drills, power blackout
Maturation
Threat to internal validity
Changes over time
Specific event and also processes w/in subjects
Eg growing older, wiser, stronger, bored, tired
Only problem if design cannot separate the effects of maturational changes from the intervention
Testing
Threat to internal validity
Effects that taking a test one time may have on subsequent performance of the test
Pre- and post-intervention tests - to evaluate how much an indiv improves or deteriorates over time
Practice or familiarity effects
Group that receives repeated testing w/out intervention eg no tx control grp can help rule out testing as an explanation
No-tx grp would be expected to show effects of testing
Tx group shows impact of intervention over and above any effects of testing
Instrumentation
Threat to internal validity
Changes in measuring instrument or procedure over time
Can greatly affect substantive conclusions about changes over time
Longitudinal data = subject to instrumentation
Not a problem when standardized tests or automated devices used
Possible that items remain the same but have a different meaning bc of social contexts
Response shift
Response shift
Instrumentation
Refers to a shift in a person’s internal standards of measurement
Shift reflects changes in values, attitudes, perspective, criteria etc. that lead to evaluation of the same or similar situations bxs, states, in a different way
Statistical regression
Threat to internal validity
Tendency for extreme scores to revert (regress) twd the mean when measurement devices is re-administered
Scores tend to be less extreme at second testing - less extreme score = closer to the mean
Evident in no-tx grp
Provide basis for evaluating incremental changes assoc w tx
Threat when change due to intervention cannot be distinguished from effect of scores reverting to the mean
Attrition
Threat to internal validity
Loss of subjects
Intervention studies and longitudinal studies
Most lost early
40-60% drop out early
Changes in overall performance may be bc loss of subjs who scored in a particular direction
Two or more grps - even if # of dropouts are the same, still a threat if believe characteristics of subjs who dropped out differ b/w groups
Remaining subjs can’t be assumed to represent original sample
Cannot assume grps are equivalent
Differential attrition = likely if conditions are differentially attractive or effective
Combinations of selection and other threats
Threat to internal validity
Whenever threats to internal validity vary for the different groups within a study - how the threats interact
Selection x History
Combination of selection and other threats
One group has historical experience that other group did not have - and experience may plausibly explain the results
Threat (hx) was selective and applies to only one (or some but not all) of the groups
Two conditions for combination of selection and other threats to pose a threat to validity
- Differences systematically vary b/w experimental grp
2. Difference could plausibly explain results
Diffusion or imitation of tx
Threat to internal validity
Intervention given to one grp accidentally provided to control group
Effect = attenuate effects of tx and alter what investigator concludes about the efficacy of tx
Effect equalizes performance - reduces or distorts effects of the intervention
No tx “spreads”
Often underestimate or inaccurately estimate effects of tx
Special tx or reactions of controls
Threat to internal validity
Control group does not receive intervention but may receive other services such as money, more monitoring of their well-being, or special privileges
May be considered an intervention in its own right
Absence of tx may lead to special performance - may react in ways that obscure the differences b/w tx and no tx (eg motivated to perform “just as well”
