Construct and statistical concl validity Flashcards
Construct validity
What is the intervention and why did it produce the effect?
Addresses the presumed cause or explanation of the causal relation b/w the intervention and outcome
Discuss after the internal validity is established
Key question for construct validity
Is the reason for the relation b/w the intervention and bx change due to the construct (explanation, interpretation) given by the investigator?
Features within the experiment that can interfere w interpretation of the results
Confounds
Possibility that a specific factor varied or co-varied w the intervention
Could be responsible for the results
Questions concerning construct validity
What is the intervention?
Does the intervention incl other components than those discussed by the investigator?
Features associated w the intervention that interfere w drawing inferences about the basis for the difference b/w groups referred to as..
Threats to construct validity
Types of threats to construct validity
Attention and contact w clients
Single operations and narrow stimulus sampling
Experimenter expectancies
Cues of the experimental situation
Attention and contact w the clients
Attention and contact accorded to the experimental grp, or differential attention across experimental and control grps may be the basis for differences observered
Intervention assoc w all aspects incl administration
Threat when attention, contact w clients, and their expectations might plausibly account for findings and were not controlled for by the design
Placebo effect and construct validity
Expectancies for improvement generated by placebos must be controlled if an investigator wishes to draw conclusions about specific effects of the interventions
To examine basis for effects (construct validity) - must incl a third grp that received a placebo on the same schedule of administration
Attn is parsimonious bc..
construct provides and explanation of the effects of many studies in which tx is better than a control grp
Single operations and narrow stimulus sampling
Features that the investigator considers irrelevant to the study, but these features may introduce ambiguity in interpreting the findings
Single operations and narrow stimulus sampling - key question for construct validity
Whether the intervention is responsible for the outcome
OR was it some seemingly irrelevant feature with which the intervention was associated
Key = being unable to separate the constructs of interest from the conditions of its delivery
Experimenter expectancies
When experimental expectancies provide a plausible rival interpretation of the effects otherwise attributed to the experimental manipulation or intervention
Statistical concl validity
Refers to facets of the quantitative evaluation that infl the concls we reach about the experimental condition and its effect
Statistical evaluation viewed from 2 standpoints
- Understanding the tests themselves and their bases - what the tests accomplish and the formulae and derivations of the tests
- (Complementary) Computational aspects of the tests - application of the tests to the data sets, use of software, and interpretations of the findings
Threats to statistical concl validity refer to…
Facets of the results and statistical evaluation that can obscure interpretation of the experiment
Ho specifies…
That there are “no” differences b/w groups
Reject the null hypothesis when…
There is a statistically significant difference found
Accept a null hypothesis when…
No statistical difference is found
Threats to statistical concl validity - types
Low statistical power Variability in procedures Subj heterogeneity Unreliability of measures Multiple comparisons and error rates
Low statistical power
Threat to statistical concl validity
Weak power or low probability of detecting a difference if one exists
May concl that no difference exists when there is a difference
How to increase power?
Increase sample size
Variability in procedures
Threat to statistical concl validity
Sometimes can tell by the design that there will be relatively high levels of variability and great difficulty in demonstrating a difference b/w conditions
Higher variability = lower effect size
Subj heterogeneity
Threat to statistical concl validity
Greater heterogeneity = less likelihood of detecting a difference b/w conditions
Greater variability in reactions to measures
Addressing subj heterogeneity
- Chose homogenous samples
2. Chose heterogenous samples but ensure that the impact of characteristics can be evaluated by the design
Unreliability of measures
Likely to obtain lower effect sizes
Reliability = matter of degree and refers to the extent of the variability in responding
Unreliable measure = greater portion of subj’s score due to unsystematic, random variation
Multiple comparisons and error rates
The more tests that are perform = the more likely a difference will be found even if no true differences exist
TYPE I ERROR
Risk = specified by alpha
Multiple comparisons increase alpha
Experiment-wise error rate
Risk across several statistical tests
