Hunsley & Meyer; Incremental validity Flashcards
What is incremental validity used for?
To determine whether a new source of data will increase the predictive ability of a criterion beyond that provided by the existing method of assessment
How is incremental validity context dependent?
An assessment may be incrementally valid for some applications but not for others
Why must one consider the cost-benefit in tandem with incremental validity?
Because the incremental validity of one test over other readily available methods may be so small that it may not be worth the financial cost
In what way is incremental validity somewhat dependent on base rates and selection ratios?
situations involving very low base rates (i.e., very rare or very common events) may allow incremental validity to be demonstrated, yet, the increment is likely to be extremely small
How is incremental validity usually studied?
Using correlational research designs
What are two methods for translating incremental validity into a measure of clinical utility?
- The shared over simple effects approach (SOS)
2. the semipartial r approach
When considering incremental validity, why must we consider “convergent” data (i.e., data sources based on similar sources/forms of information)?
Because if the data sources are based on very similar sources or form of information, “convergent” data will not demonstrate incremental validity
The accuracy of conclusions only improves when data sources provide non-redundant information (i.e., data sources are non-overlapping)
Give some examples of incremental validity
Personnel psychology studies incremental validity (IV) frequently to examine issues such as employment interview evaluations, informant ratings to assess job performance, and personnel selection strategies
How does personnel psychology conceptualize incremental validity?
Personnel psychology generalizes the % increase in incremental validity as equivalent to the % increase in the utility of the test
Information from a new source of data (such as a test scale or an observational coding system) is examined in terms of its contribution to improving on the prediction of a clinically relevant criterion (e.g., diagnosis, adjustment, treatment attendance, or treatment outcome). How is prediction assessed?
Prediction is assessed by the extent to which the sources of data can account for the variance in the criterion
When examining incremental validity for a new source of data, what other forms of validity may also be provided?
May provide evidence for concurrent validity or predictive validity
What are the 3 research applications for incremental validity?
- validation for new measures
- addition of a testing instrument
- test-informed clinical results (e.g., the addition of MMPI to medical residents’ diagnostic decisions)
What does incremental validity in the context of test-informed clinical inference relate to?
how clinicians synthesize test results, not the test content itself
Example: medical residents’ diagnostic decisions were more accurate when informed by MMPI test results than in the absence of MMPI test results → MMPI test results improve the predictive ability of medical diagnostics, demonstrating incremental validity of the MMPI
Why is it important to consider reliability alongside incremental validity?
A measure may have IV over other measures simply because it is more reliable than other measures
Criterion contamination
when the results of the to-be-validated test scale inform or influence the criterion that are supposed to validate the scale
E.g., if IQ scores are supposed to predict teacher ratings of IQ, but the teacher ratings are completed after the teachers have SEEN the results of the IQ test
