Test Construction: Validity Flashcards
Validity: Overview and 3 types
Validity = Accuracy = test measures what it is intended to measure
Content validity
Construct validity
Criterion-Related validity
Content Validity
Test items are representative of target domain
i.e. the sample content was adequate
Content Validity vs. Face Validity
Content = test items adequately sampled the relevant domain
Face = test “looks like” it measures what it is intended to measure
Content Validity Ratio (CVR)
Ratio of subject matter expert raters to total number of raters
Construct Validity
Test measures the hypothetical trait (construct) it is intended to measure
Construct Validity: Various Methods of measurement
Internal Consistency
Group Differences
Hypothesis Testing
Convergent and Discriminant Validity
Factorial Validity
Construct Validity: Convergent and Discriminant Validity, Overview
Compare scores on different measures/tests that:
Aim to test the same trait (convergent)
Do not aim to test the same trait (discriminant/divergent)
Construct Validity: High Convergent Validity
High correlations with measures of the same related traits
e.g. BAI & GAD7
Construct Validity: High Discriminant validity
Low correlations with measures of unrelated characteristics
e.g. BAI & SASSI
Construct Validity: The Multitrait-Multimethod Matrix
data organization into mono/hetero traits and methods
used to evaluate convergent and discriminant validity
Construct Validity: Factor Analysis
Factor analysis identifies the minimum number of common factors (dimensions) required to account for inter-correlations among tests/subtests/items
Factor Analysis: Comunality
Variability in scores accounted for by all identified factors, aka, Common Variance
Criterion-Related Validity: Purpose
To estimate examinee’s standing or performance on another measure
e.g.: Employee selection test (predictor) used to estimate how well an applicant will do on a measure of job performance (external criterion) after they are hired
The Two Forms of Criterion-Related Validity
Concurrent validity
Predictive validity
Concurrent Validity
Criterion data are collected prior to/about same time as data on the predictor
Predictor is used to estimate current status
e.g. current job performance is acceptable immediately
Predictive validity
Criterion data is measured sometime after the predictor has been administered
e.g. predict future job performance
“Come together to build”
Converge to Construct
Construct Validity associated with convergent and discriminant validity
Standard Error of Estimate (SEE)
Error in regression prediction for criterion- related validity
Used to create confidence interval around an estimated (predicted score)
SEM vs SEE
SEM = used to create CI for test score
SEE = used to crate CI for predicted score
“SEE the future to confidently predict it”
Incremental Validity: Exam Takeaways
Predictor determines a person is a positive or negative (score on first measure)
Criterion determines if they are a “true” or a “false” (comparison of second measure
Criterion-related Predictive validity accuracy indexes (6)
Sensitivity and Specificity
Positive and Negative predictive value
Positive and Negative likelihood ratio
Criterion-Related Validity: Sensitivity and Specificity
Examinees are known to have or not have the disorder of interest
Sensitivity: percent accurately identified to have the disorder by predictor
Specificity: percent accurately identified to not have the disorder by the predictor
Criterion-Related Validity: Sensitivity and Specificity, MNEMONIC
“Please don’t ask me about my disorder, it’s a sensitive topic, thanks”
“I would like to specify that I do NOT have the disorder!”
How Reliability is necessary but not sufficient for Validity
Low reliability prevents high content, construct or criterion-related validity
*But high reliability DOES NOT guarantee validity
