Test Construction: Validity Flashcards
Validity: Overview and 3 types
Validity = Accuracy = test measures what it is intended to measure
Content validity
Construct validity
Criterion-Related validity
Content Validity
Test items are representative of target domain
i.e. the sample content was adequate
Content Validity vs. Face Validity
Content = test items adequately sampled the relevant domain
Face = test “looks like” it measures what it is intended to measure
Content Validity Ratio (CVR)
Ratio of subject matter expert raters to total number of raters
Construct Validity
Test measures the hypothetical trait (construct) it is intended to measure
Construct Validity: Various Methods of measurement
Internal Consistency
Group Differences
Hypothesis Testing
Convergent and Discriminant Validity
Factorial Validity
Construct Validity: Convergent and Discriminant Validity, Overview
Compare scores on different measures/tests that:
Aim to test the same trait (convergent)
Do not aim to test the same trait (discriminant/divergent)
Construct Validity: High Convergent Validity
High correlations with measures of the same related traits
e.g. BAI & GAD7
Construct Validity: High Discriminant validity
Low correlations with measures of unrelated characteristics
e.g. BAI & SASSI
Construct Validity: The Multitrait-Multimethod Matrix
data organization into mono/hetero traits and methods
used to evaluate convergent and discriminant validity
Construct Validity: Factor Analysis
Factor analysis identifies the minimum number of common factors (dimensions) required to account for inter-correlations among tests/subtests/items
Factor Analysis: Comunality
Variability in scores accounted for by all identified factors, aka, Common Variance
Criterion-Related Validity: Purpose
To estimate examinee’s standing or performance on another measure
e.g.: Employee selection test (predictor) used to estimate how well an applicant will do on a measure of job performance (external criterion) after they are hired
The Two Forms of Criterion-Related Validity
Concurrent validity
Predictive validity
Concurrent Validity
Criterion data are collected prior to/about same time as data on the predictor
Predictor is used to estimate current status
e.g. current job performance is acceptable immediately
Predictive validity
Criterion data is measured sometime after the predictor has been administered
e.g. predict future job performance
“Come together to build”
Converge to Construct
Construct Validity associated with convergent and discriminant validity
Standard Error of Estimate (SEE)
Error in regression prediction for criterion- related validity
Used to create confidence interval around an estimated (predicted score)
SEM vs SEE
SEM = used to create CI for test score
SEE = used to crate CI for predicted score
“SEE the future to confidently predict it”
Incremental Validity: Exam Takeaways
Predictor determines a person is a positive or negative (score on first measure)
Criterion determines if they are a “true” or a “false” (comparison of second measure
Criterion-related Predictive validity accuracy indexes (6)
Sensitivity and Specificity
Positive and Negative predictive value
Positive and Negative likelihood ratio
Criterion-Related Validity: Sensitivity and Specificity
Examinees are known to have or not have the disorder of interest
Sensitivity: percent accurately identified to have the disorder by predictor
Specificity: percent accurately identified to not have the disorder by the predictor
Criterion-Related Validity: Sensitivity and Specificity, MNEMONIC
“Please don’t ask me about my disorder, it’s a sensitive topic, thanks”
“I would like to specify that I do NOT have the disorder!”
How Reliability is necessary but not sufficient for Validity
Low reliability prevents high content, construct or criterion-related validity
*But high reliability DOES NOT guarantee validity
Criterion-Related Validity: Correction for Attenuation
Estimate of what the predictor’s validity coefficient would be if the predictor and/or the criteria were perfectly reliable
“A 10 out of 10 is perfect prediction”
[a 10 = ATTENuation]
Criterion-Related Validity: Criterion Contamination
When criterion scores are affected by knowledge of predictor scores
Construct Validity: Factorial Validity
High correlations with related factors (convergent)
Low correlations with unrelated factors (discriminant)
Multitrait-Multimethod Combinations:
High Convergent Validity
Large Monotrait-Heteromethod
CV = LMH
Multitrait-Multimethod Combinations:
High Discriminant Validity
Small Heterotrait-Monomethod
DV = SHM
Construct
Abstract characteristic that cannot be observed directly, but must be inferred by observing his effects.
E.g. intelligence, self-esteem, neuroticism
Factor Analysis: Comunality Calculation
Sum of squares of each factor loading
e.g.:
For Test A, Factor I = .80, Factor II = .20
Comunality = .64 (.80²) + .04 (.20²) = .68
Two Components of Reliability: Comunality and Specificity
Reliability = Comunality + Specificity
Comunality: variability of scores due to factors in common with other tests
Specificity: Unique variability not explained by factor analysis
Comunality as lower-limit estimate of reliability
Comunality is only one part of true score variability (other part is specificity)
Therefore the reliability coefficient can never be lower than comunality
