Study Guide 12: Criterion Related Validity

Question 1

Area under the Curve (AUC)

Answer 1

Graph of true positive rate (sensitivity) vs. False positive rate (1-specificity). Interpreted as an estimate of the probability that a randomly chosen depressed person will have a higher depression score than a randomly chosen non-depressed person. AUC of 0.5 = line of no information, recommend: AUC ≥ 0.8.

Question 2

Base rate

Answer 2

Proportion of current employees who are successful, but have not been selected using any test

Question 3

Binomial effect size display (BESD)

Answer 3

a tool for reporting the magnitude of effect size
Concurrent evidence: validity evidence demonstrating the degree to which criterion and test that are measured at the same time correlate.

Question 4

Criterion (or criterion variable):

Answer 4

a measure of some attribute or outcome that is of primary interest.

Question 5

Criterion-related evidence

Answer 5

validity evidence demonstrating the correlation between performance on the test with performance on relevant criterion.

Question 6

Negative predictive value (NPV):

Answer 6

TN (TN + TP) - % of people who are truly not depressed (according to the criterion) out of those that the scale identified as non-depressed

Question 7

Positive predictive value (PPV)

Answer 7

TP/(FP + TP) - % of individuals who are truly depressed (according to the criterion) of those that the scale identified as depressed

• influenced by prevalence of X in population
• As prevalence , PPV
• small differences in  specificity levels can strongly influence  PPVs
• changes in sensitivity have little impact on PPV

Question 8

Predictive evidence:

Answer 8

validity evidence demonstrating the degree to which criterion and test that are measured at different times correlate.

Question 9

Prevalence:

Answer 9

the proportion of the population or sample found to have the condition.

Question 10

Receiver operating characteristic (ROC) curve

Answer 10

graph sensitivity and specificity fro all of the possible scores of a scale by plotting true positives (sensitivity) versus false positives (1 – specificity).

Question 11

Selection ratio:

Answer 11

of applicants hired/# of applicants selected. All else being equal, the smaller the selection ratio, the more useful the test.

Question 12

Sensitivity:

Answer 12

TP/(FN + TP). The proportion of actual positives which are correctly identified as such. How sensitive a measure is at detecting the condition

Question 13

Specificity:

Answer 13

TN/(TN + FP). The proportion of actual negatives which are correctly identified. How specific a measure is at ruling out those without the condition.

Question 14

Standard error of estima

Answer 14

an estimate of the amount of error to be expected in the predicted criterion score, speaks to the validity of the measure. Whereas, SEM speaks to the reliability of the measure.

Question 15

Taylor-Russell tables

Answer 15

table demonstrating the connection between the validity of a test and the likelihood of the test resulting in a successful selection of a candidate.

Question 16

Utility analysis

Answer 16

frames validity in terms of a cost versus benefit analysis of test use. Is a test worth using? Does it outweigh the costs?

Question 17

Validity generalization

Answer 17

Type of meta-analysis that seeks to evaluate validity coefficients across a group of studies. Can reveal a) average level of validity coefficient found across a number of studies, b) degree of variability among smaller studies, and c) source of variability across smaller studies.