Test Validity - Criterion-Related Validity

Question 1

A test’s _____ validity is of interest whenever scores on the test will be used to predict or estimate scores on another measure.

Answer 1

criterion-related

Question 2

What are the two types of criterion-related validity?

Answer 2

1. Concurrent - evaluated by obtaining scores on the predictor and criterion at about the same time (current criterion).
2. Predictive - evaluated by obtaining scores on the predictor before obtaining scores on the criterion (future criterion)

Question 3

However, whenever the criterion-related validity coefficient is not +1 or -1, there will be some error in prediction, and a person’s actual criterion score may be higher or lower than his/her predicted score. Consequently, the _____ is often used to construct a confidence interval around a person’s predicted criterion score.

Answer 3

standard error of estimate

A 68% confidence interval is constructed by adding and subtracting one standard error of estimate to and from the person’s predicted criterion score, a 95% confidence interval is constructed by adding and subtracting two standard errors, and a 99% confidence interval is constructed by adding and subtracting three standard errors.

Question 4

How do you find the standard error of the estimate?

Answer 4

SD of Criterion measure x √(1-r2)

If SD of criterion measure = 5
If criterion r = .60

5 x √1-.60-squared
5 x √1-.36
5 x √.64
5 x .8 = 4

Question 5

_____ refers to the increase in the accuracy of predictions about criterion performance that occurs by adding a new predictor to the current methods used to make predictions

Answer 5

Incremental validity

Question 6

In criterion validity studies _____ are those measured who obtained high scores on the predictor and low scores on the criterion.

Answer 6

False positives

Question 7

In criterion validity studies ____ are those measured who obtained high scores on the predictor and criterion.

Answer 7

True positives

Question 8

In criterion validity studies ____ are those obtained low scores on the predictor and criterion.

Answer 8

True negatives

Question 9

In criterion validity studies ____ are those obtained ow scores on the predictor and high scores on the criterion.

Answer 9

False negatives

Question 10

In criterion validity studies if one raises the predictor score, it results in _____ people selected, _____ positives, and _____ negatives.

Answer 10

Fewer; fewer; more

Question 11

In criterion validity studies the base rate is obtained by:

Answer 11

Taking the number of people in the study obtaining high scores on criterion (regardless of predictor scores) divided by number of people in the study.

Question 12

In criterion validity studies the positive hit rate is obtained by:

Answer 12

proportion of people in the study who would be selected using their scores on the new predictor and obtained high scores on the criterion. It’s calculated by dividing the number of true positives by the total number of positives. (True positives and False).

Question 13

To calculate incremental validity, the _____ is subtracted from the _____

Answer 13

Base rate; positive hit rate

Question 14

In criterion validity studies if one lowers the predictor score, it results in _____ people selected, _____ positives, and _____ negatives.

Answer 14

More, more, fewer

Question 15

In terms of diagnostic efficacy, ______ is the proportion of people with the disorder who are identified by the test as having what you’re measuring. It’s calculated by dividing the true positives by the true positives plus the false negatives (TP/TP + FN).

Answer 15

Sensitivity (mnemonic - sensing what you need)

Question 16

In terms of diagnostic efficacy, _____ is the proportion of people without the disorder who are identified by the test as not having what you’re measuring. It’s calculated by dividing the true negatives by the true negatives plus the false positives (TN/TN + FP).

Answer 16

specificity

Question 17

In terms of diagnostic efficacy, _____ is also known as overall correct classification rate and is the proportion of people who are correctly categorized by the test. It’s calculated by dividing the true positives and true negatives by the sample size (TP + TN)/(TP + TN + FP +FN).

Answer 17

The hit rate

Question 18

In terms of diagnostic efficacy, _____ indicates the probability that a person who tests positive for a disorder actually has the disorder. It’s calculated by dividing the number of true positives by the total number of positives (TP/(TP + FP).

Answer 18

The positive predictive value

Question 19

In terms of diagnostic efficacy, _____ indicates the probability that a person who tests negative for a disorder does not actually have the disorder. It’s calculated by dividing the number of true negatives by the total number of negatives (TN/(TN + FN).

Answer 19

The negative predictive value

Question 20

A predictor’s reliability always places a ceiling on its validity – and, for criterion-related validity, a formula defines this relationship. What is this formula?

Answer 20

The formula states the maximum criterion-related validity coefficient for a predictor is equal to the square root of its reliability coefficient. As an example, when a predictor’s reliability coefficient is .81, its criterion-related validity coefficient can be no larger than the square root of .81, which is .90.