Test construction (Validity)

Question 1

A test is valid when it _____ what it is designed to measure

Answer 1

accurately measures

Question 2

What are the three main forms of validity?

Answer 2

content,

construct, and criterion related

Question 3

Content validity is of concern whenever a test has been designed to measure one or more ______

Answer 3

content or behaviour domains

Question 4

______ validity is important when a test will be used to measure a hypothetical ______ such as achievement, motivation, intelligence, or mechanical aptitiude

Answer 4

construct; trait (construct)

Question 5

______ validity is of interest when a test has been designed to estimate or predict performance on another measure.

Answer 5

criterion-related

Question 6

A test has _____ validity when its items are a representative sample of the domain(s) that the test is intended to measure

Answer 6

content

Question 7

Content validity is usually built into a test as it is being constructed through the selection of a _______ sample of items

Answer 7

representative

Question 8

After a test has been developed, its content validity is checked by having ______ experts evaluate the test in a systematic way

Answer 8

subject matter

Question 9

A test has construct validity when it has been shown that the test actually measures the hypothetical ______ it is intended to measure

Answer 9

trait (construct)

Question 10

One method for assessing a test’s construct validity is to determine if the test has both _____ and ______ validity

Answer 10

convergent; discriminant (divergent)

Question 11

When a test has _____ correlations with measures that assess the same construct, this provides evidence of the test’s _____ validity

Answer 11

high; convergent

Question 12

When a test has ______ correlation with measures of unrelated characteristics, this indicates that the test has ______ validity

Answer 12

low; discriminant

Question 13

The ______ matrix provides a systematic way to organize the data collected when assessing a test’s convergent and discriminant validity

Answer 13

mutitrait-multimethod

Question 14

The matrix is a table of ______ coefficients

Answer 14

correlation

Question 15

The matrix indicates that a test has convergent validity when when the monotrait- _______ coefficients are large and discriminant validity when the heterotrait- _______ and the _______-heteromethod coefficients are small

Answer 15

heteromethod; monomethod; heterotrait

Question 16

Factor analysis is used to identify the factors (dimensions) that underlie the ______ among a set of tests, subtests, or test items

Answer 16

intercorrelations

Question 17

One use of factor analysis is to determine if a test has construct validity: A test is shown to have construct validity when it has ______ correlations with the factor(s) it is expected to correlate with and ________ correlations with the factor(s) it is not expected to correlate with

Answer 17

high; low

Question 18

In a factor matrix, the correlation between a tests and a factor is referred to as a _______. This correlation can be interpreted in terms of shared variability by squaring the correlation coefficient.

Answer 18

factor loading

Question 19

If a test has a correlation of .50 with Factor I, this means that _____ percent of variability in test scores is explained by Factor I

Answer 19

25

Question 20

When the identified factors are _______ (uncorrelated), a test’s communality can be calculated by summing the ______

Answer 20

orthogonal; squared factor loadings

Question 21

If a test has a correlation of .50 with Factor I and a correlation of .20 with Factor II and the factors are uncorrelated, the test’s communality is equal to ______. This means that ____ of the variability in test scores is explained by the identified factors, while remaining variability is due to some combination of _____ and measurement error

Answer 21

.50 squared + .20 squared = .29;

29%; specificity

Question 22

when the purpose of testing is to draw conclusions about performance on another measure, the test is referred to as the _______ and the other measure is called the _________

Answer 22

predictor; criterion

Question 23

criterion-related validity is established by ______ scores on the test with scores on the other measure

Answer 23

correlating

Question 24

If the resulting criterion-related _______ is sufficiently large, this demonstrates that the test has criterion-related validity

Answer 24

validity coefficient

Question 25

There are two types of criterion-related validity: When evaluating _____ validity, the predictor is administered to a sample of examinees prior to collecting data on the criterion

Answer 25

predictive

Question 26

Predictive validity is the appropriate type of validity when the goal of testing is to predict _____ status on the criterion

Answer 26

future

Question 27

When evaluating ____ validity, the predictor and criterion are administered at about the same time. It is the preferred method for assessing validity when the purpose of testing is to estimate _____ status on the criterion

Answer 27

concurrent; current

Question 28

Because a validity coefficient is never perfect (equal to +or- 1.0) there is always some ______ in prediction when a regression equation is used to predict a person's _____ score from his/her obtained predictor score

Answer 28

error; criterion

Question 29

The standard error of _____ is used to construct a confidence interval around an individual's predicted ______ score

Answer 29

estimate; criterion

Question 30

If the criterion's standard deviation is 5 and the predictor's validity coefficient is 60, the standard error is equal to ____; and, if a person's predicted criterion score is 60, the 68% confidence interval is _______

Answer 30

4.0; 56 to 64

Question 31

Even when a predictor has a large criterion-related validity coefficient, it might not be more useful for predicting criterion scores than the current method. Therefore, before using a new predictor, its ______ validity should be checked to determine if there will be an increase in ______ accuracy when the predictor is used to make selection decisions

Answer 31

incremental; decision-making

Question 32

incremental validity is calculated by subtracting the base rate from the ______

Answer 32

positive hit rate

Question 33

The predictor and criterion cutoff scores divide a scatterplot into four quadrants. The _____ cutoff determines id someone is a positive or negative, while the _____ cutoff determines if he/she is a true or false

Answer 33

predictor; criterion

Question 34

The false positives are the people who score above the cutoff on the ______ but below the cutoff on the _______

Answer 34

predictor; criterion

Question 35

To reduce the number of false positives, the predictor cutoff can be _____ and/or the criterion cutoff can be ______

Answer 35

raised; lowered

Question 36

A predictor's validity is limited by the reliability of both the predictor and the ______

Answer 36

criterion

Question 37

A test's criterion-related validity coefficient can be no greater than the _____ of the product of the reliabilities of the predictor and the ______

Answer 37

square root; criterion

Question 38

Criterion _____ occurs when a rater's knowledge of a person's predictor performance affect how he/she rates the person on the criterion

Answer 38

contamination

Question 39

When criterion contamination occurs, the validity coefficient is likely to be spuriously _____

Answer 39

high

Question 40

Because the size of a validity coefficient is affected by group characteristics, a test that is found to be highly valid for one group might be less valid for another group. For example, the validity coefficient calculated for the initial validation sample often _____ (decreases) when the predictor is _____ -validated on another sample

Answer 40

shrinks; cross

Question 41

An examinee's raw test score is usually easier to interpret when it can be compared to the scores of other examinees or to a prespecified standard of performance. The former technique is referred to as ______- referenced interpretation and may ential converting an examinees' raw score to a percentile rank or standard score

Answer 41

norm

Question 42

A percentile rank indicates the percent of examinees in the normative sample who obtained _____ scores

Answer 42

lower

Question 43

The transformation of raw scores to percentile ranks is _____ because the shape of the distribution of percentile ranks is always ____ regardless of the shape of the raw score distribution

Answer 43

nonlinear; flat (rectangular)

Question 44

Standard scores anchor an examinee's test score to those of the norm group by reporting the examinee's score in terms of _____ from the mean

Answer 44

standard deviations

Question 45

One of the most commonly used standard scores is the z-score, The z-score distribution has a mean of ____ and a SD of 1.0.

Answer 45

0

Question 46

A z-core of ____ indicates that an examinee's raw score is one standard deviation below the mean

Answer 46

-1.0

Question 47

Scores can also be interpreted in terms of a prespecified standard of performance. ______ scores are one means of criterion-referenced interpretations

Answer 47

Percentage

Question 48

percentage scores indicate the proportion of the test content (e.g., percent of test items) that examinees answered correctly. An ______ table makes it possible to use an examinee's predictor (test) score to estimate the probability that he/she will attain different scores on a _______

Answer 48

expectancy; criterion

Question 49

Scores on objective tests are sometimes _____ to ensure that examinees don't benefit from random guessing

Answer 49

corrected for guessing