Test construction (Validity) Flashcards
A test is valid when it _____ what it is designed to measure
accurately measures
What are the three main forms of validity?
content,
construct, and criterion related
Content validity is of concern whenever a test has been designed to measure one or more ______
content or behaviour domains
______ validity is important when a test will be used to measure a hypothetical ______ such as achievement, motivation, intelligence, or mechanical aptitiude
construct; trait (construct)
______ validity is of interest when a test has been designed to estimate or predict performance on another measure.
criterion-related
A test has _____ validity when its items are a representative sample of the domain(s) that the test is intended to measure
content
Content validity is usually built into a test as it is being constructed through the selection of a _______ sample of items
representative
After a test has been developed, its content validity is checked by having ______ experts evaluate the test in a systematic way
subject matter
A test has construct validity when it has been shown that the test actually measures the hypothetical ______ it is intended to measure
trait (construct)
One method for assessing a test’s construct validity is to determine if the test has both _____ and ______ validity
convergent; discriminant (divergent)
When a test has _____ correlations with measures that assess the same construct, this provides evidence of the test’s _____ validity
high; convergent
When a test has ______ correlation with measures of unrelated characteristics, this indicates that the test has ______ validity
low; discriminant
The ______ matrix provides a systematic way to organize the data collected when assessing a test’s convergent and discriminant validity
mutitrait-multimethod
The matrix is a table of ______ coefficients
correlation
The matrix indicates that a test has convergent validity when when the monotrait- _______ coefficients are large and discriminant validity when the heterotrait- _______ and the _______-heteromethod coefficients are small
heteromethod; monomethod; heterotrait
Factor analysis is used to identify the factors (dimensions) that underlie the ______ among a set of tests, subtests, or test items
intercorrelations
One use of factor analysis is to determine if a test has construct validity: A test is shown to have construct validity when it has ______ correlations with the factor(s) it is expected to correlate with and ________ correlations with the factor(s) it is not expected to correlate with
high; low
In a factor matrix, the correlation between a tests and a factor is referred to as a _______. This correlation can be interpreted in terms of shared variability by squaring the correlation coefficient.
factor loading
If a test has a correlation of .50 with Factor I, this means that _____ percent of variability in test scores is explained by Factor I
25
When the identified factors are _______ (uncorrelated), a test’s communality can be calculated by summing the ______
orthogonal; squared factor loadings
If a test has a correlation of .50 with Factor I and a correlation of .20 with Factor II and the factors are uncorrelated, the test’s communality is equal to ______. This means that ____ of the variability in test scores is explained by the identified factors, while remaining variability is due to some combination of _____ and measurement error
.50 squared + .20 squared = .29;
29%; specificity
when the purpose of testing is to draw conclusions about performance on another measure, the test is referred to as the _______ and the other measure is called the _________
predictor; criterion
criterion-related validity is established by ______ scores on the test with scores on the other measure
correlating
If the resulting criterion-related _______ is sufficiently large, this demonstrates that the test has criterion-related validity
validity coefficient
There are two types of criterion-related validity: When evaluating _____ validity, the predictor is administered to a sample of examinees prior to collecting data on the criterion
predictive
Predictive validity is the appropriate type of validity when the goal of testing is to predict _____ status on the criterion
future
When evaluating ____ validity, the predictor and criterion are administered at about the same time. It is the preferred method for assessing validity when the purpose of testing is to estimate _____ status on the criterion
concurrent; current
Because a validity coefficient is never perfect (equal to +or- 1.0) there is always some ______ in prediction when a regression equation is used to predict a person’s _____ score from his/her obtained predictor score
error; criterion
The standard error of _____ is used to construct a confidence interval around an individual’s predicted ______ score
estimate; criterion
If the criterion’s standard deviation is 5 and the predictor’s validity coefficient is 60, the standard error is equal to ____; and, if a person’s predicted criterion score is 60, the 68% confidence interval is _______
4.0; 56 to 64
Even when a predictor has a large criterion-related validity coefficient, it might not be more useful for predicting criterion scores than the current method. Therefore, before using a new predictor, its ______ validity should be checked to determine if there will be an increase in ______ accuracy when the predictor is used to make selection decisions
incremental; decision-making
incremental validity is calculated by subtracting the base rate from the ______
positive hit rate
The predictor and criterion cutoff scores divide a scatterplot into four quadrants. The _____ cutoff determines id someone is a positive or negative, while the _____ cutoff determines if he/she is a true or false
predictor; criterion
The false positives are the people who score above the cutoff on the ______ but below the cutoff on the _______
predictor; criterion
To reduce the number of false positives, the predictor cutoff can be _____ and/or the criterion cutoff can be ______
raised; lowered
A predictor’s validity is limited by the reliability of both the predictor and the ______
criterion
A test’s criterion-related validity coefficient can be no greater than the _____ of the product of the reliabilities of the predictor and the ______
square root; criterion
Criterion _____ occurs when a rater’s knowledge of a person’s predictor performance affect how he/she rates the person on the criterion
contamination
When criterion contamination occurs, the validity coefficient is likely to be spuriously _____
high
Because the size of a validity coefficient is affected by group characteristics, a test that is found to be highly valid for one group might be less valid for another group. For example, the validity coefficient calculated for the initial validation sample often _____ (decreases) when the predictor is _____ -validated on another sample
shrinks; cross
An examinee’s raw test score is usually easier to interpret when it can be compared to the scores of other examinees or to a prespecified standard of performance. The former technique is referred to as ______- referenced interpretation and may ential converting an examinees’ raw score to a percentile rank or standard score
norm
A percentile rank indicates the percent of examinees in the normative sample who obtained _____ scores
lower
The transformation of raw scores to percentile ranks is _____ because the shape of the distribution of percentile ranks is always ____ regardless of the shape of the raw score distribution
nonlinear; flat (rectangular)
Standard scores anchor an examinee’s test score to those of the norm group by reporting the examinee’s score in terms of _____ from the mean
standard deviations
One of the most commonly used standard scores is the z-score, The z-score distribution has a mean of ____ and a SD of 1.0.
0
A z-core of ____ indicates that an examinee’s raw score is one standard deviation below the mean
-1.0
Scores can also be interpreted in terms of a prespecified standard of performance. ______ scores are one means of criterion-referenced interpretations
Percentage
percentage scores indicate the proportion of the test content (e.g., percent of test items) that examinees answered correctly. An ______ table makes it possible to use an examinee’s predictor (test) score to estimate the probability that he/she will attain different scores on a _______
expectancy; criterion
Scores on objective tests are sometimes _____ to ensure that examinees don’t benefit from random guessing
corrected for guessing
