Validity

Question 1

____ refers to a test’s accuracy: A test is valid when it ____ what it is ____ to ____. The intended uses for most tests fall into one of three categories, and each category is associated with a ____ ____ for establishing validity.

Answer 1

Validity; Measures; Intended to Measure; Different Method

Question 2

The test will be used to obtain information about an examinee’s familiarity with a particular content or behavior domain: ____ ____.

Answer 2

Content Validity

Question 3

The test will be used to determine the extent to which an examinee possesses a particular hypothetical trait: ____ ____.

Answer 3

Construct Validity

Question 4

The test will be used to estimate or predict an examinee’s standing or performance on an external criterion: ____-____ ____.

Answer 4

Criterion-Related Validity

Question 5

For some tests, it is necessary to demonstrate only ____ ____ of ____; for others, it is desirable to establish ____ than ____ ____. For example, if an arithmetic achievement test will be used to assess the classroom learning of 8th grade students, establishing the test’s ____ ____ would be sufficient. If the same test will be used to predict the performance of 8th grade students in an advanced high school math class, the test’s ____ and ____-____ ____ will both be of concern.

Answer 5

One Type of Validity; More than One Type; Content Validity; Content and Criterion-Related Validity

Question 6

Note that, even when a test is found valid for a particular purpose, it might not be valid for that purpose for ____ ____. It is quite possible for a test to be a valid measure of intelligence or a valid predictor of job performance for one group of people but not for ____ ____.

Answer 6

All People; Another Group

Question 7

A test has ____ ____ to the extent that it adequately samples the content or behavior domain that it is designed to measure. If test items are not a good sample, results of testing will be ____. Although content validation is sometimes used to establish the validity of personality, aptitude, and attitude tests, it is most associated with ____-____ ____ that measure knowledge of one or more ____ ____ and with tests designed to assess a well-defined ____ ____. Adequate content validity would be important for a ____ ____ and for a ____ (job) ____ ____.

Answer 7

Content Validity; Misleading; Achievement-Type Tests; Content Domains; Behavior Domain; Statistics Test; Work Simple Test

Question 8

Content validity is usually “built into” a test as it is constructed through a ____, ____, and ____ ____ that involves ____ ____ the ____ or ____ ____ to be ____ and then ____ or ____ ____ that ____ that ____. Once a test has been developed, the establishment of content validity relies primarily on the ____ of ____ ____ ____. When experts agree that test items are an ____ and ____ ____ of the ____ ____, this provides evidence of the test’s ____ ____.

Answer 8

Systematic, Logical, and Qualitative Process; Clearly Identifying; Content; Behavior Domain to be Sampled; Writing or Selecting Items; Represent that domain; Judgment of Subject Matter Experts; Adequate and Representative Sample of the Target Domain; Content Validity

Question 9

Several methods for quantifying the judgments of subject matter experts are available. For example, the ____ ____ ____ (CVR) is the ratio of the subject matter experts who rated an item as essential to defining the content domain of interest to the total number of raters.

Answer 9

Content Validity Ratio

Question 10

Content validity must not be confused with ____ ____. ____ ____ refers to the systematic evaluation of a test by experts who determine whether test items adequately sample the relevant domain, while ____ ____ refers simply to whether or not a test “looks like” it measures what it is intended to measure. Although face validity is ____ an actual type of validity, it is a desirable feature for many tests. If a test lacks face validity, examinees may not be motivated to respond to items in an ____ or ____ manner. A high degree of face validity does not, however, indicate that a test has ____ ____.

Answer 10

Face Validity; Content Validity; Face Validity; Not; Honest or Accurate; Content Validity

Question 11

A test is valid when it (1) ____ what it is designed to measure. There are three main forms of validity: Content validity is of concern whenever a test has been designed to measure one or more (2) ____. (3) ____ validity is important when a test will be used to measure a hypothetical (4) ____, such as achievement, motivation. intelligence, or mechanical aptitude. (5) ____ validity is of interest when a test has been designed to estimate or predict performance on another measure.

Answer 11

(1) accurately measures; (2) content or behavior domains; (3) Construct; (4) trait (construct); (5) Criterion-related

Question 12

A test has (6) ____ validity when its items are a representative sample of the domain(s) that the test is intended to measure. Content validity is usually built into a test as it is being constructed through the selection of a (7) ____ sample of items. After a test has been developed, its content validity is checked by having (8) ____ experts evaluate the test in a systematic way.

Answer 12

(6) content; (7) representative; (8) subject matter

Question 13

a test has been found to measure the hypothetical trait (construct) it is intended to measure, the test is said to have ____ ____. A ____ is an abstract characteristic that cannot be observed directly but must be inferred by observing its ____. Intelligence, mechanical aptitude, self-esteem, and neuroticism are all ____.

Answer 13

Construct Validity; Construct; Effects; Constructs

Question 14

There is no ____ ____ to establish a test’s construct validity. Instead, construct validation entails a systematic accumulation of evidence showing that the test ____ ____ the ____ it was ____ to ____. The various methods used to establish this type of validity answer a slightly different question about the construct.

Answer 14

Single Way; Actually Measures the Construct; Designed to Measure

Question 15

________________: Do scores on individual test items correlate highly with the total test score; i.e., are all of the test items measuring the same construct?

Answer 15

Assessing the test’s internal consistency

Question 16

_____________: Do scores on the test accurately distinguish between people who are known to have different levels of the construct?

Answer 16

Studying group differences

Question 17

___________________________: Do test scores change, following an experimental manipulation, in the direction predicted by the theory underlying the construct?

Answer 17

Conducting research to test hypotheses about the construct

Question 18

_________________________: Does the test have high correlations with measures of the same and related traits (convergent validity) and low correlations with measures of unrelated traits (discriminant validity)?

Answer 18

Assessing the test’s convergent and discriminant validity

Question 19

_______________: Does the test have the factorial composition it would be expected to have — i.e., does it have factorial validity?

Answer 19

Assessing the test’s factorial validity

Question 20

Construct validity is said to be the most ____-____ of the methods of test validation. The developer of a test designed to measure a construct begins with a ____ about the ____ of the ____, which then guides the test developer in ____ ____ ____ and in choosing the ____ for ____ the ____ ____. If the developer of a creativity test believes that creativity is unrelated to general intelligence, that creativity is an innate characteristic that cannot be learned, and that creative people can be expected to generate more alternative solutions to certain types of problems than non-creative people, she would want to determine the correlation between scores on the creativity test and a measure of intelligence, see if a course in creativity affects test scores, and find out if test scores distinguish between people who differ in the number of solutions they generate to relevant problems.

Answer 20

Theory-Laden; Theory; Nature; Construct; Selecting Test Items; Methods for Establishing the Test’s Validity

Question 21

____ and ____ ____: As noted above, one way to assess a test’s construct validity is to ____ test scores with ____ on ____ that do and do not purport to ____ the ____ ____. High correlations with measures of the same and related traits provide evidence of the test’s ____ ____, while low correlations with measures of unrelated characteristics provide evidence of the test’s ____ ____ (which is also known as divergent validity).

Answer 21

Convergent and Discriminant Validity; Correlate; Scores on Measures; Assess the Same Trait; Convergent Validity; Discriminant Validity

Question 22

The ____-____ ____ (Campbell & Fiske, 1959) is used to systematically organize the data collected when assessing a test’s convergent and discriminant validity. The multitrait-multimethod matrix is a ____ of ____ ____, and, as its name suggests, it provides information about the ____ of ____ between ____ or ____ ____ that have each been assessed using ____ or ____ ____. When the correlations between different methods measuring the same trait are larger than the correlations between the same and different methods measuring different traits, the matrix provides evidence of the test’s ____ and ____ ____.

Answer 22

Multitrait-Multimethod Matric; Table of Correlation Coefficients; Degree of Association; Two or More Traits; Two or More Methods; Convergent and Discriminant Validity

Question 23

____-____ ____ (“same trait-same method”): The ____-____ ____ (coefficients in italics) are reliability coefficients and indicate the correlation between a measure and itself. Although these coefficients are not directly relevant to a test’s convergent and discriminant validity, they should be ____ for the matrix to provide useful information.

Answer 23

Monotrait-Monomethod Coefficients; Monotrait-Monomethod Coefficients; Large

Question 24

____-____ ____ (“same trait-different methods”): These coefficients (underlined coefficients) indicate the correlation between different measures of the same trait. When these coefficients are ____, they provide evidence of ____ ____.

Answer 24

Monotrait-Heteromethod Coefficients; Large; Convergent Validity

Question 25

____-____ ____ (“different traits-same method”): These coefficients (coefficients in parentheses) show the correlation between different traits that have been measured by the same method. When the heterotrait-monomethod coefficients are ____, this indicates that a test has ____ ____.

Answer 25

Heterotrait-Monomethod Coefficients; Small; Discriminant Validity

Question 26

____-____ ____ (“different traits-different methods”): The heterotrait-heteromethod coefficients (coefficients in brackets) indicate the correlation between different traits that have been measured by different methods. These coefficients also provide evidence of ____ ____ when they are ____.

Answer 26

Heterotrait-Heteromethod Coefficients; Discriminant Validity; Small

Question 27

Note that, in a multitrait-multimethod matrix, only those correlation coefficients that include the test that is being ____ are of ____. For the above example, the correlation between the assertiveness rating and the aggressiveness rating (r = .16) is a heterotrait-monomethod coefficient, but it isn’t of interest because it doesn’t provide information about the assertiveness test. Also, the number of correlation coefficients that can provide evidence of convergent and discriminant validity depends on the ____ of ____ ____ in the ____. In the example, only four measures were included (the minimum number), but there could have been more.

Answer 27

Validated; Interest; Number of Measures Included; Matrix

Question 28

Example: Three of the correlation coefficients in the above multitrait-multimethod matrix are relevant to the construct validity of the assertiveness test. The correlation between the assertiveness test and the assertiveness rating (monotrait-heteromethod coefficient) is .71. Since this is a relatively high correlation, it suggests that the test has ____ ____. The correlation between the assertiveness test and the aggressiveness test (heterotrait-monomethod coefficient) is .13 and the correlation between the assertiveness test and the aggressiveness rating (heterotrait-heteromethod coefficient) is .04. Because these two correlations are low, they confirm that the assertiveness test has ____ ____. This pattern of correlation coefficients confirms that the assertiveness test has ____ ____. Note that the monotrait-monomethod coefficient for the assertiveness test is .93, which indicates that the test also has adequate reliability. (The other correlations in the matrix are not relevant to the psychologist’s validation study because they do not include the assertiveness test.)

Answer 28

Convergent Validity; Discriminant Validity; Construct Validity

Question 29

In the context of the multitrait-multimethod matrix, “mono” means same and “hetero” means ____. Keeping this in mind will help you “translate” the names of the correlation coefficients contained in the matrix: For example, a heterotrait-monomethod coefficient indicates the correlation between two different traits that have been measured with the ____ ____ of ____.

Answer 29

Different; Same Kind of Method

Question 30

For the exam, know that convergent and discriminant validity are methods for assessing ____ ____ and be familiar with the names and Interpretations of the various types of correlation coefficients contained in a multitrait-multimethod matrix (e.g., know that a large monotrait-heteromethod coefficient provides evidence of ____ ____). It is NOT likely that you will encounter a question on the exam that resents a matrix and requires you to interpret a specific correlation coefficient or identify a particular type of coefficient.

Answer 30

Construct Validity; Convergent Validity

Question 31

A test has construct validity when it has been shown that the test actually measures the hypothetical (1) ____ it is intended to measure. One method for assessing a test’s construct validity is to determine if the test has both (2) ____ and (3) ____ (divergent) validity. When a test has (4) ____ correlations with measures that assess the same construct, this provides evidence of the test’s (5) ____ validity; when a test has (6) ____ correlations with measures of unrelated characteristics, this indicates that the test has (7) ____ validity.

Answer 31

(1) trait (construct); (2) convergent; (3) discriminant; (4) high; (5) convergent; (6) low; (7) discriminant

Question 32

The (8) ____ matrix provides a systematic way to organize the data collected when assessing a test’s convergent and discriminant validity. The matrix is a table of (9) ____ coefficients. It indicates that a test has convergent validity when the monotrait-(10) ____ coefficients are large and discriminant validity when the heterotrait-(11) ____ and the (12) ____-heteromethod coefficients are small.

Answer 32

(8) multitrait-multimethod; (9) correlation; (10) heteromethod; (11) monomethod; (12) heterotrait

Question 33

A ____ ____ is conducted to identify the minimum number of common factors (dimensions) required to account for the intercorrelations among a set of tests. subtests, or test items. The information obtained from a factor analysis is used for several reasons including assessing a test’s ____ ____. When used for this purpose, a test is considered to have construct (factorial) validity when it has ____ ____ with the factor(s) that it would be ____ to ____ ____ and ____ ____ with the factor(s) It would ____ be ____ to ____ ____. In other words, when employed as a method for assessing construct validity, factor analysis is another way to obtain information about a ____ ____ and ____ ____.

Answer 33

Factor Analysis; Construct Validity; High Correlations; Expected to Correlate With and Low Correlations; Not be Expected to Correlate With; Test’s Convergent and Discriminant Validity

Question 34

____ ____ ____ to a ____ of ____: When using factor analysis to assess the construct validity of a test, the test is administered to a sample of examinees along with other tests, some that measure the same construct and some that measure different constructs.

Answer 34

Administer Several Tests to a Group of Examinees

Question 35

____ ____ on ____ ____ with ____ on ____ ____ ____ to ____ a ____ (_) ____: High correlations between tests suggests that they are measuring the same construct(s), while low correlations indicate that they are measuring different constructs. It is the pattern of correlations between the tests that indicates how many factors should be “extracted” in the factor analysis.

Answer 35

Correlate Scores on Each Test with Scores on Every Other Test to Obtain a Correlation (R) Matrix

Question 36

____ ____ of ____ ____ ____ ____, ____ the ____ ____ to a ____ ____: The data in the correlation matrix are used to derive a factor matrix. The factor matrix contains correlation coefficients (“factor loadings”), which indicate the degree of association between each test and each factor.

Answer 36

Using One of Several Factor Analytic Techniques, Convert the Correlation Matrix to a Factor Matrix

Question 37

____ the ____ of the ____ by “____” ____: The pattern of factor loadings tn the original factor matrix is usually difficult to interpret, so the factors are “rotated” to obtain a clearer pattern of correlations. As noted below, the rotation can produce either orthogonal or oblique factors.

Answer 37

Simplify the Interpretation of the Factors by “Rotating” Them

Question 38

____ and ____ the ____ in the ____ ____ ____: The appropriate names (labels) for the factors are determined by examining the tests that do and do not correlate with each factor. When the factor analysis is being conducted to assess a test’s construct validity, validity is demonstrated when the test has high correlations with the factor(s) it would be expected to correlate with and low correlations with the factor(s) it would not be expected to correlate with.

Answer 38

Interpret and Name the Factors in the Rotated Factor Matrix

Question 39

Example: A psychologist decides to assess the construct validity of an interpersonal assertiveness test (Test A) by administering it to a group of salespeople along with two other measures of assertiveness (Tests B and C) and three measures of aggressiveness (Tests X, Y, and Z). The psychologist first derives a correlation matrix from the data she has collected. The correlation matrix provides her with the information she needs to determine how many factors to extract and with the data required to derive the initial factor matrix. Because the initial factor matrix is difficult to interpret, the psychologist ____ the initial factors and obtains the following rotated factor matrix:

Answer 39

Rotates

Question 40

The numbers listed under the headings “Factor I” and “Factor Il” are ____ ____, which are correlation coefficients that indicate the degree of association between each test and each factor. One way to interpret a factor loading is to ____ ____ to determine the amount of ____ in ____ ____ that is accounted for by (explained by) the ____: The interpersonal assertiveness test (Test A) has factor loadings of .78 and .16 with Factors I and Il, respectively. This means that 61% (.782) of variability in the assertiveness test is accounted for by Factor I, while 3% (.162) is accounted for by Factor II.

Answer 40

Factor Loadings; Square It; Variability in Test Scores; Factor

Question 41

The numbers in the last column of the factor matrix are the ____. Each test’s ____ indicates its “common variance,” or the amount of variability in test scores that Is due to the factors that the test shares in common, to some degree, with the other tests included in the analysis. In other words, a test’s communality indicates the ____ ____ of ____ in ____ ____ that is accounted for by ____ of the ____ ____. The communality for the interpersonal assertiveness test is .64. This means that 64% of variability in assertiveness test scores is accounted for by a combination of Factor I and Factor II.

Answer 41

Communalities; Communality; Total Amount of Variability in Test Scores; All of the Identified Factors

Question 42

Recall that a ____ ____ indicates the amount of variability in test scores that represents true score variability. From the perspective of factor analysis, true score variability (reliability) can be divided into two components: The first component is the ____, which represents variability that is due to factors that the test shares in common with the other tests included in the factor analysis — i.e., it is the portion of ____ ____ ____ that has been explained by the ____ ____.

Answer 42

Reliability Coefficient; Communality; True Score Variability; Factor Analysis

Question 43

The second true score variability (reliability) component is the ____, which is variability that is due to factors that are specific, or unique, to the test and that are not measured by any other test included in the analysis. In other words, specificity is the portion of true score variability that has not been ____ by the ____ ____.

Answer 43

Specificity; Explained; Factor Analysis

Question 44

Because of the relationship between reliability and communality, it is possible to conclude that a test’s reliability will always be ____ ____ as ____ as its ____. In other words, the communality is a ____-____ ____ of a ____ ____ ____. Since the interpersonal assertiveness test has a communality of .64, its reliability coefficient will necessarily be .64 or larger.

Answer 44

At Least as Large; Communality; Lower-Limit Estimate; Test’s Reliability Coefficient

Question 45

Finally, to use factor analysis to determine if a test has construct validity, the ____ must be ____. This is done by ____ the ____ of ____ ____.

Answer 45

Factors; Named; Inspecting the Pattern of Factor Loadings

Question 46

Rotation is a very complex procedure that you don’t need to be familiar with for the exam and involves “____” the communality Of each test included in the analysis. As a result of the ____, each factor accounts for a different proportion of a test’s ____ than it did ____ to ____. As noted previously, the purpose of a rotation is to make it ____ to ____ the ____ ____.

Answer 46

Redividing; Redivision; Variability; Prior to Rotation; Easier to Interpret the Factor Loadings

Question 47

‘There are two types of rotation, ____ and ____. When the rotation is ____, the resulting factors are uncorrelated. That is, the attribute measured by one factor is ____ from the attribute(s) measured by the other factor(s). In contrast, when the rotation is ____, the resulting factors are ____, and the attributes measured by the factors are ____ ____. A test developer decides on the basis of his or her ____ about the ____ ____ by the ____ included in the ____, whether to conduct an ____ or an ____ ____.

Answer 47

Orthogonal and Oblique; Orthogonal; Independent; Oblique; Correlated; Not Independent; Theory; Constructs Measured; Tests; Analysis; Orthogonal; Oblique Rotation

Question 48

If a psychologist believes that interpersonal assertiveness is unrelated to aggressiveness, she would choose an ____ ____. But, she thinks that assertiveness and aggressiveness are not entirely independent, she would conduct an ____ ____. There are several different methods of orthogonal and oblique rotation: Varimax, quartimax, and equimax are commonly-used types of ____ ____; quartimin, oblimin, and oblimax are types of ____ ____.

Answer 48

Orthogonal Rotation; Oblique Rotation; Orthogonal Rotation; Oblique Rotation

Question 49

An important distinction between orthogonal and oblique factors is that, when factors are orthogonal, a test’s communality can be calculated from its ____ ____: With orthogonal factors. a test’s communality is ____ to the ____ of the ____ ____ ____. The factors in the above example are orthogonal. Therefore, the communality for the interpersonal assertiveness test is equal to the sum of its squared factor loadings; i.e., .78^2 + .16^2 + 03 = .64. (When the factors are oblique, the sum of the squared factor loadings ____ the ____.)

Answer 49

Factor Loading; Equal to the Sum of the Squared Factor Loadings; Exceeds the Communality

Question 50

The most important things to remember about factor analysis are (1) that a squared factor loading provides a measure of “____ ____”; (2) that, when factors are orthogonal, a test’s communality can be calculated by ____ and ____ the ____ ____ ____; and (3) that orthogonal factors are ____, while oblique factors are ____. You do NOT have to understand the processes involved in extracting actors and rotating a actor matrix.

Answer 50

Shared Variability; Squaring and Adding the Test’s Factor Loadings; Uncorrelated; Correlated

Question 51

Factor analysis is used to identify the factors (dimensions) that underlie the (1) ____ among a set of tests, subtests, or test items. One use of factor analysis is to determine if a test has construct validity: A test is shown to have construct validity when it has (2) ____ correlations with the factor(s) it is expected to correlate with and (3) ____ correlations with the factor(s) it is not expected to correlate with.

Answer 51

(1) intercorrelations; (2) high; (3) low

Question 52

In a factor matrix, the correlation between a test and a factor is referred to as a (4) ____. This correlation can be interpreted in terms of shared variability by squaring the correlation coefficient. For example, if a test has a correlation of .50 with Factor I, this means that (5) ____ percent of variability in test scores is explained by Factor I.

Answer 52

(4) factor loading; (5) 25

Question 53

When the identified factors are (6) ____ (uncorrelated), a test’s communality can be calculated by summing the (7) ____. If a test has a correlation of .50 with Factor I and a correlation of .20 with Factor Il and the factors are uncorrelated, the test’s communality is equal to (8) ____. This means that (9) ____ of the variability in test scores is explained by the identified factors, while the remaining variability is due to some combination of (10) ____ and measurement error.

Answer 53

(6) orthogonal; (7) squared factor loadings; (8) .502 + .202 = .25 + .04 = .29; (9) 29%; (10) specificity

Question 54

____-____ ____ is of interest whenever test scores are to be used to draw conclusions about an examinee’s likely status or performance on another measure. For example, a test developer would want to make sure that an employee selection test has ____ ____-____ ____ if the goal of testing is to predict how well an applicant will do on a measure of job performance after he or she is hired. In such situations, the test is referred to as the ____ and is symbolized with the letter “,” while the other measure is the ____ and is symbolized with the letter “.” Criterion-related validity is assessed by ____ the ____ of a ____ of ____ on the ____ with their scores on the ____. When the resulting criterion-related validity coefficient is sufficiently ____, this confirms that the predictor has ____-____ ____.

Answer 54

Criterion-Related Validity; Adequate Criterion-Related Validity; Predictor; X; Criterion; Y; Correlating the Scores; Sample of Individuals; Predictor; Criterion; Larger; Criterion-Related Validity

Question 55

____ vs. ____ ____: There are two forms of criterion-related validity, ____ and ____. The difference between the two is the ____ of the ____ of ____ and ____: When criterion data are collected ____ ____ or at about the ____ ____ as ____ on the ____, the predictor’s ____ ____ is being assessed. When the criterion is measured some time after the predictor has been administered, ____ ____ is being evaluated.

Answer 55

Concurrent vs. Predictive Validity; Concurrent and Predictive; Timing; Administration; Predictor and Criterion; Prior To; Same Time as Data; Predictor; Concurrent Validity; Predictive Validity

Question 56

Concurrent validity is of interest when a predictor will be used to ____ ____ ____, while predictive validity is the preferred type of criterion-related validity when the purpose of testing is to ____ ____ ____ on the ____. Concurrent validity would be the best method for a selection test that is to be used to determine if the job performance of job applicants will be acceptable immediately. In contrast predictive validity would be more appropriate for a selection test that is being administered to job applicants to predict their future job performance. In some situations, predictive validity is the preferred method, but practical considerations make it necessary to conduct a concurrent validity study.

Answer 56

Estimate Current Status; Predict Future Performance on the Criterion

Question 57

Be sure that you have convergent and discriminant (divergent) validity associated with ____ ____ and concurrent and predictive validity associated with ____-____ ____.

Answer 57

Construct Validity; Criterion-Related Validity

Question 58

____ of the ____-____ ____ ____: Like other correlation coefficients. a criterion-related validity coefficient can be interpreted in terms of its ____. In most situations, validity coefficients rarely exceed _, and a coefficient as low as .20 or .30 might be ____. A selection test with a validity coefficient of .30, for instance, may be useful if ____ ____ are ____ or have even ____ ____ ____. or if the test will be administered in ____ with other ____. Because the validity coefficient represents the ____ ____ ____ ____ ____, it can also be interpreted in terms of ____ ____ by ____ the ____.

Answer 58

Interpretation of the Criterion-Related Validity Coefficient; Magnitude; .60; Acceptable; Alternative Predictors are Unavailable; Lower Validity Coefficients; Conjunction; Predictors; Correlation Between Two Different Measures; Shared Variability by Squaring the Coefficient

Question 59

You can square a correlation coefficient to interpret It only when it represents the ____ ____ ____ ____ ____ or ____ ____. When the correlation between two different measures is squared, it provides a measure of ____ ____. Terms that suggest shared variability include “____ ____ ____” and “____ ____.” In other words, (fan exam question gives you the correlation coefficient for Variable X and Variable Y and asks how much variability In Variable Y is explained by Variable X. you’ll know that you need to ____ the ____ ____ to obtain the correct answer.

Answer 59

Correlation Between Two Different Tests or Other Variables; Shared Variability; Accounted For By; Explained By; Square the Correlation Coefficient

Question 60

The ____ ____ of ____: The information collected in a criterion-related validity study can be used to derive a ____ ____, which is then employed in the future to predict or estimate an individual’s ____ () ____ from his or her ____ ____ () ____. A problem with using a regression equation to predict criterion scores is that, unless the criterion-related validity coefficient is ____ to either __ or __, there will be some ____ in ____.

Answer 60

The Standard Error of Estimate; Regression Equation; Criterion (Y) Score; Obtained Predictor (X) Score; Equal; -1.0 or +1.0; Error in Prediction

Question 61

In much the same way that the standard error of measurement is used to calculate a confidence interval for an examinee’s obtained test score, the ____ ____ of ____ (___) is used to construct a ____ ____ around a predicted (estimated) criterion score: The 68% confidence interval is constructed by ____ and ____ ____ ____ ____ from the ____ ____ ____, the 95% confidence interval by adding and subtracting two standard errors, and the 99% confidence interval by adding and subtracting three standard errors.

Answer 61

Standard Error of Estimate (SEE); Confidence Interval; Adding and Subtracting One Standard Error; Predicted Criterion Score

Question 62

The magnitude of the standard error of estimate is affected by two factors, the ____ ____ of the ____ ____ and the ____ ____-____ ____ ____. The ____ the standard deviation and the ____ the validity coefficient, the ____ the standard error and vice versa. For example, when the validity coefficient is equal to __, the standard error of estimate equals _; conversely, when the validity coefficient equals , the standard error is equal to the ____ ____ of the ____ () ____.

Answer 62

Standard Deviation; Criterion Scores; Predictor’s Criterion-Related Validity Coefficient; Smaller; Larger; Smaller ±1.0; 0; 0; Standard Deviation; Criterion (Y) Scores

Question 63

The standard error of estimate is used to construct a ____ ____ around an ____ (____) ____. For the exam, be sure you know the difference between the standard error of estimate and the standard error of measurement.

Answer 63

Confidence Interval; Estimated (Predicted) Score

Question 64

When the purpose of testing is to draw conclusions about performance on another measure, the test is referred to as the (1) ____ and the other measure is called the (2) ____. In this situation, the test’s criterion-related validity is of interest, and it is established by (3) ____ scores on the test with scores on the other measure. If the resulting criterion-related (4) ____ is sufficiently large, this demonstrates that the test has criterion-related validity.

Answer 64

(1) predictor; (2) criterion; (3) correlating; (4) validity coefficient

Question 65

There are two types of criterion-related validity: When evaluating (5) ____ validity, the predictor is administered to a sample of examinees prior to collecting data on the criterion. It is the appropriate type of validity when the goal of testing is to predict (6) ____ status on the criterion. When evaluating (7) ____ validity, the predictor and criterion are administered at about the same time. It is the preferred method for assessing validity when the purpose of testing is to estimate (8) ____ status on the criterion.

Answer 65

(5) predictive; (6) future; (7) concurrent; (8) current

Question 66

Because a validity coefficient is never perfect (equal to ± I .0), there is always some (9) ____ in prediction when a regression equation is used to predict a person ‘s (10) score from his or her obtained predictor score. Consequently, the standard error of (11) ____ is used to construct a confidence interval around an individual’s predicted (12) ____ score. For example, if the criterion ‘s standard deviation is 5 and the predictor’s validity coefficient is .60, the standard error is equal to (13) ____; and, if a person’s predicted criterion score is 60, the 68% confidence interval is (14) ____.

Answer 66

(9) error; (10) criterion; (11) estimate; (12) criterion; (13) 4.0: (14) 56 to 64

Question 67

____ ____: The data collected in a concurrent or predictive validity study can also be used to assess a predictor’s ____ ____, or the increase in correct decisions that can be expected if the predictor is used as a ____-____ ____. This is important because, even when a predictor’s validity coefficient is ____, use of the predictor might not result in a ____ ____ of ____ ____ than the technique that is ____ ____ ____.

Answer 67

Incremental Validity; Incremental Validity; Decision-Making Tool; Large; Larger Proportion of Correct Decisions; Currently Being Used

Question 68

The first step in evaluating a predictor’s incremental validity is to use the data collected in the ____ ____ to construct a ____.

Answer 68

Validity Study; Scatterplot

Question 69

Example Scatterplot: Assertiveness Test (with blanks)

Answer 69

Answers

Question 70

To use a scatterplot to assess a predictor’s ____ ____, the criterion and predictor cutoff scores ____ ____ ____. The dark horizontal line in the above scatterplot is the ____ ____ for average monthly sales (i.e., the criterion cutoff score). It divides the people represented by the data points in the scatterplot into ____ ____: people who are currently successful on the job and people who are not successful. The scatterplot shows that 61 of the 100 salespeople included in the study are successful, while the remaining 39 are unsuccessful. (Recall that the salespeople in the validation study were people who had been hired without considering their scores on the interpersonal assertiveness test.)

Answer 70

Incremental Validity; Must Be Set; Cutoff Score; Two Groups

Question 71

The dark vertical line in the scatterplot is the ____ ____ for the assertiveness test (i.e., the predictor cutoff score). It also divides the people into ____ ____: the ____ and the ____. The positives are those people who scored ____ the ____ ____ and who would have been hired if the predictor had been used as a selection tool (i.e., they were identified by the predictor as people who would be successful on the job). The negatives are those people who scored ____ the ____ ____ and who would not have been hired if the predictor had been used to make selection decisions (i.e., they were identified as people who would not be successful on the job). If the assertiveness test had been used to select the 100 salespeople included in the concurrent validity study, 22 of them would have been hired (the positives) and 78 would not have been hired (the negatives).

Answer 71

Cutoff Score; Two Groups; Positives; Negatives; Above the Predictor Cutoff; Below the Predictor Cutoff

Question 72

Together, the criterion and predictor cutoff scores divide the scatterplot into ____ ____.

Answer 72

Four Quadrants

Question 73

The ____ ____ are the people who are predicted to succeed by the predictor and who are actually successful on the criterion.

Answer 73

True Positives

Question 74

The ____ ____ are the people who are predicted to succeed by the predictor but who are not successful on the criterion.

Answer 74

False Positives

Question 75

The ____ ____ are the people who are predicted to be unsuccessful by the predictor and, in fact, are unsuccessful on the criterion.

Answer 75

True Negatives

Question 76

The ____ ____ are the people who are predicted to be unsuccessful by the predictor but who are actually successful on the criterion.

Answer 76

False Negatives

Question 77

The true positives and the true negatives represent ____ ____, while the false positives and the false negatives are ____ ____.

Answer 77

Correct Decisions; Incorrect Decisions

Question 78

The number of cases in each quadrant of a scatterplot would ____, of course, if the predictor and/or the criterion cutoff scores were ____ or ____. If the predictor cutoff score were lowered (moved to the left), the number of ____ would ____ and the number of ____ would ____.

Answer 78

Differ; Raised or Lowered; Positives would Increase; Negatives would Decrease

Question 79

If the predictor cutoff were raised (moved to the right), the number of positives would ____ and the number of negatives would ____. The selection of an optimal predictor cutoff is based on consideration of the ____ of ____ in the ____ ____ and the ____ of ____: If the goal is to maximize the proportion of true positives (the ratio of true positives to total positives), for example, a ____ ____ ____ ____ is ____ since this will reduce the number of ____ ____.

Answer 79

Decrease; Increase; Number of People; Four Quadrants; Goal of Testing; High Predictor Cutoff Score is Set; False Positives

Question 80

Raising or lowering the criterion cutoff score also changes the ____ of ____ who fall into each ____ Although the criterion cutoff is theoretically ____, in many situations it ____ be ____. For instance, it might not be acceptable to the company to consider with lower average monthly sales as successful and, consequently, the criterion cutoff score could not be lowered.

Answer 80

Number of People; Quadrant; Alterable; Cannot be Changed

Question 81

Once the criterion and predictor cutoff scores have been set, the predictor’s ____ ____ can be ____. This is accomplished by ____ the ____ ____ from the ____ ____ ____.

Answer 81

Incremental Validity; Calculated; Subtracting the Base Rate from the Positive Hit Rate

Question 82

The ____ ____ is the proportion of people who were selected without use of the predictor and who are currently considered successful on the criterion. The base rate is calculated by ____ the ____ ____ of ____ ____ (the true positives plus the false negatives) by the ____ ____ of ____.

Answer 82

Base Rate; Dividing the Total Number of Successful People; Total Number of People

Question 83

The ____ ____ ____ is the proportion of people who would have been selected on the basis Of their predictor scores and who are successful on the criterion. The positive hit rate is calculated by ____ the ____ of ____ ____ by the ____ ____ of ____.

Answer 83

Positive Hit Rate; Dividing the Number of True Positives by the Total Number of Positives

Question 84

Example: To assess the incremental validity of the interpersonal assertiveness test, the psychologist calculates the ____ ____ and the ____ ____ ____. The base rate is the number of salespeople who are currently considered successful divided by the total number of salespeople: (21 + 40)/ 100 = .61. In other words, 61% of the people who were hired without considering their scores on the interpersonal assertiveness test are successful salespeople. The ____ ____ ____ is the number of true positives divided by the total number of positives: 21/22 .95. If the assertiveness test had been used as a selection tool for the 100 employees included in the validity study, 95% of those hired would have been successful. Finally, to determine the test’s incremental validity, the ____ ____ is ____ from the ____ ____ ____: Internal Validity = /95 - .61 = .34

Answer 84

Base Rate and the Positive Hit Rate; Positive Hit Rate; Base Rate is Subtracted from the Positive Hit Rate

Question 85

This means that use of the interpersonal assertiveness test as a selection tool can be expected to increase the proportion of successful employees by 34%. (Whether or not this is an Acceptable increase in decision-making accuracy is a ____ ____.)

Answer 85

Subjective Judgement

Question 86

The Above discussion focused on the assessment of incremental validity for a selection test that will be used to predict whether job applicants will be successful in terms of job performance. Incremental validity is also important in other situations in which a test will be used as a ____-____ ____.

Answer 86

Decision-Making Tool

Question 87

A psychologist would want to make sure that a screening test that will be used as a substitute for a lengthier diagnostic procedure has ____ ____ ____. In that situation, the procedure for evaluating incremental validity is ____ except that the ____ is ____ ____: The positives and negatives are, respectively, the people who are and are not identified by the ____ (screening test) as having the ____ ____, while the ____ ____ divides people into those who have been identified by the lengthier diagnostic procedure as actually having or not having the disorder.

Answer 87

Adequate Incremental Validity; Similar; Terminology is Somewhat Different; Predictor; Target Disorder; Criterion Cutoff

Question 88

It is the ____ that determines if a person is a positive or negative, and the ____ that determines if they are a “true” or a “false. “ Also note that, on the exam, the formation may be presented in a table (rather than a scatterplot) and in a different orientation so that the false positives, for instance, won’t be located in the lower right-hand quadrant. For this kind of question, keep in mind that. when trying to determine whether to raise or lower the predictor and criterion cutoffs to change the number of people in a particular quadrant, it’s easier to identify the correct answer if you use the orientation o the above scatter lot.

Answer 88

Predictor; Criterion

Question 89

____ ____: The data used to evaluate a predictor’s accuracy can also be expressed in terms of ____ ____ ____ — i.e., _________________________________________.

Answer 89

Classification Accuracy; Six Validity Indexes – i.e., sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood ratio, and negative likelihood ratio

Question 90

These six validity indexes are often used to evaluate the ____ of a ____ ____ ____ for identifying people who ____ and ____ ____ have a ____ ____; and, in this situation, the new test is the ____ and the ____ ____ using a ____-____ ____ ____ (i.e., the “gold standard”) is the ____.

Answer 90

Accuracy of a New Diagnostic Test; Do and Do Not; Particular Disorder; Predictor; Diagnosis Assigned; Well-Established Diagnostic Procedure; Criterion

Question 91

____ and ____: Sensitivity and specificity provide information about a ____ ____ when administered to a ____ of ____ who are known to have or not have the ____ of ____.

Answer 91

Sensitivity and Specificity; Predictor’s Accuracy; Group of Individuals; Disorder of Interest

Question 92

____ is the percent of people in the validation sample who have the disorder and were accurately identified by the predictor as having the disorder. It is calculated by ____ the number of ____ ____ by the number of ____ ____ ____ ____ ____.

Answer 92

Sensitivity; Dividing; True Positives; True Positives Plus False Negatives

Question 93

____ is the percent of people in the validation sample who do not have the disorder and were accurately identified by the predictor as not having the disorder. It is calculated by ____ the number of ____ ____ by the number of ____ ____ ____ ____ ____.

Answer 93

Specificity; Dividing; True Negatives; True Negatives Plus False Positives

Question 94

____ and ____ ____ ____: The ____ ____ of a ____ provide the information needed to estimate the probability that people have or do not have the disorder when they test ____ or ____ on the ____. The ____ ____ ____ (PPV) indicates the probability that people who test positive have the disorder. It is calculated by ____ the ____ of ____ ____ by the number of ____ and ____ ____. The ____ ____ ____ (NPV) indicates the probability that people who test negative do not have the disorder. It is calculated by ____ the number of ____ ____ by the number of ____ and ____ ____.

Answer 94

Positive and Negative Predictive Values; Predictive Values of a Predictor; Positive or Negative; Predictor; Positive Predictive Value; Dividing the Number of True Positives; True and False Positives; Negative Predictive Value; Dividing; True Negatives; True and False Negatives

Question 95

____ and ____ ____ ____: ____ ____ indicate the extent to which a positive or negative result on the predictor changes the probability that a person has the disorder that is assessed by the predictor. The ____ ____ ____ (LR+) expresses the extent to which a positive result affects the probability that the person has the disorder. It is calculated using the following formula: ____ /(_ — ____). To be useful, a predictor should have an LR+ of at least ; and the higher the LR+, the ____ the ____ that the disorder is ____ ____. The ____ ____ ____ (LR-) expresses the extent to which a negative result affects the probability that the person has the disorder. It is calculated with the following formula: ( — ____)/ ____. To be useful, a predictor should have an LR- of ____ than _; and the closer the LR- is to O, the ____ the ____ that the ____ is ____.

Answer 95

Positive and Negative Likelihood Ratios; Likelihood Ratios; Positive Likelihood Ratio; Sensitivity/(1 – Specificity); 1.0; Greater the Probability; Actually Present; Negative Likelihood Ratio; (1 – Sensitivity)/Specificity; Less than 1; Lower the Probability; Disorder is Present

Question 96

The ____ ____ ____ and ____: A test’s reliability always places a ____ on its validity. This means that, when a test has low ____, it cannot have a ____ ____ of ____, ____, or ____-____ ____. However, high reliability does not ____ ____. A test can be relatively free from the effects of measurement error but not measure what it was ____ to ____. In other words, while reliability is a necessary condition for ____, it is not a ____ ____. For criterion-related validity, the ____ ____ ____ and ____ is defined by Formula 3.

Answer 96

The Relationship Between Validity and Reliability; Ceiling; Validity; High Degree of Content, Construct, or Criterion-Related Validity; Guarantee Validity; Designed to Measure; Necessary; Validity; Sufficient Condition; Relationship Between Reliability and Validity

Question 97

This formula indicates that the predictor’s criterion-related validity coefficient cannot ____ the ____ ____ of its ____ ____. For example, if a predictor has a reliability coefficient of .81, its validity coefficient will necessarily be .90 or less.

Answer 97

Exceed the Square Root of its Reliability Coefficient

Question 98

Although many authors emphasize the impact of the ____ ____ on its ____, validity is actually ____ by the ____ of the ____ and the ____: To obtain a criterion-related validity coefficient, the ____ and ____ must ____ be ____ ____. The relationship between the validity coefficient and the reliability coefficients Of the predictor and the criterion is expressed by the following formula, which is simply an expansion of Formula 3.

Answer 98

Predictor’s Reliability; Validity; Limited; Reliability; Predictor; Criterion; Predictor and Criterion; Both; Highly Reliable

Question 99

If a predictor has a reliability coefficient of .81 and the criterion has a reliability coefficient of .91, use of Formula 4 indicates that the ____ ____ ____ for the ____ is approximately .86, which is lower than the ____ ____ ____ when only the ____ of the ____ is ____.

Answer 99

Maximum Validity Coefficient; Predictor; Maximum Coefficient Estimated; Reliability; Predictor is Considered

Question 100

____ for ____: Sometimes a test developer wants to estimate what a predictor’s validity coefficient would be if the predictor and/or the criterion were ____ ____ (i.e., had reliability coefficients of _). This can be accomplished by using the ____ for ____ ____. To use this formula, three pieces of information are needed: the predictor’s ____ ____ ____, the criterion’s ____ ____ ____, and the ____-____ ____ ____. A shortcoming of the correction for attenuation formula is that it tends to ____ the ____ ____ ____ that can be ____.

Answer 100

Correction for Attenuation; Perfectly Reliable; 1.0; Correction for Attenuation Formula; Current Reliability Coefficient; Current Reliability Coefficient; Criterion-Related Validity Coefficient; Overestimate the Actual Validity Coefficient; Achieved

Question 101

8. ____ ____: The accuracy of a criterion measure can be “contaminated” by the way in which scores on the criterion are ____. If a supervisor knows employees’ predictor scores when rating their job performance, the ratings might be affected by that ____. Such ____ ____ tends to inflate the relationship between a predictor and a criterion, resulting in an artificially high criterion-related validity coefficient. The best way to avoid criterion contamination is to make sure that the individual who is rating people on the criterion measure is ____ ____ with their ____ on the ____.

Answer 101

Criterion Contamination; Determined; Knowledge; Criterion Contamination; Not Familiar; Performance; Predictor

Question 102

____-____: When a predictor is developed, the items from the original item pool that are included in the final version of the test are those that are found to ____ ____ ____ with the ____. Some of the high correlations are due, however, to the ____ ____ of the ____ ____ ____ (i.e., to chance factors) rather than to an actual relationship between the items and the criterion measure. As a result, the predictor will be “____-____ “ for the ____ ____ ____ and, if the same sample is used to validate the test, the criterion-related validity coefficient is likely to be ____ ____.

Answer 102

Cross-Validation; Correlate Most Highly with the Criterion; Unique Characteristics; Item Tryout Sample; Tailor-Made; Original Tryout Sample; Spuriously High

Question 103

It Is always important to ____-____ a ____ on ____ ____. Because all of the same chance factors operating in the original sample will not be present in the new sample, the ____-____ coefficient tends to “shrink” (to be smaller than the original coefficient). The smaller the initial validation sample, the greater the ____ of the validity coefficient when the predictor is cross-validated.

Answer 103

Cross-Validate a Predictor on Another Sample; Cross-Validation; Shrinkage

Question 104

Even when a predictor has a large criterion-related validity coefficient, it might not be more useful for predicting criterion scores than the current method. Therefore, before using a new predictor, its (1) ____ validity should be checked to determine if there will be an increase in (2) ____ accuracy when the predictor is used to make selection decisions. This is done by subtracting the base rate from the (3) ____.

Answer 104

(1) incremental; (2) decision-making; (3) positive hit rate

Question 105

The predictor and criterion cutoff scores divide a scatterplot into four quadrants. The (4) ____ cutoff determines if someone is a positive or negative, while the (5) ____ cutoff determines if he/she is a “true” or “false.” The false positives are the people who score above the cutoff on the (6) ____ but below the cutoff on the (7) ____. To reduce the number of false positives, the predictor cutoff can be (8) ____ and/or the criterion cutoff can be (9) ____.

Answer 105

(4) predictor; (5) criterion; (6) predictor; (7) criterion; (8) raised; (9) lowered

Question 106

A predictor’s validity is limited by the reliability of both the predictor and the (10) ____. Specifically, a test’s criterion-related validity coefficient can be no greater than the (11) ____ of the product of the reliabilities of the predictor and the (12) ____.

Answer 106

(10) criterion; (11) square root; (12) criterion

Question 107

Criterion (13) ____ occurs when a rater’s knowledge of a person’s predictor performance affects how he/she rates the person on the criterion. Wien this occurs, the validity coefficient is likely to be spuriously (14) ____. Because the size of a validity coefficient is affected by group characteristics, a test that is found to be highly valid for one group might be less valid for another group. For example, the validity coefficient calculated for the initial validation sample often (15) ____ (decreases) when the predictor is (16) ____-validated on another sample.

Answer 107

(13) contamination; (14) high; (15) shrinks; (16) cross