Chapter 6: Validity

Question 1

as applied to a test, is a judgment or estimate of how well a test measures what it purports to measure in a particular context. More specifically, it is a judgment based on evidence about the appropriateness of inferences drawn from test scores.

is the process of gathering and evaluating evidence about validity. Both the test developer and the test user may play a role in the validation of a test for a specific purpose.

Answer 1

Validity

Question 2

One way measurement specialists have traditionally conceptualized validity is according to three categories (trinitarian view):

Answer 2

1) content validity
2) criterion-related validity
3) construct validity = umbrella validity

Question 3

Three approaches to assessing validity—associated, respectively, with content validity, criterion-related validity, and construct validity—are:

Answer 3

1. scrutinizing the test’s content
2. relating scores obtained on the test to other test scores or other measures
3. executing a comprehensive analysis of
   a. how scores on the test relate to other test scores and measures
   b. how scores on the test can be understood within some theoretical framework for understanding the construct that the test was designed to measure

Question 4

Another term you may come across in the literature is _____. This variety of validity has been described as the “Rodney Dangerfield of psychometric variables” because it has “received little attention—and even less respect—from researchers examining the construct validity of psychological tests and measures”.

Answer 4

face validity

Question 5

_____ relates more to what a test appears to measure to the person being tested than to what the test actually measures. Face validity is a judgment concerning how relevant the test items appear to be.

**from the perspective of the testtaker, not the test user.

A test’s lack of _____ could contribute to a lack of confidence in the perceived effectiveness of the test—with a consequential decrease in the testtaker’s cooperation or motivation to do his or her best.

Answer 5

Face validity

Question 6

_____ describes a judgment of how adequately a test samples behavior representative of the universe of behavior that the test was designed to sample.

Answer 6

Content validity

Question 7

The quantification of content validity

The measurement of _____ is important in employment settings, where tests used to hire and promote people are carefully scrutinized for their relevance to the job, among other factors.

Answer 7

content validity

Question 8

One method of measuring content validity, developed by C. H. Lawshe, is essentially a method for gauging agreement among raters or judges regarding how essential a particular item is.

**content validity ratio (CVR)

Answer 8

Lawshe

Question 9

Criterion-Related Validity (3)

Answer 9

1) Criterion-related
2) Concurrent
3) Predictive

Question 10

Criterion-Related Validity:

is a judgment of how adequately a test score can be used to infer an individual’s most probable standing on some measure of interest—the measure of interest being the criterion.

Ex. a company could administer a sales personality test to its sales staff to see if there is an overall correlation between their test scores and a measure of their productivity.

Answer 10

Criterion-Related Validity

Question 11

_____ is an index of the degree to which a test score is related to some criterion measure obtained at the same time (concurrently).

Ex. let’s say a group of nursing students take two final exams to assess their knowledge. One exam is a practical test and the second exam is a paper test. If the students who score well on the practical test also score well on the paper test, then ______ has occurred.

Answer 11

Concurrent validity

Question 12

_____ is an index of the degree to which a test score predicts some criterion measure.

Ex. the SAT test is taken by high school students to predict their future performance in college (namely, their college GPA).

Answer 12

Predictive validity

Question 13

Characteristics of a criterion (3)

Answer 13

1) An adequate criterion is relevant
2) An adequate criterion measure must also be valid
3) Ideally, a criterion is also uncontaminated.

Question 14

Judgments of criterion-related validity, whether concurrent or predictive, are based on two types of statistical evidence: (2)

Answer 14

Validity Coefficient and Expectancy Data.

Question 15

Type of statistical evidence:

The _____ is a correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure.

Typically, the Pearson correlation coefficient is used to determine the validity between the two measures.

Answer 15

validity coefficient

Question 16

Let’s say a medical practitioner is more likely to correctly diagnose a kidney infection if a urine test is ordered, rather than relying on a physical examination and discussion of symptoms alone. We can say that the urine test has _____.

Answer 16

Incremental validity

Question 17

Type of statistical evidence:

_____ provide information that can be used in evaluating the criterion-related validity of a test. Using a score obtained on some test(s) or measure(s), expectancy tables illustrate the likelihood that the testtaker will score within some interval of scores on a criterion measure—an interval that may be seen as “passing”, “acceptable”, and so on.

Answer 17

Expectancy data

Question 18

An _____ shows the percentage of people within specified test-score intervals who subsequently were placed in various categories of the criterion (for example, placed in “passed” category or “failed” category).

Answer 18

expectancy table

Question 19

_____ provide an estimate of the extent to which inclusion of a particular test in the selection system will actually improve selection. More specifically, the tables provide an estimate of the percentage of employees hired by the use of a particular test who will be successful at their jobs, given different combinations of three variables: the test’s validity, the selection ratio used, and the base rate.

Answer 19

Taylor-Russel Tables

Question 20

_____ limitations:

1. Relationship between the predictor (the test) and the criterion (rating of performance on the job) must be linear
2. Potential difficulty of identifying a criterion score that separates “successful” from “unsuccessful” employees.

Answer 20

Taylor-Russell tables

Question 21

_____ entails obtaining the difference between the means of the selected and unselected groups to derive an index of what the test (or some other tool of assessment) is adding to already established procedures.

Answer 21

Naylor-Shine tables

Question 22

Perhaps the most oft-cited application of statistical decision theory to the field of psychological testing is _____ Psychological Tests and Personnel

Answer 22

Cronbach and Gleser’s

Question 23

Stated generally, _____ presented

1. a classification of decision problems;
2. various selection strategies ranging from single-stage processes to sequential analyses;
3. a quantitative analysis of the relationship between test utility, the selection ratio, cost of the testing program, and expected value of the outcome; and
4. a recommendation that in some instances job requirements be tailored to the applicant’s ability instead of the other way around (a concept they refer to as adaptive treatment).

Answer 23

Cronbach and Gleser

Question 24

Generally, a _____ is the extent to which a particular trait, behavior, characteristic, or attribute exists in the population (expressed as a proportion).

Answer 24

base rate

Question 25

In psychometric parlance, a _____ may be defined as the proportion of people a test accurately identifies as possessing or exhibiting a particular trait, behavior, characteristic, or attribute

For example, _____ could refer to the proportion of people accurately predicted to be able to perform work at the graduate school level or to the proportion of neurological patients accurately identified as having a brain tumor.

Answer 25

hit rate

Question 26

In like fashion, a _____ may be defined as the proportion of people the test fails to identify as having, or not having, a particular characteristic or attribute. Here, a miss amounts to an inaccurate prediction

Answer 26

miss rate

Question 27

The category of misses may be further subdivided (2)

Answer 27

false positive (type 1 error) and false negative (type 2 error)

Question 28

The category of misses may be further subdivided (2)

A _____ is a miss wherein the test predicted that the testtaker did possess the particular characteristic or attribute being measured when in fact the testtaker did not.

Ex. A pregnancy test is positive, when in fact you aren’t pregnant

Answer 28

Type 1 false positive

Question 29

The category of misses may be further subdivided (2)

A _____ is a miss wherein the test predicted that the testtaker did not possess the particular characteristic or attribute being measured when the testtaker actually did.

Ex. a pregnancy test indicates a woman is not pregnant, but she is

Answer 29

Type 2 false negative

Question 30

The category of misses may be further subdivided (2)

A _____ is a miss wherein the test predicted that the testtaker did not possess the particular characteristic or attribute being measured when the testtaker actually did.

Ex. a pregnancy test indicates a woman is not pregnant, but she is

Answer 30

Type 2 false negative

Question 31

_____ provides guidelines for setting optimal cutoff scores. In setting such scores, the relative seriousness of making false-positive or false-negative selection decisions is frequently taken into
account.

It is concerned with how real decision-makers make decisions, and with how optimal decisions can be reached.

Answer 31

Decision theory

Question 32

Evidence of Construct Validity (7)

Answer 32

1) Evidence of homogeneity
2) Evidence of changes with age
3) Evidence of pretest–posttest changes
4) Evidence from distinct groups
5) Convergent evidence
6) Discriminant evidence
7) Factor analysis

Question 33

1) Evidence of homogeneity
2) Evidence of changes with age
3) Evidence of pretest-posttest changes
4) Evidence from distinct groups
5) Convergent evidence
6) Discriminant evidence
7) Factor analysis

Answer 33

Evidence of Construct Validity (7)

Question 34

Evidence of Construct Validity (7)

When describing a test and its items, homogeneity refers to how uniform a test is in measuring a single concept.

Correlations between subtest scores and total test score are generally reported in the test manual as _____.

One way a test developer can improve the homogeneity of a test containing items that are scored dichotomously (for example, true–false) is by eliminating items that do not show significant correlation coefficients with total test scores.

Coefficient alpha may also be used in estimating the homogeneity of a test composed of multiple-choice items

Answer 34

Evidence of homogeneity

Question 35

Evidence of Construct Validity (7)

_____, like evidence of test homogeneity, does not in itself provide information about how the construct relates to other constructs.

If a test score purports to be a measure of a construct that could be expected to change over time, then the test score, too, should show the same progressive changes with age to be considered a valid measure of the construct

Answer 35

Evidence of changes with age

Question 36

Evidence of Construct Validity (7)

Evidence that test scores change as a result of some experience between a _____ can be evidence of construct validity.

Answer 36

Evidence of pretest–posttest changes

Question 37

Evidence of Construct Validity (7)

Also referred to as the method of contrasted groups, one way of providing evidence for the validity of a test is to demonstrate that scores on the test vary in a predictable way as a function of membership in some group.

iba result ng test from groups

Answer 37

Evidence from distinct groups

Question 38

Evidence of Construct Validity (7)

Evidence for the construct validity of a particular test may converge from a number of sources, such as other tests or measures designed to assess the same (or a similar) construct.

Thus, if scores on the test undergoing construct validation tend to correlate highly in the predicted direction with scores on older, more established, and already validated tests designed to measure the same (or a similar) construct, this would be an example of _____.

similar constructs

Answer 38

Convergent evidence

Question 39

Evidence of Construct Validity (7)

A validity coefficient showing little (that is, a statistically insignificant) relationship between test scores and/or other variables with which scores on the test being construct-validated should not theoretically be correlated provides _____ of construct validity (also known as _____).

Marital Satisfaction Scale (MSS), its authors correlated scores on that instrument with scores on the Marlowe-Crowne Social Desirability Scale. Roach et al. hypothesized that high correlations between these two instruments would suggest that respondents were probably not answering items on the MSS entirely honestly but instead were responding in socially desirable ways.

The multitrait-multimethod matrix is the matrix or table that results from correlating variables (traits) within and between methods.

different constructs

Answer 39

discriminant evidence/validity

Question 40

Evidence of Construct Validity (7)

_____ is a shorthand term for a class of mathematical procedures designed to identify factors or specific variables that are typically attributes, characteristics, or dimensions on which people may differ.

In psychometric research, _____ is frequently employed as a data reduction method in which several sets of scores and the correlations between them are analyzed.

Answer 40

Factor analysis

Question 41

Evidence of Construct Validity (7)

_____ typically entails “estimating, or extracting factors; deciding how many factors to retain; and rotating factors to an interpretable orientation”.

Ex. the physical height, weight, and pulse rate of a human being.

Answer 41

Exploratory factor analysis

Question 42

Evidence of Construct Validity (7)

By contrast, in _____, “a factor structure is explicitly hypothesized and is tested for its fit with the observed covariance structure of the measured variables”

Answer 42

confirmatory factor analysis

Question 43

Evidence of Construct Validity (7)

_____ in a test conveys information about the extent to which the factor determines the test score or scores. A new test purporting to measure bulimia, for example, can be factor-analyzed with other known measures of bulimia, as well as with other kinds of measures (such as measures of intelligence, self-esteem, general anxiety, anorexia, or perfectionism).

High _____ by the new test on a “bulimia factor” would provide convergent evidence of construct validity. Moderate to low _____ by the new test with respect to measures of other eating disorders such as anorexia would provide discriminant evidence of construct validity.

Answer 43

Factor loading

Question 44

For the general public, the term _____ as applied to psychological and educational tests may conjure up many meanings having to do with prejudice and preferential treatment.

For federal judges, the term _____ as it relates to items on children’s intelligence tests is synonymous with “too difficult for one group as compared to another”.

For psychometricians, _____ is a factor inherent in a test that systematically prevents accurate, impartial measurement.

_____ implies systematic variation

Ex. if idiomatic cultural expressions—such as “an old flame” or “an apples-and-oranges comparison”—are used that may be unfamiliar to recently arrived immigrant students who may not yet be proficient in the English language or in American cultural

Answer 44

Test Bias

Question 45

If, for example, a test systematically underpredicts or overpredicts the performance of members of a particular group (such as people with green eyes) with respect to a criterion (such as supervisory rating), then it exhibits what is known as _____. _____ is a term derived from the point where the regression line intersects the Y-axis.

Ex. validity coefficients and criterion performance for different groups are the same, but their mean scores on the predictor differ.

Answer 45

intercept bias

Question 46

If a test systematically yields significantly different validity coefficients for members of different groups, then it has what is known as _____ —so named because the slope of one group’s regression line is different in a statistically significant way from the slope of another group’s regression line.

Ex. where the GIA predicts math achievement in Grades 1 to 4. Note. Females’ scores are underpredicted and males’ scores are overpredicted in lower scores, but after the interaction, males’ scores are underpredicted and females’ scores are overpredicted.

Answer 46

slope bias

Question 47

Simply stated, a _____ is a judgment resulting from the intentional or unintentional misuse of a rating scale.

Answer 47

Rating error

Question 48

At the other extreme is a _____. Movie critics who pan just about everything they review may be guilty of _____. Of course, that is only true if they review a wide range of movies that might consensually be viewed as good and bad.

Ex. sobrang negats dimunyu

Answer 48

severity error

Question 49

A _____ is, as its name implies, an error in rating that arises from the tendency on the part of the rater to be lenient in scoring, marking, and/or grading.

Ex. indicated by a tendency to rate all individuals at the high end of the scale kahit iba iba efficiency ng ratees

Answer 49

leniency error (also known as a generosity error)

Question 50

Another type of error might be termed a _____. Here the rater, for whatever reason, exhibits a general and systematic reluctance to giving ratings at either the positive or the negative extreme. Consequently, all of this rater’s ratings would tend to cluster in the middle of the rating continuum.

Ex. na-stuck sa gitna yu’ng scores

Answer 50

central tendency error

Question 51

One way to overcome what might be termed restriction-of-range rating errors (central tendency, leniency, severity errors) is to use _____, a procedure that requires the rater to measure individuals against one another instead of against an absolute scale. By using rankings instead of ratings, the rater (now the “ranker”) is forced to select first, second, third choices, and so forth.

Answer 51

rankings

Question 52

_____ describes the fact that, for some raters, some ratees can do no wrong. More specifically, a _____ may also be defined as a tendency to give a particular ratee a higher rating than he or she objectively deserves because of the rater’s failure to discriminate among conceptually distinct and potentially independent aspects of a ratee’s behavior.

Answer 52

Halo effect

Question 53

With that caveat in mind, and with exceptions most certainly in the offing, we will define _____ in a psychometric context as the extent to which a test is used in an impartial, just, and equitable way.

Answer 53

Test Fairness

Question 54

With that caveat in mind, and with exceptions most certainly in the offing, we will define _____ in a psychometric context as the extent to which a test is used in an impartial, just, and equitable way.

Answer 54

Test Fairness