PA3

Question 1

Define validity and validation

Answer 1

Validity: a judgment or estimate of how well a test measures what it purports to measure.
Validation: the process of gathering and evaluating evidence about validity

Question 2

What are the 4 main types of validity, and which of these is the most important “umbrella” term?

Answer 2

Face validity
Content validity
Criterion validity
Construct validity***

Question 3

What is face validity?

Answer 3

How relevant the test items APPEAR to be.

If a test appears to measure what it purports to measure “on the face of it,” it has high face validity.

Question 4

What is content validity?

Answer 4

How adequately a test samples the whole “universe” of the behaviour or criterion that the test was designed to sample (including the types of information to be covered, the number of items tapping each area of coverage, the organisation of the items in the test, etc.)

Question 5

Describe how Lawshe’s (1975) content validity ratio (CVR) works

Answer 5

1. Select a panel of experts in the content area
2. Ask them to rate each item as one of:
   a) essential
   b) useful but not essential
   c) not necessary
3. Use the content validity ratio (CVR) formula:
   If CVR is:
   • Negative= fewer than half panelists chose “essential”
   • Zero=exactly half panelists chose “essential”
   • Positive =more than half panelists chose “essential”

Question 6

Describe two ways content validity can be variable

Answer 6

1. Can change over time as construct research evolves

2. Can vary between cultures

Question 7

Define criterion related validity and criterion

Answer 7

Criterion related validity: a judgment of how adequately a test score correlates with some measure of interest (i.e. the criterion). The criterion is an accepted standard against which a test or a test score is evaluated.

Question 8

Describe and give examples of the two main types of criterion related validity

Answer 8

Concurrent validity: an index of the degree to which a test score is related to some criterion measured at the same time, eg. client does two tests at the same session, one of which is new and one is the “gold standard”.
Predictive validity: an index of the degree to which a test score predicts some criterion, or outcome, measured in the future eg. how well does an IQ test predict future exam results

Question 9

In criterion‐related validity, what is the validity coefficient?

Answer 9

A correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure.

Question 10

In criterion‐related validity, what is incremental validity?

Answer 10

The degree to which an added predictor in a test explains additional variation in the criterion (outcome) measure

Question 11

In criterion‐related validity, what is an expectancy table?

Answer 11

A table showing the proportions of people within different test‐score intervals who subsequently rated in various categories of the criterion (e.g. “passed” vs “failed” a job interview)

Question 12

What is construct validity?

Answer 12

The ability of a test to measure the theorized construct (e.g. intelligence, aggression, personality, etc.) that it purports to measure.
If a test is a valid measure of a construct, high scorers and low scorers should behave as theorized.
Al other types of validity “feed into” construct validity.

Question 13

What are convergent evidence and discriminant evidence, in relation to construct validity?

Answer 13

Convergent evidence: scores on test undergoing construct validation correlate highly in the predicted direction with scores on older, more established, tests measuring the same (or a similar) construct.
Discriminant evidence: validity coefficient shows little relationship between test scores and other variables with which scores on the test should NOT theoretically be correlated

Question 14

Apart from the main types of convergent and discriminant evidence, what are 4 other types of evidence for construct validity, and what do they mean?

Answer 14

1. Evidence of homogeneity: how uniform a test is in measuring a single concept.
2. Evidence of changes with age: some constructs are expected to change over time (e.g. reading rate).
   Evidence of pretest/posttest changes: test scores change as a result of some experience between a pretest and a posttest (e.g. therapy).
   Evidence from distinct groups: scores on a test vary in predictable way as function of membership to a group (e.g. impulsivity should be higher in substance users)

Question 15

Out of the different main types of validity, which is most closely and least closely linked to construct validity?

Answer 15

Most linked: criterion validity

Least linked: face validity

Question 16

In test validity, what does bias mean?

Answer 16

A factor inherent in a test that systematically prevents accurate, impartial measurement. Implies systematic variation (bad) as opposed to random variation (expected)

Question 17

In test validity, what does rating error mean? Describe two common types.

Answer 17

Rating error: judgment resulting from intentional or unintentional misuse of a rating scale eg:

1. Raters may be too lenient, too severe, or reluctant to give ratings at the extremes (“central tendency error”).
2. Halo effect: tendency to give particular person higher rating than they deserve because of a favourable overall impression.

Question 18

In test validity, what does “fairness” mean? Describe two common types.

Answer 18

The extent to which a test is used in an impartial, just, and equitable way.

Question 19

In test terminology, what is “utility”?

Answer 19

The usefulness or practical value of a test

Question 20

Describe three of the main factors effecting test utility

Answer 20

1. Psychometric soundness (generally, higher validity = greater utility. But valid tests not always useful.)
2. Costs:
   - Economic costs e.g., purchasing test, training etc
   • Non‐economic costs e.g., time, ethics etc
3. Benefits eg cheaper, better data, more reliable, more valid, better for a specific population

Question 21

In testing, what is utility analysis?

Answer 21

A family of techniques that entail a cost–benefit analysis to assist in decision about usefulness of the assessment tool.

Question 22

In utility analysis, what is expectancy data?

Answer 22

The likelihood that a test-taker will score within some interval of scores on a criterion measure. Various reference tables have been created to check expectancy data.

Question 23

In testing, what is a cut score and why is it’s accuracy important?

Answer 23

A score on a continuous scale that will be used to differentiate people into categorical outcomes.
Accuracy of cut score will affect reliability and validity of outcomes.

Question 24

What is a relative cut score?

Answer 24

A cut score determined in reference to normative data

Question 25

What is a fixed cut score?

Answer 25

A cut score based on a minimum acceptable level

Question 26

What are multiple cut scores?

Answer 26

The use of multiple cut points for a single predictor e.g. grades A, B, C, D; or categorized outcomes such as mild, moderate, severe

Question 27

In testing, what are multiple hurdles

Answer 27

A requirement to achieve a lower cut point or cut score before advancing to the next stage of testing

Question 28

What is the Angoff Method for setting cut scores?

Answer 28

Recruiting a panel of experts to sit the test as hypothetical clients with certain characteristics, and then averaging the expert scores to yield the cut scores for the test.

Question 29

What is the Known Groups Method for setting cut scores?

Answer 29

Collecting data on the predictor of interest from groups known to possess (and not to possess) a trait, attribute, or ability of interest. Works for some variables (eg clinical issues such as depression) but not others (eg impulsivity)

Question 30

What is the IRT method for setting cut scores?

Answer 30

Item Response Theory - based on item difficulty ratings. In order to “pass” the test, test taker must answer items deemed above some minimum level of difficulty worked out using IRT

Question 31

What is the “discriminant analysis” method for setting cut scores?

Answer 31

Statistical techniques used to quantify how well a set of identified variables (such as scores on a battery of tests) can predict membership to groups of interest

Question 32

What is the “receiver operating curves” method for setting cut scores, and what are the two main factors used in this method?

Answer 32

Derives the sensitivity and specificity associated with different cutpoints:
Sensitivity = proportion of people correctly identified as having condition
Specificity = proportion of people correctly identified as not having the condition

Question 33

What is the Youden Index?

Answer 33

Index used to select appropriate cutpoint based on

maximizing the sensitivity and specificity of a test

Question 34

What is a base rate and why is it important in setting a cutpoint?

Answer 34

Base rate = the true prevalence of the condition in the population. Base rate can have a big effect on the validity of different cutpoints.