Module 3: Validity and Utility

Question 1

Validity:

Answer 1

a judgement or estimate of how well a test measures what is purports to measure.

Does it measure what is says it measures/

Question 2

Validation:

Answer 2

the process of gathering and evaluating evidence about validity.

Question 3

Face Validity:

Answer 3

the test appears to cover relevant content.

Question 4

Content validity: See Notes

Answer 4

based on the evaluation of content covered by a test.

Question 5

Criterion validity:

Answer 5

obtained by evaluating relationship between scores on your test and other tests/measures.

Question 6

Construct validity:

Answer 6

the super ordinate the over ties all validity- all validity is construct validity.

arrived at by comprehensive analysis of:

a. How scores on the test relate to other test scores and measures, and
b. How scores on the test can be understood within some theoretical framework for understanding construct test was designed to measure.

Question 7

Face validity:

Answer 7

a judgment concerning how relevant the test items appear to be.
• If a test appears to measure what it purports to measure on the face of it, it has face validity.
• The patient could be offended/turned off if it doesn’t have face validity.
• It is not necessary.

Question 8

Test blueprint:

Answer 8

a plan regarding the types of information to be covered by the items, the number of items tapping each area of coverage, the organization of the items in the test

Question 9

Criterion- related validity:

Answer 9

a criterion is the standard against which a test or a test score is evaluated.
• Characteristics of an adequate criterion:
o A relevant for the matter at hand.
o Valid for the purpose for which it is being used.
o Uncontaminated (i.e., it is not part of the predictor).

Question 10

The validity coefficient:

Answer 10

a correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure.

Question 11

Incremental validity:

Answer 11

the degree to which an additional predictor explains additional variation in the criterion measure.

Question 12

2Construct validity:

Answer 12

ability to test to measure theorized construct (e.g., intelligence, aggression, personality, etc.) that it purports to measure.
• If test is validity measure of construct, high scores and low scorers should behave as theorized.
• All types of validity evidence, including evidence from the content and criterion-related varieties of validity, come under the umbrella of construct validity.

Question 13

Evidence of construct validity:

Answer 13

• Evidence of homogeneity: how uniform a test is in measuring a single concept.
• Evidence of changes with age: some constructs are expected to change over time (e.g., reading late).
• Evidence of pretest/post-test changes: test scores change as a result of some experience between a pretest and post-test (e.g., therapy).
• Evidence from distinct groups: scores on a test vary in predictable way as function of membership to a group (e.g., impulsivity should be higher in substance users).
• Convergent evidence: scores on a test undergoing construct validation tend to correlate highly in predicted direction with scores on older, more established, tests designed to measure the same (or similar) construct.
• Discriminant evidence: validity coefficients show little relationship between test scores and other variables with which scores on the test should not theoretically be correlated.
o WE DO NOT NEED FACE VALIDITY

Question 14

Bias:

Answer 14

a factor inherent in a test that systematically prevents accurate, impartial measurement.
• Bias implies systematic variation in test scores.
• Only want random error.
• DO not want systematic error.

Question 15

Fairness:

Answer 15

the extent to which a test is used in an impartial, just, and equitable way.

Question 16

Rating error:

Answer 16

judgment resulting from intentional or unintentional misuse of a rating scale.
• Raters may be too lenient, too severe, or reluctant to give ratings at the extremes (central tendency error).
• Halo effect: tendency to give particular person higher rating than objectively deserves because of favorable overall impression.

Question 17

Halo effect:

Answer 17

tendency to give particular person higher rating than objectively deserves because of favorable overall impression.

Question 18

Utility:

Answer 18

the usefulness or practical value of testing to improve efficiency.

Question 19

Psychometric soundness:

Answer 19

• Generally, high validity = greater utility.
• But many factors affect utility and utility assessed in different ways.
• Valid tests not always useful

Question 20

Types of Costs:

Answer 20

• Economic costs? E.g., purchasing test and scoring sheeting, training programs, software/hardware, cost of not using the best test.
• Non-economic costs? E.g., time, ethical considerations, face validity, poor data acquisition.

Question 21

Utility analysis:

Answer 21

family of techniques that entail a cost-benefit analysis to assist in decision about useful of assessment tool.
• Some utility tests straightforward, others are more sophisticated (e.g., using meth models).
• Often utility tests address the question of “which test gives us the most bang for the buck?
• Endpoint of utility analysis yields educated decision as to which of several alternative courses of action is most optimal (in terms of costs and benefits).

Question 22

Expectancy data:

Answer 22

likelihood that a test taker will score within some interval of scores on a criterion measure.

Question 23

Cut scores:

Answer 23

what score will be used to differentiate people on your test? I.e., only for categorical outcomes.

Question 24

Relative cut scores:

Answer 24

determined in reference to normative data.

Question 25

Fixed cut scores

Answer 25

made on basis of minimum acceptable level. E.g., driving test

Question 26

Multiple cut scores:

Answer 26

use of multiple cut points for a single predictor (e.g., grades A, B, C, etc., categorized outcomes mild, moderate).

Question 27

Multiple hurdles:

Answer 27

need to achieve lower cut point before advancing to next stage of testing.

Question 28

Methods of setting cut scores:

Answer 28

• The Angoff method
• The Known groups method
• IRT Based Methods
• Discriminant analysis
• Receiver operating curves
• Youden index

Question 29

The Angoff method:

Answer 29

judgements of experts are averaged to yield cut scores for the test.

Question 30

The Known groups method

Answer 30

entails collection of data on the predictor of interest from groups known to possess, and not to possess, a trait, attribute, or ability of interest.
o After analysis of data, a cut score is chosen that best discriminates the groups.
o One problem with known groups method is how do you know which “known groups” to select?

Question 31

IRT Based Methods:

Answer 31

use the item difficulty parameter.
o In order to “pass” the test, test taker must answer items deemed above some minimum level of difficulty (hence a specific point along the latent trait (theta).

Question 32

Discriminant analysis:

Answer 32

statistical techniques used to quantify how well a set of identified variables (such as scores on a battery of tests) can predict membership to groups of interest.

Question 33

Receiver operating curves

Answer 33

derives the sensitivity and specificity associated with different cut points that classify individuals as having or not having a condition of interest.

o Sensitivity: proportion of people correctly identified as having condition.
o Specificity: proportion of people correctly identified as not having the condition.

Question 34

Youden index:

Answer 34

can be used to select appropriate cut point based on maximising sensitivity and specificity of a test.

Question 35

Why Cut Points Matter:

Answer 35

• Accuracy of classification will affect reliability and validity.
• Need to consider base rates when selecting cut points and interpreting scores.
o Base rate: the true prevalence of the condition in the population