Utility

Question 1

According to article, what’s wrong with traditional indexes like specificity, sensitivity, PPV, and NPV?

Answer 1

They vary based on cutoff scores, base rates, and cost/benefits with a particular context or assessment.

Question 2

What do the article authors encourage to do?

Answer 2

Use methods of signal detection theory, like ROC for utility

Question 3

What happens when cutoff scores change? Which of the following are affected? (Selection Ratio, Base Rate, Sensitivity, Specificity, Positive Predictive Power, Negative Predictive Power)

Answer 3

Only base rate does not change.

Question 4

What metrics are directly affected by the base rate?

Answer 4

PPV and NPV

Question 5

What is the ROC-based index is independent of?

Answer 5

1) base rate / prevalence
2) cutoff score
3) values of the 4 decision-making outcomes (i.e., TN TP FN FP)

Question 6

How to use ROC analysis?

Answer 6

Look at area under ROC curve. This shows discriminative power of an assessment independent of cutoff value

Question 7

What does it mean when AUC = 1?

What about when AUC = 0.5 or less than 0.5?

Answer 7

When AUC = 1, the model is perfect across all cutoffs. Sensitivity and Specificity both are 100%

When AUC = 0.5, it’s no better than guessing.

When AUC < 0.5, the test inverts values

Question 8

Test Utility

Answer 8

Usefulness or practical value of testing to improve efficiency

Question 9

3 factors for utility of a test (what makes it have high/low utility?)

Answer 9

1) psychometric soundness (validity + reliability)
2) costs
3) benefits

Question 10

Does utility require measuring something?

Answer 10

No, there can be utility without measurement

Question 11

Is utility an empirical or conceptual endeavour?

Answer 11

Empirical

Question 12

How are costs complicated?

Answer 12

It’s complicated to calculate costs due to nonmonetary costs. (eg. what are the costs of not administering a test, resulting in possibly failing to diagnose a disease?)

Question 13

What general procedure determines costs and benefits of a test?

Answer 13

Utility analysis

Question 14

Expectancy data

Answer 14

One way of doing utility analyses. Uses predictions of outcomes to judge the benefits of a test?

Question 15

What does a Taylor-Russell table accomplish?

Answer 15

A type of expectancy data.

Helps to see how much using a test will improve selection over existing method

Question 16

Naylor-Shine tables

Answer 16

Difference in means between selected and unselected groups to see what a test adds (increase in avg score)

Question 17

What is a Taylor-Russell table based on?

Answer 17

Selection Ratio and Validity estimate

Question 18

What formula is used when assessing costs and benefits of a test?

Answer 18

Brogden-Cronbach-Gleser formula.

Estimates utility gain based on dollar amount.

Question 19

Higher selection ratio means probably higher or lower false positives?

Answer 19

More false positives, less false negatives.

Question 20

Why is the pool of job applicants a practical consideration when determining utility of a test?

Answer 20

Utility analyses assume that there exists an unlimited supply of the type of people who is qualified according to the test.

Question 21

Why is the fact that some people won’t accept job offers a practical consideration for utility of a test?

Answer 21

It will overestimate utility if not considered. Some people will deny job offers. 80% of utility is a better estimate.

Question 22

Alternative name for relative cut score

Answer 22

Norm-referenced cut score

Question 23

Fixed vs. Relative cut scores

Answer 23

a specific number vs. a distribution or percentile.

Question 24

Multiple hurdle or multistage selection process

What’s multiple cut scores and how does that differ?

Answer 24

Ensures a tester must have a few predictors. Multiple cut scores

Multiple cut scores can refer to different categories of same predictor (eg. A, B, C, D grades have multiple cut scores)

Question 25

Compensatory model of selection

what does this compare with?

Answer 25

A high score on one predictor can balance out a low score of another

Compared to a multiple hurdle/multistage process.

Question 26

What statistical tool for compensatory model of selection?

Answer 26

Multiple regression

Question 27

Angoff Method

Answer 27

A method to determine cut scores.

Average experts judgements on expectations.

Question 28

What can show that Angoff method might not be best?

What are its weaknessess

Answer 28

Low interrater reliability (if the experts disagree).

Lack of data-driven techniques, more subjective.

Question 29

Known Groups Method

(alt name)

Answer 29

Method of contrasting Groups. A way to determine test scores.

Take scores of those known to possess and not possess some predictor.

Question 30

What’s a problem with known groups method?

Answer 30

The choosing of groups are arbitrary

Question 31

IRT-based methods

Answer 31

A way to determine cutoff scores.

Each item has some deemed level of difficulty. People must answer items that are above a certain difficulty.

Question 32

Item-mapping method

Answer 32

a way to make cut score with IRT-based method.

Arranges items on histogram, judges judge whether the questions will be answered right at least half the time = cut score.

Question 33

Bookmark Method

Answer 33

a way to make cut score with IRT-based method.

Experts place a ‘bookmark’ between two pages in a given booklet that separate minimum difficulty (cut off)

Question 34

Drawbacks of IRT-based methods

Answer 34

Need expert opinion, floor/ceiling effects, optimal numbers of items in booklet (for bookmark method)

Question 35

Method of predictive yield

Answer 35

Thorndike’s method of setting cut scores.

Needs posititions to be filled, projections of likelihood to offer acceptance, and distribution of applicant scores.

Question 36

Discriminant Analysis

Answer 36

See relationship between identified variables and two naturally occurring groups. (Discriminant Function Analysis)