Module 4: Utility

Question 1

Utility

Answer 1

usefulness or practical value of testing to improve efficiency

Question 2

What can utility identify?

Answer 2

can tell us something about the practical value of the information derived from scores on the test

Question 3

What can utility help with?

Answer 3

Helps us make better decisions

Question 4

What makes higher utility?

Answer 4

Higher criterion-related validity = higher utility

Question 5

What is one of the most basic elements in utility analysis?

Answer 5

One of the most basic elements in utility analysis is financial cost of the selection device

Question 6

Cost

Answer 6

disadvantages, losses, or expenses both economic and noneconomic terms

Question 7

Benefit

Answer 7

profits, gains or advantages

Question 8

When does the cost of test administration become worth it?

Answer 8

The cost of test administration can be well worth it if the results is certain noneconomic benefits

Question 9

Utility Analysis

Answer 9

family of techniques that entail a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment

Question 10

Expectancy table

Answer 10

provide an indication that a testtaker will score within some interval of scores on
a criterion measure – passing, acceptable, failing

Question 11

What might utility predict?

Answer 11

Might indicate future behaviors, then if successful, the test is working as it should

Question 12

Taylor-Russel Tables

Answer 12

provide an estimate of the extent to which inclusion of a particular test in the selection system will improve selection

Question 13

Selection Ratio

Answer 13

numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired

Question 14

Selection Ratio Formula

Answer 14

number of hired candidates / total number of candidates

Question 15

Base Rate

Answer 15

percentage of people hired under the existing system for a particular position

Question 16

What is one limitation of Taylor-Russel Tables?

Answer 16

One limitation of Taylor-Russel Tables is that the relationship between the predictor (test) and criterion must be linear

Question 17

Naylor-Shine Tables

Answer 17

entails obtaining the difference between the means of the selected and unselected groups to derive an index of what the test
is adding to already established procedures

Question 18

Brogden-Cronbach-Gleser Formula

Answer 18

used to calculate the dollar amount of a utility gain resulting from the use of a particular selection instrument

Question 19

Utility Gain

Answer 19

estimate of the benefit of using a particular test

Question 20

Productivity Gains

Answer 20

an estimated increase in work output

Question 21

What does it mean when a job is more complex?

Answer 21

The more complex the job, the more people differ on how well or poorly they do that job

Question 22

Cut Score

Answer 22

reference point derived as a result of a
judgement and used to divide a set of data into two or more classifications

Question 23

Types of Cut Scores

Answer 23

1. Relative Cut Score
2. Fixed Cut Scores
3. Multiple Cut Scores
4. Multiple Hurdle
5. Compensatory Model of Selection

Question 24

Relative Cut Score

Answer 24

reference point based on norm-related considerations (norm-referenced), not fixed per se

Question 25

Fixed Cut Scores

Answer 25

set with reference to a judgement concerning minimum level of proficiency required ## Footnote e.g., Board Exams

Question 26

Multiple Cut Scores

Answer 26

refers to the use of two or more cut scores with reference to one predictor for the purpose of categorization

Question 27

Multiple Hurdle

Answer 27

multi-stage selection process, a cut score is in place for each predictor

Question 28

Compensatory Model of Selection

Answer 28

assumption that high scores on one attribute can compensate for lower scores

Question 29

Angoff Method

Answer 29

+ setting fixed cut scores + low interrater reliability + requires expert judges to discuss the issues involved in determining a pass mark and to evaluate the examination by using a well-defined and rational procedure

Question 30

Known Groups Method

Answer 30

+ collection of data on the predictor of interest from group known to possess and not possess a trait of interest + the determination of where to set cutoff score is inherently affected by the composition of contrasting groups

Question 31

IRT-Based Methods

Answer 31

cut scores are typically set based on testtaker’s performance across all the items on the test

Question 32

Types of IRT-Based Methods

Answer 32

1. Item-Mapping Method 2. Bookmark Method

Question 33

Item-Mapping Method

Answer 33

arrangement of items in histogram, with each column containing items with deemed to be equivalent value

Question 34

Bookmark Method

Answer 34

expert places “bookmark” between the two pages that are deemed to separate testtakers who have acquired the minimal knowledge, skills, and/or abilities from those who are not

Question 35

Method of Predictive Yield

Answer 35

took into account the number of positions to be filled, projections regarding the likelihood of offer acceptance, and the distribution of applicant scores

Question 36

Discriminant Analysis

Answer 36

shed light on the relationship between identified variables and two naturally occurring groups

Question 37

What is discriminant analysis used for?

Answer 37

used by the researcher to analyze the research data when the criterion or the dependent variable is categorical and the predictor or the independent variable is interval in nature

Question 38

What does discriminant analysis enable?

Answer 38

enables the researcher to examine whether significant differences exist among the groups, in terms of the predictor variables

Question 39

What does the discriminant analysis identify?

Answer 39

identify two groups of people who represent two distinct categories of some trait