Chapter 7: Utility

Question 1

Utility

Answer 1

Usefulness or practical value of testing to improve efficiency; used to refer to the usefulness or practical value of a training program or intervention

Question 2

Psychometric Soundness

Answer 2

Reliability and valibidy of a test; reliability and validity coefficients are acceptably high

Question 3

Costs

Answer 3

Financial cost of the selection device under study; expenditures associated with testing or not testing

Question 4

Allocation of Funds is Needed to Purchase

Answer 4

A particular test
A supply of blank test protocols
Computerized test processing, scoring, and interpretation from the test publisher or some independent service

Question 5

Associated Costs of Testing

Answer 5

Payment to professional personnel and staff associated with test administration, scoring, and interpretation
Facility rental, mortgage, and/or other charges related to the usage of the test facility
Insurance, legal, accounting, licensing, and other routine costs of doing business

Question 6

Benefits

Answer 6

Profits, gains, or advantages accrued from testing

Question 7

Examples of Noneconomic Benefits in Industrial settings

Answer 7

Increase in quality of workers’ performance
Increase in quantity of workers’ performance
Decreases in time to train workers
Reduction in number of accidents
Reduction in worker turnover

Question 8

Utility Analysis

Answer 8

Broadly defined as a family of techniques that entail a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment

Question 9

Family of Techniques

Answer 9

Utility analysis is an umbrella term covering various possible methods, each requiring various kinds of data to be inputted and yielding arious kinds of output

Question 10

Utility Analysis Undertaken for the Purpose of Evaluating a Test and make decisions regarding whether

Answer 10

One test is preferable to another test for use for a specific purpose
One tool of assessment (such as a test) is preferable to another tool of assessment (such as behavioral observation) for a specific purpose
The addition of one or more tests (or other tools of assessment) to one or more tests (or other tools of assessment) that are already in use is preferable for a specific purpose
No testing or assessment is preferable to any testing or assessment

Question 11

Utility Analysis Undertaken for the Purpose of Evaluating a Training Program or Intervention will help make decision regarding whether

Answer 11

One training program is preferable to another training program
One method of intervention is preferable to another method of intervention
The addition or subtraction of elements to an existing training program improves the overall training program by making it more effective and efficient
The addition or subtraction of elements to an existing method of intervention improves the overall intervention by making it more effective and efficient
No training program is preferable to a given training program
No intervention is preferable to a given intervention

Question 12

General Aproaches to Utility Analysis

Answer 12

Expectancy Data

Brogden-Cronbach-Gleser Formula

Question 13

Expectancy Data

Answer 13

Will show that a test is working as it should and, by instituting this new test on a permanent basis, the company could reasonably expect to improve its productivity

Question 14

Expectancy Table

Answer 14

Provides an indication of measure - an interval that may be categorized as parring, acceptable, or failing

Question 15

Brogden-Cronbach-Gleser Formula

Answer 15

Developed a formula for the dollar amount of a utility gain resulting from the use of a particular selection instrument under specified conditions

Question 16

Utility gain

Answer 16

Refers to an estimate of the benefit (monetary or otherwise) of using a particular test or selection method, taking into consideration the following factors:
number of applicants selected per year
Average length of time in the position
Criterion-Related Validity coefficient for the given predictor and criterion
Standard deviation of performance in Dollars or employees
Mean (standardized) score on the test for selected applicants

Question 17

Productivity Gain

Answer 17

Refers to an estimated increase in work output

Question 18

Practical Considerations

Answer 18

Pool of Job Applicants
Complexity of the Job
Cut Score in Use

Question 19

Pool of Job Applicants

Answer 19

Issue of how many people would actually accept the employment position offered to them even if they were found to be a qualified candidate; Estimates of people who will actually accept the position should be adjusted downward as much as 80% in order to provide a more realistic estimate of the utility of a tool of assessment used for selection purposes

Question 20

Complexity of the Job

Answer 20

The more complex the job, the more people differ on how well or poorly they do that job

Question 21

Cut Score

Answer 21

Numerical reference point derived as a result of a judgment and used to dived a set of data into two or more classifications, with some action to be taken or some inference to be made on the basis of these classifications

Question 22

Relative Cut Score

Answer 22

Reference point, in a distribution of test scores used to divide a set of data into two or more classifications - that is set based on norm-related considerations rather than on the relationship of test scores to a criterion

Question 23

Norm-Referenced Cut Score

Answer 23

Set with reference to the performance of a group (or some target segment of a group)

Question 24

Fixed Cut Score (Absolute Cut Scores)

Answer 24

Reference point - in a distribution of test scores used to divide a set of data into two or more classifications - that is typically set with reference to a judgment concerning a minimum level of profieciency required to be included in a particular classification;

Question 25

Multiple Cut Scores

Answer 25

Refers to the use for the purpose of categorizing testtakers

Question 26

Multistage/Multiple Hurdle Selection Process

Answer 26

At every stage, a cut score is in place for each predictor used; Cut score used for each predictor will be assigned to ensure that each applicant possess some minimum level of a specific attribute or skill; assumes that an individual must possess a certain minimum amount of knowledge, skill, or ability for each attribute measured by a predictor to be successful in a desired position

Question 27

Multiple Hurdles

Answer 27

One collective element of a multistage decision-making process in which the achievement of a particular cut score on one test is necessary in order to advance to the next stage of evaluation in the selection process

Question 28

Written Application Stage

Answer 28

Completeness of the filling out of the form may be used to disqualify the applicant

Question 29

Additional Materials

Answer 29

Individuals with low test scores, GPA’s or poor letters of recommendation are eliminated

Question 30

Personal Interview

Answer 30

Entails unique demands to be successfully met

Question 31

Compensatory Model of Selection

Answer 31

An assumption is made that high scores on one attribute can balance out or compensate for low scores on another attribute; a person strong in some areas relevant to the position in question

Question 32

William Angoff

Answer 32

Devised the Angoff Method

Question 33

Angoff Method

Answer 33

Method for setting fixed cut scores applicable to personnel selection tasks as well as to questions regarding the presence or absence of a particular trait, attribute, or ability; judgments of experts are averaged to yield cut scores for the test; weak when there is low inter-rated reliability and major disagreement regarding how certain populations of testtakers should respond to items

Question 34

Known Groups Method/Method of Contrasting Groups

Answer 34

Entails the collection of data on the predictor of interest from groups know to possess, and not to possess, a trait, attribute, or ability of interest

Question 35

IRT-Based Methods

Answer 35

Cut scores are typically set based on testtakers’ performance across all items on the test; some portion of the total number of items on the test must be scorred correct in order for the testtaker to pass the test; each item is associated with a particular level of difficulty; in order to pass the test, the testtaker must answer items that are deemed to be above some minimum level of difficulty, which is determined by experts and serves as the cut score

Question 36

Item-Mapping Method

Answer 36

Entails the arrangement of items in a histogram, which each column in the histogram containing items deemed to be of equivalent value; the process involves several rounds of judgments in which experts may receive feedback regarding how their ratings compare to ratings made by other experts

Question 37

Bookmark Method

Answer 37

Typically used in academic applications; begins with the training of experts with regard to the minimal, knowledge, skills, and/or abilities that testtakers should possess in order to pass; Level of difficulty to use as the cut score is decided upon by the test developers

Question 38

Steps Involved in the Bookmark Method

Answer 38

Experts are trained with regard to the minimal knowledge, skills, and/or abilities that testtakers should possess in order to pass
Experts are given a book of items with one item printed per page, items are arranged in an ascending order of difficulty
Experts place a bookmark between two pages that are deemed to separate testtakers who have acquired the minimal knowledge, skills, and/or abilities from those have not
Bookmark serves as the cut score
Additional rounds of bookmarking with the same or other judges may take place as necessary
Feedback regarding placement may be provided, and discussion among experts about the bookmarking may be allowed
The level of difficulty to use as the cut score is decided upon by the test developers

Question 39

Problem with Bookmarking Method

Answer 39

Training of experts
Possible floor
Ceiling effects
Optimal length of item booklets

Question 40

Edward L. Thorndike

Answer 40

Proposed a norm-referenced method for setting cut scores; called the Method of predictive yield

Question 41

Wayne Cascio

Answer 41

Proposed the use of regression for setting cut scores when criterion-related data are available

Question 42

Discriminant Analysis/Discriminant Function Analysis

Answer 42

Family of statistical techniques typically used to shed light on the relationship between certain variables (such as scores on a battery of test) and two (and in some cases more) naturally occuring groups (such as persons judged to be successful on the job and persons judged not to be successful on the job