Chapter 7: Utility Flashcards
Utility
Usefulness or practical value of testing to improve efficiency; used to refer to the usefulness or practical value of a training program or intervention
Psychometric Soundness
Reliability and valibidy of a test; reliability and validity coefficients are acceptably high
Costs
Financial cost of the selection device under study; expenditures associated with testing or not testing
Allocation of Funds is Needed to Purchase
A particular test
A supply of blank test protocols
Computerized test processing, scoring, and interpretation from the test publisher or some independent service
Associated Costs of Testing
Payment to professional personnel and staff associated with test administration, scoring, and interpretation
Facility rental, mortgage, and/or other charges related to the usage of the test facility
Insurance, legal, accounting, licensing, and other routine costs of doing business
Benefits
Profits, gains, or advantages accrued from testing
Examples of Noneconomic Benefits in Industrial settings
Increase in quality of workers’ performance
Increase in quantity of workers’ performance
Decreases in time to train workers
Reduction in number of accidents
Reduction in worker turnover
Utility Analysis
Broadly defined as a family of techniques that entail a cost-benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment
Family of Techniques
Utility analysis is an umbrella term covering various possible methods, each requiring various kinds of data to be inputted and yielding arious kinds of output
Utility Analysis Undertaken for the Purpose of Evaluating a Test and make decisions regarding whether
One test is preferable to another test for use for a specific purpose
One tool of assessment (such as a test) is preferable to another tool of assessment (such as behavioral observation) for a specific purpose
The addition of one or more tests (or other tools of assessment) to one or more tests (or other tools of assessment) that are already in use is preferable for a specific purpose
No testing or assessment is preferable to any testing or assessment
Utility Analysis Undertaken for the Purpose of Evaluating a Training Program or Intervention will help make decision regarding whether
One training program is preferable to another training program
One method of intervention is preferable to another method of intervention
The addition or subtraction of elements to an existing training program improves the overall training program by making it more effective and efficient
The addition or subtraction of elements to an existing method of intervention improves the overall intervention by making it more effective and efficient
No training program is preferable to a given training program
No intervention is preferable to a given intervention
General Aproaches to Utility Analysis
Expectancy Data
Brogden-Cronbach-Gleser Formula
Expectancy Data
Will show that a test is working as it should and, by instituting this new test on a permanent basis, the company could reasonably expect to improve its productivity
Expectancy Table
Provides an indication of measure - an interval that may be categorized as parring, acceptable, or failing
Brogden-Cronbach-Gleser Formula
Developed a formula for the dollar amount of a utility gain resulting from the use of a particular selection instrument under specified conditions
Utility gain
Refers to an estimate of the benefit (monetary or otherwise) of using a particular test or selection method, taking into consideration the following factors:
number of applicants selected per year
Average length of time in the position
Criterion-Related Validity coefficient for the given predictor and criterion
Standard deviation of performance in Dollars or employees
Mean (standardized) score on the test for selected applicants
Productivity Gain
Refers to an estimated increase in work output
Practical Considerations
Pool of Job Applicants
Complexity of the Job
Cut Score in Use
Pool of Job Applicants
Issue of how many people would actually accept the employment position offered to them even if they were found to be a qualified candidate; Estimates of people who will actually accept the position should be adjusted downward as much as 80% in order to provide a more realistic estimate of the utility of a tool of assessment used for selection purposes
Complexity of the Job
The more complex the job, the more people differ on how well or poorly they do that job
Cut Score
Numerical reference point derived as a result of a judgment and used to dived a set of data into two or more classifications, with some action to be taken or some inference to be made on the basis of these classifications
Relative Cut Score
Reference point, in a distribution of test scores used to divide a set of data into two or more classifications - that is set based on norm-related considerations rather than on the relationship of test scores to a criterion
Norm-Referenced Cut Score
Set with reference to the performance of a group (or some target segment of a group)
Fixed Cut Score (Absolute Cut Scores)
Reference point - in a distribution of test scores used to divide a set of data into two or more classifications - that is typically set with reference to a judgment concerning a minimum level of profieciency required to be included in a particular classification;
Multiple Cut Scores
Refers to the use for the purpose of categorizing testtakers
Multistage/Multiple Hurdle Selection Process
At every stage, a cut score is in place for each predictor used; Cut score used for each predictor will be assigned to ensure that each applicant possess some minimum level of a specific attribute or skill; assumes that an individual must possess a certain minimum amount of knowledge, skill, or ability for each attribute measured by a predictor to be successful in a desired position
Multiple Hurdles
One collective element of a multistage decision-making process in which the achievement of a particular cut score on one test is necessary in order to advance to the next stage of evaluation in the selection process
Written Application Stage
Completeness of the filling out of the form may be used to disqualify the applicant
Additional Materials
Individuals with low test scores, GPA’s or poor letters of recommendation are eliminated
Personal Interview
Entails unique demands to be successfully met
Compensatory Model of Selection
An assumption is made that high scores on one attribute can balance out or compensate for low scores on another attribute; a person strong in some areas relevant to the position in question
William Angoff
Devised the Angoff Method
Angoff Method
Method for setting fixed cut scores applicable to personnel selection tasks as well as to questions regarding the presence or absence of a particular trait, attribute, or ability; judgments of experts are averaged to yield cut scores for the test; weak when there is low inter-rated reliability and major disagreement regarding how certain populations of testtakers should respond to items
Known Groups Method/Method of Contrasting Groups
Entails the collection of data on the predictor of interest from groups know to possess, and not to possess, a trait, attribute, or ability of interest
IRT-Based Methods
Cut scores are typically set based on testtakers’ performance across all items on the test; some portion of the total number of items on the test must be scorred correct in order for the testtaker to pass the test; each item is associated with a particular level of difficulty; in order to pass the test, the testtaker must answer items that are deemed to be above some minimum level of difficulty, which is determined by experts and serves as the cut score
Item-Mapping Method
Entails the arrangement of items in a histogram, which each column in the histogram containing items deemed to be of equivalent value; the process involves several rounds of judgments in which experts may receive feedback regarding how their ratings compare to ratings made by other experts
Bookmark Method
Typically used in academic applications; begins with the training of experts with regard to the minimal, knowledge, skills, and/or abilities that testtakers should possess in order to pass; Level of difficulty to use as the cut score is decided upon by the test developers
Steps Involved in the Bookmark Method
Experts are trained with regard to the minimal knowledge, skills, and/or abilities that testtakers should possess in order to pass
Experts are given a book of items with one item printed per page, items are arranged in an ascending order of difficulty
Experts place a bookmark between two pages that are deemed to separate testtakers who have acquired the minimal knowledge, skills, and/or abilities from those have not
Bookmark serves as the cut score
Additional rounds of bookmarking with the same or other judges may take place as necessary
Feedback regarding placement may be provided, and discussion among experts about the bookmarking may be allowed
The level of difficulty to use as the cut score is decided upon by the test developers
Problem with Bookmarking Method
Training of experts
Possible floor
Ceiling effects
Optimal length of item booklets
Edward L. Thorndike
Proposed a norm-referenced method for setting cut scores; called the Method of predictive yield
Wayne Cascio
Proposed the use of regression for setting cut scores when criterion-related data are available
Discriminant Analysis/Discriminant Function Analysis
Family of statistical techniques typically used to shed light on the relationship between certain variables (such as scores on a battery of test) and two (and in some cases more) naturally occuring groups (such as persons judged to be successful on the job and persons judged not to be successful on the job
