Chapter 7: Utility

Question 1

We may define _____ in the context of testing and assessment as the usefulness or practical value of testing to improve efficiency.

In the language of psychometrics, _____ (also referred to as test _____) means much the same thing; it refers to how useful a test is. More specifically, it refers to the practical value of using a test to aid in decision-making

Answer 1

utility

Question 2

Factors That Affect a Test’s Utility (3)

Answer 2

1) Psychometric soundness
2) Costs
3) Benefits

Question 3

Factors That Affect a Test’s Utility:

A test is said to be _____ for a particular purpose if reliability and validity coefficients are acceptably high.

An index of reliability can tell us something about how consistently a test measures what it measures; and an index of validity can tell us something about whether a test measures what it purports to measure; but an index of utility can tell us something about the practical value of the information derived.

Answer 3

psychometrically sound

Question 4

Factors That Affect a Test’s Utility

Generally speaking, the higher the _____ of test scores for making a particular decision, the higher the utility of the test is likely to be.

Answer 4

criterion-related validity

Question 5

Would it be accurate to conclude that “a valid test is a useful test”?

Answer 5

no; it is not the case that “a valid test is a useful test”.

Question 6

Factors That Affect a Test’s Utility

In fact, one of the most basic elements in any utility analysis is the financial cost of the selection device (or training program or clinical intervention) under study

Answer 6

Cost

Question 7

Factors That Affect a Test’s Utility

_____ refers to profits, gains, or advantages. As we did in discussing costs associated with testing (and not testing), we can view benefits in both economic and noneconomic terms.

In industrial settings, a partial list of such noneconomic benefits—many carrying with them economic benefits as well—would include increase in quality of workers’ performance, increase in quantity of workers’ performance, decreases in time to train workers’ reduction in number of accidents, reduction in worker turnover.

Answer 7

Benefits

Question 8

A _____ may be broadly defined as a family of techniques that entail a cost–benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment.

In a most general sense, a _____ may be undertaken for the purpose of evaluating whether the benefits of using a test (or training program or intervention) outweigh the costs.

Answer 8

utility analysis

Question 9

How Is a Utility Analysis Conducted? (2)

Answer 9

1. Expectancy data

2. The Brogden-Cronbach-Gleser formula

Question 10

How Is a Utility Analysis Conducted? (2)

An _____ can provide an indication of the likelihood that a testtaker will score within some interval of scores on a criterion measure—an interval that may be categorized as “passing”, “acceptable”, or “failing”

Answer 10

expectancy table

Question 11

The _____ tables are formulated in terms of differences in average criterion score between the selected group and the original group; Taylor and Russell use differences in the percent successful between the selected group and the original group.

Answer 11

Naylor-Shine

Question 12

How Is a Utility Analysis Conducted? (2)

The independent work of Hubert E. Brogden and a team of decision theorists Cronbach and Gleser, has been immortalized in a formula for the dollar amount of a utility gain resulting from the use of a particular selection instrument under specified conditions.

In general, utility gain refers to an estimate of the benefit (monetary or otherwise) of using a particular test or selection method.

By the way, a modification of the _____ formula exists for researchers who prefer their findings in terms of productivity gains rather than financial ones. Here, productivity gain refers to an estimated increase in work output.

Estimate utility kung magkano matitipid sa company if may ginamit na particular test

Answer 12

Brogden-Cronbach-Gleser formula

Question 13

Some Practical Considerations: (3)

Answer 13

1) The pool of job applicants
2) The complexity of the job
3) The cut score in use

Question 14

Some Practical Considerations: (3)

Closely related to issues concerning the available pool of job applicants is the issue of how many people would actually accept the employment position offered to them even if they were found to be a qualified candidate

Answer 14

The pool of job applicants

Question 15

Some Practical Considerations: (3)

The more _____, the more people differ on how well or poorly they do that job. Whether or not the same utility models apply to jobs of varied complexity, and whether or not the same utility analysis methods are equally applicable, remain matters of debate.

Answer 15

The complexity of the job

Question 16

Some Practical Considerations: (3)

Also called a cutoff score, we have previously defined a cut score as a (usually numerical) reference point derived as a result of a judgment and used to divide a set of data into two or more classifications, with some action to be taken or some inference to be made on the basis of these classifications.

Answer 16

The cut score in use

Question 17

Some Practical Considerations: (3) The cut score in use

A _____ may be defined as a reference point—in a distribution of test scores used to divide a set of data into two or more classifications—that is set based on norm-related considerations rather than on the relationship of test scores to a criterion.

Ex. basic, proficient and so on

Answer 17

relative cut score

Question 18

Some Practical Considerations: (3) The cut score in use

In contrast to a relative cut score is the _____, which we may define as a reference point—in a distribution of test scores used to divide a set of data into two or more classifications—that is typically set with reference to a judgment concerning a minimum level of proficiency required to be included in a particular classification.

Fixed cut scores may also be referred to as absolute cut scores

Ex. hindi siya tulad ng norm-referenced, may criteria talaga like 20-30 are excellent

Answer 18

fixed cut score

Question 19

Some Practical Considerations: (3) The cut score in use

_____ refers to the use of two or more cut scores with reference to one predictor for the purpose of categorizing testtakers.

one collective element of a multistage decision-making process in which the achievement of a particular cut score on one test is necessary in order to advance to the next stage of evaluation in the selection process

Ex. may levels

Answer 19

Multiple cut scores

Question 20

Some Practical Considerations: (3) The cut score in use

In what is referred to as a _____, an assumption is made that high scores on one attribute can, in fact, “balance out” or compensate for low scores on another attribute.

Ex. allow for the candidate’s interview scores to compensate for their online assessment scores and potentially still be hired.

Answer 20

compensatory model of selection

Question 21

Methods for Setting Cut Scores: (4.c)

Answer 21

1) The Angoff Method
2) The Known Groups Method
3) IRT-Based Methods
4) Other Methods
a) method of predictive yield.
b) Regression
c) Discriminant analysis (also referred to as discriminant function analysis)

Question 22

Methods for Setting Cut Scores: (4.c)

Devised by William _____, this method for setting fixed cut scores can be applied to personnel selection tasks as well as to questions regarding the presence or absence of a particular trait, attribute, or ability.

The Achilles heel of the _____ method is when there is low inter-rater reliability and major disagreement regarding how certain populations of testtakers should respond to items.

Ex. relies on subject-matter experts (SMEs) who examine the content of each test question (item) and then predict how many minimally-qualified candidates would answer the item correctly. The average of the judges’ predictions for a test question becomes its predicted difficulty.

Answer 22

The Angoff Method

Question 23

Methods for Setting Cut Scores: (4.c)

Also referred to as the method of contrasting groups, the _____ entails collection of data on the predictor of interest from groups known to possess, and not to possess, a trait, attribute, or ability of interest.

The main problem with using known groups is that determination of where to set the cutoff score is inherently affected by the composition of the contrasting groups. No standard set of guidelines exist for choosing contrasting groups.

For example, a group of individuals known to be not depressed should have lower scores on a depression scale then the group known to be depressed.

Answer 23

The Known Groups Method

Question 24

Methods for Setting Cut Scores: (4.c)

In the _____, each item is associated with a particular level of difficulty. In order to “pass” the test, the testtaker must answer items that are deemed to be above some minimum level of difficulty, which is determined by experts and serves as the cut score.

Answer 24

IRT-Based Methods

Question 25

Methods for Setting Cut Scores: (4.c) IRT-Based Methods

A technique that has found application in setting cut scores for licensing examinations is the _____. It entails the arrangement of items in a histogram, with each column in the histogram containing items deemed to be of equivalent value.

Judges who have been trained regarding minimal competence required for licensure are presented with sample items from each column and are asked whether or not a minimally competent licensed individual would answer those items correctly about half the time. If so, that difficulty level is set as the cut score; if not, the process continues until the appropriate difficulty level has been selected.

Answer 25

Item-mapping method

Question 26

Methods for Setting Cut Scores: (4.c) IRT-Based Methods

An IRT-based method of setting cut scores that is more typically used in academic applications is the _____.

Ex. How are test items in the _____ organized?
Each rater receives an ordered item booklet. The raters go through the entire booklet once to read every item. They then go back through and place a bookmark between the two items in the booklet that represent the cut point for what minimally qualified participants should know and be able to do.

Answer 26

Bookmark method

Question 27

Methods for Setting Cut Scores: (4.c) Other Methods (3)

Answer 27

a) method of predictive yield.
b) Regression
c) Discriminant analysis (also referred to as discriminant function analysis)

Question 28

Methods for Setting Cut Scores: (4.c) Other Methods (3)

Edward L. Thorndike proposed a norm-referenced method for setting cut scores called the method of _____.

His contribution in the area of personnel psychology came in the form of a book entitled Personnel Selection. In that book he described a method of predictive yield, a technique for identifying cut scores based on the number of positions to be filled. The method takes into account projections regarding the likelihood of offer acceptance, the number of position openings, and the distribution of applicant scores.

Answer 28

predictive yield

Question 29

Methods for Setting Cut Scores: (4.c) Other Methods (3)

Wayne Cascio, a prolific researcher in the area of test utility, proposed the use of _____for setting cut scores when criterion-related data are available.

Answer 29

regression

Question 30

Methods for Setting Cut Scores: (4.c) Other Methods (3)

_____ (also referred to as _____ function _____) is a family of statistical techniques typically used to shed light on the relationship between certain variables (such as scores on a battery of tests) and two (and in some cases more) naturally occurring groups (such as persons judged to be successful on the job and persons judged not to be successful on the job.

Ex. a doctor could perform a discriminant analysis to identify patients at high or low risk for stroke.

Answer 30

Discriminant analysis