Chapter 7: Utility Flashcards
this term refers to how useful a test
is. More specifically, it refers to the practical value of using a test to aid in decision making.
In addition, in the context of testing and assessment as the usefulness or practical value of testing to improve efficiency.
Utility (test utility)
what are the indexes related to psychometric properties?
- Index of Reliability
- Index of Validity
- Index of Utility
this term refers to the reliability and validity of a test, indicating its overall quality for a specific purpose.
Psychometric Soundness
this index reflects how consistently a
test measures what it is intended to measure.
this index indicates the degree to which
a test measures what it claims to measure.
this index measures the practical value or usefulness of the information derived from test scores, especially in aiding decision-making
Index of Utility
what are the costs associated with test utility?
- Economic Costs
- Personnel and Facility Costs
- Additional Business Costs
this term refers to the profits, gains, or advantages obtained from administering a test.
are weighed against costs to assess the overall value and effectiveness of using a particular test.
benefits
what is benefit in the context of test utility?
benefit refers to profits, gains, or
advantages in both economic and noneconomic terms.
what is cost in the context of test utility?
cost refers to disadvantages, losses, or expenses in both economic and noneconomic terms.
this term may be be broadly defined as a family of techniques that entail a cost–benefit analysis designed to yield information relevant to a decision about the usefulness and/or practical value of a tool of assessment
is an umbrella term covering various possible methods, each requiring various kinds of data to be inputted and yielding various kinds of output
utility analysis
this type of utility analsis instrument can provide an indication of the likelihood that a testtaker will score within some interval of scores on a criterion measure—an interval that may be categorized as “passing,” “acceptable,” or “failing.”
expectancy table/chart (scatterplot and analysis regression)
what does a hit (true positive) mean?
a test score correctly identifies the condition of interest and implies that a qualified employee was hired
what does a correct rejection (true negative) mean?
the test score correctly identifies the absence of the condition of interest and implies that an unqualified employee was not hired
what does a miss (false negative) mean?
the test score failed to identify a trait or condition of interest and implies that a qualified employee was not hired
what does a false alarm (false positive) mean?
the test score incorrectly identifies the condition of interest and implies that a unqualified employee was hired
what is base rate (prevalence)?
base rate is the proportion of individuals with the condition of interest
this term refers to an estimate of the benefit (monetary or otherwise) of using a particular test or selection method.
utility gain
this type of utility analysis instrument provides an estimate of the extent to which inclusion of a particular test in the selection system will improve selection.
provides the personnel officer with
an estimate of how much using the test would improve selection over existing methods.
determines the increase over current procedures
Taylor-Russell Tables
what is selection ratio?
selection ratio is the proportion of individuals indicating the presence of the condition of interest
this term is a numerical value that reflects the relationship between the number of people to be hired and the number of people available to be hired
For instance, if there are 50 positions
and 100 applicants, then the ________ _____ is 50/100, or .50
selection ratio
this utility analysis instrument provided an indication of the difference in average criterion scores for the selected group as compared with the original group in response to the potential problems of the Taylor-Russell tables
the use of this entails obtaining the difference between the means of the selected and unselected groups to derive an index of what the test (or some other tool of assessment) is adding to already established procedures.
determines the increase in average score on some criterion measure
Naylor-Shine tables
this term refers to the percentage of people hired under the existing system for a particular position
For example, a firm employs 25 computer programmers and 20 are considered successful, the ____ ____ would be .80
base rate
what are the limitations of the Taylor-Russell tables?
- if job performance levels off at some point in time, then the Taylor-Russell tables would be inappropriate no matter how high the score is
- the potential difficulty of identifying a criterion score that separates “successful” from “unsuccessful” employees
this type of utility analysis instrument is used to calculate the dollar amount of a utility gain resulting from the use of a particular selection instrument under specified conditions
Brogden-Cronbach-Gleser
this utility analysis instrument is also used to predict performance and is under the expectancy tables or charts category
regression analysis
this term refers to a set of cost-benefit analysis techniques designed to assess the usefulness or practical value of an assessment tool, training program,
or intervention.
utility analysis
this utility analysis provides estimates of how adding a particular test to a selection system may enhance selection quality.
Taylor-Russel Tables
what is the purpose of utility analysis in test evaluation?
Utility analysis can guide decisions on whether:
○ One test is preferable over another for
a specific purpose.
○ One type of assessment tool (e.g., a
test) is more useful than another (e.g.,
behavioral observation) for the
intended purpose
○ Adding new assessments to an existing
process is beneficial
this term refers to a value indicating the
proportion of positions available
relative to the number of applicants.
Selection Ratio
this utility analysis is used to estimate the
likelihood that a test-taker will achieve scores within specific intervals (e.g., “passing,” “acceptable,” “failing”) on a criterion measure.
Expectancy Tables
what is a limitation of the Taylor-Russel Tables
Requires a linear relationship between the test (predictor) and performance (criterion) and a clear distinction between
“successful” and “unsuccessful” outcomes.
this utility analysis tool calculates utility gain, often as a monetary estimate, from
implementing a specific selection tool, helping determine the financial benefit of using a test.
Brogden-Cronbach-Gleser Formula
this utility analysis tool helps calculate the added value of a test by measuring the difference between the performance means of selected and unselected groups
Naylor-Shine Tables
this term refers to an estimated benefit (monetary or otherwise) from using a test or selection method, often in terms of productivity gain, or the increased
Utility Gain
this theory offers a framework for setting
optimal cutoff scores, helping maximize test utility by determining the most effective thresholds for selection or classification decisions
Decision Theory
this term refers to a numerical reference point used to divide test scores into classifications, guiding actions or
inferences based on these divisions.
Cut Scores
what are the five types of cut scores?
- Relative Cut Score (Norm-referenced)
- Fixed Cut Score (Absolute)
- Multiple Cut Scores
- Multiple Hurdles
this type of cut score is set based on the performance of a group rather than on an absolute standard.
It divides scores in relation to the norm group’s performance
relative cut score
this type of cut score involves using two
or more cut scores within one predictor
to categorize test-takers into distinct
groups or performance levels
Multiple Cut Scores
this selection model allows for high scores in one area to compensate for lower scores in another, supporting a more flexible assessment approach.
Compensatory Model of Selection
this type of cut score is set based on a minimum level of required
proficiency, regardless of group
performance.
This standard is fixed and criterion-referenced.
Fixed Cut Score
this type of cut score is a multistage
approach where test-takers must meet
a cut score at each stage to advance,
often used in sequential selection
processes.
Multiple Hurdles
what are some factors that influence cut scores?
● The size and characteristics of the job applicant pool.
● The complexity of the job, which may dictate higher or more specific cut scores.
● The cut score in use, whether relative, fixed, or part of a multiple hurdle model.
what are the main methods for setting cut scores?
- Angoff Method
- Known Groups Method
- IRT-Based Methods
- Item-Mapping Method
- Bookmark Method
this method for setting fixed cut scores can be applied in personnel selection or for identifying the presence or absence of a particular trait or ability
Angoff Method
this method for setting cut scores collects data on a predictor from groups known
to possess or lack a particular trait, attribute, or ability.
A cut score is established based on which score best distinguishes between these two groups.
Known Groups Method
this method for setting cut scores involves creating a histogram of test items grouped by equivalent value in terms of difficulty or importance.
Item-Mapping Method
this method for setting cut scores has each test item tied to a specific difficulty level.
Test-takers must correctly answer items above a set minimum difficulty to “pass,” with experts determining this level as the cut score.
IRT-Based Methods
this method of setting cut scores considers the number of positions available, likely acceptance rates, and score distributions in setting cut scores.
Method of Predictive Yield
this method for setting cut scores has experts receive training on the
minimum required knowledge or skills needed to pass the test. They review a “book” of items arranged in ascending order of difficulty and place a bookmark between items separating those who meet the minimal requirements
from those who do not.
Multiple rounds of bookmmarking may occur, with experts allowed to discuss their decisions. The final difficulty level set as the cut score is agreed upon by test developers
Bookmark Method
this method of setting cut scores uses decision theory principles to guide cut score placement.
Decision-Theoretic Approach
this method of setting cut scores analyzes relationships between variables (e.g., test scores) and groups (e.g., successful vs. unsuccessful job candidates)to help determine an optimal cut score.
Discriminant Analysis
