Week 6 - Cut Off Scores

Question 1

Biddle, R. E. (1993). How to set cutoff scores for knowledge tests used in promotion, training, certification, and licensing. Public Personnel Management, 22, 63-79.

Answer 1

• Cutoff scores in knowledge tests delineate pass or fail for promotions, certifications, and licensing.
• Litigation influences cutoff setting processes, emphasizing job analysis and adverse impact analysis.
• The Angoff Method calculates minimum competency scores using Subject Matter Expert estimates, modified by statistical considerations.
• The Uniform Guidelines on Employee Selection Procedures provide vague guidance on setting cutoff scores.
• Legal and professional considerations influence the determination of cutoff scores, including job relatedness and the minimization of adverse impact.
• Statistical and practical significance in rate differences and pool differences guide the evaluation of adverse impact in test outcomes.
• Recommendations for setting cutoffs include using Subject Matter Expert opinions, considering job relatedness, consequences of not knowing test information, and statistical adjustments to mitigate adverse impact.
• The Supreme Court and Uniform Guidelines influence the process, emphasizing job relatedness, adverse impact analysis, and legal defensibility.

Question 2

Cascio, W.F., Alexander, R.A., & Barrett, G.V. (1988). Setting cutoff scores: Legal, psychometric, and professional issues and guidelines. Personnel Psychology, 41(1), 1-24.

Answer 2

• Cutoff scores are widely used in personnel decisions, but guidance on their derivation and use is dispersed across legal, psychometric, and professional fields.
• This paper critically analyzes and integrates information from these diverse literatures to clarify the use and misuse of cutoff scores.
• It examines alternative methods for setting cutoff scores in the context of legal precedents, psychometric principles, and professional guidelines.
• The paper suggests directions for future research to address gaps in understanding about cutoff scores.
• The authors conclude with recommendations for acceptable professional practice regarding the establishment and application of cutoff scores.

Question 3

Cizek, G. J. (2006). Standard setting. In S. M. Downing and T. M. Haladyna (Eds.). Handbook of test development (pp. 225-258). Lawrence Earlbaum.

Question 4

Ock & Oswald (2018) The Utility of Personnel Selection Decisions

Answer 4

The document by Ock & Oswald (2018) compares the utility of compensatory and multiple-hurdle selection models in personnel decisions. It highlights:

• Compensatory selection models generally offer higher reliability and utility than multiple-hurdle models.
• Despite higher costs, compensatory models are preferable for their overall effectiveness in selecting high-performing applicants.
• Multiple-hurdle models may be favored for cost and time efficiency, especially in less selective contexts.
• The study uses Monte Carlo simulations to illustrate trade-offs between reliability, cost, and utility in different selection scenarios.
• Results support compensatory models for higher complexity jobs where the return on selection accuracy is greatest.
• In lower complexity jobs or when selection costs are a concern, multiple-hurdle models may offer sufficient utility.
• This study provides insights into the strategic choice between selection models based on job complexity, cost considerations, and expected utility.