Evaluating a Predictor's Criterion-Related Validity

Question 1

reliability

Answer 1

the extent performance on a measure is unaffected by measurment error

Question 2

validity

Answer 2

does the measure assess what it was designed to assess

Question 3

types of validity

Answer 3

1. content
2. construct
3. criterion-related:
   
   1. greatest concern is on the validity of the predictor because it is used to predict status on a criterion (performance) measure.
   2. Criterion-related Validity of the predictor is most important

Question 4

steps in evaluating

criterion-related validity

Answer 4

1. conduct a job analysis
   
   1. identify the KSAO required for successful job performance.
2. Select/Develop the predictor
   
   1. predictor that measures the attributes identified by the job analysis.
3. Administer the predictor and criterion
   
   1. give predictor to the sample of applicants (predictive validity) or current employees (concurrent validity)
   2. obtain criterion information for all in sample
4. Correlate predictor and criterion scores
   
   1. get criterion-related validity coefficient to see if there is a statistically significant relationship between predictor and criterion
5. Check for adverse impact
   
   1. if the use of the predictor will discriminate against a legally protected group
6. Evaluate incremental validity
   
   1. will use of predictor increase the proportion of correct decisions
7. Cross-Validate
   
   1. do 3-6 for new sample
   2. cross-validation coefficent is usuallly smaller than original coefficient (shrinkage).
   3. chance factors maximized in the original validity coefficient are not present in the second sample !

Question 5

Adverse Impact

Answer 5

title VII of the civil rights acts of 1964

apply to any measure/procedure used as the basis for employment decisions and to a wide range of selection procedures.

when use of that procedure results in a substantially different selection, placement, or promotion rate for members of that group

Question 6

80% Rule

(4/5^thsrule)

Answer 6

1. used to determine if a procedure is having an adverse impact.
2. the hiring rate for the majority group is multiplied by 80% to determine the minimum hiring rate for the minority group.
   
   1. requires that a pattern of discrimination has already been established.

Question 7

reason for adverse impact:

differential validity

Answer 7

1. measure is valid for one group but is not valid for another group.
2. moderator variable: the characteristic that distinguishes between the two groups (age, sex)
3. Scatterplots:
   
   1. criterion cut off vs. predictor cut off
   2. circle is bad // elipse to upper right is better
   3. want to have a strong relationship between predictor and criterion scores.
   4. not valid if low score on the predictor were just as likely to receive acceptable score on the criterion as those receiving high predictor scores.
4. Remedy:
   
   1. do not use the predictor for the adverse impacted group
   2. use a different predictor that is equally valid for both groups.

Question 8

adverse impact #2

unfairness

Answer 8

1. members of one group consistently obtain lower scores on the predictor than members of another group
   
   1. but the difference in predictor scores is NOT related to differences in scores on the criterion.
2. Scatterplots
   
   1. have the same shape = predictor has the same level of validity for members of both groups
   2. but one group does better on the predictor than the other group! Unfairly favored.
3. Remedy:
   
   1. use different predictor cutoff scores for men and women.

Question 9

LEGAL proof of Adverse Impact

Answer 9

1. burden of proof is on employer
2. Business necessity:
   
   1. measure is job-related and relate required for the safe and efficient operation of the business.
3. Bona fide occupational qualification (bfoq)
   
   1. characteristic in question is bfoq (male part/acceptable community expectation)

Question 10

Incremental Validity

Answer 10

1. increase in decision-making accuracy an employer will achieve by using the predictor to make selection decisions.
2. validity coefficient (variable depending on the selection ratio and base rate).

Question 11

incremental validity

Selection Ratio

Answer 11

1. ratio of job opening to job applicants (O to A)
2. Want this to be LOW RATIO (1:50) as opposed to (2:1).
   
   1. good for employer to be more selective and thus increase predictor cutoff
   2. decrease risk of highering false positives since higher predictor scores will have higher criterion scores as well (weed out those who did good on predictor but are low on criterion).

Question 12

incremental validity

Base Rate

Answer 12

1. percent of employees who are performing satisfactorily w/o use of the proposed predictor
   
   1. ranges from 0 to 1.0
   2. MODERATE base rate is best!
   3. addition of additional predictor will have little effect on the quality of the work force
   4. High Base rate: predictor is acceptable
   5. Low Base rate: something other than the procedure is the problem

Question 13

Taylor-Russel Tables

Answer 13

1. estimates % of new hires that will be successful as employees given various combinations of validity coefficients, selection ratios, and base rates.
   
   1. if selection ratio is low and the base rate moderate (ideal), a predictor with a low validity coefficeint can improve decision-making accuracy.

Question 14

combining predictors

multiple regression (MR)

Answer 14

1. MR: predictor scores are weighted and summed to give an estimated criterion score
   
   1. Compensatory: exceptional performance on one predictor can offset poor performance on another predictor

Question 15

combining predictors

Multiple Cutoff

Answer 15

1. Noncompensatory: a minimum score on each predictor must be obtained before a job applicant will be considered for selection.
2. used with MR : first select using MC and then MR

Question 16

combining predictors

Multiple Hurdles

Answer 16

1. predictors are administered one at a time in a predetermined order.
   
   1. only if the previous one was passed/completed
   2. saves time/money since not all of the predictors are given to all people.