14 - Bias

Question 1

Bias

Answer 1

Systematic measurement error

Question 2

Predictive vs Test Structure Bias

Answer 2

Predictive: differences between groups in criterion prediction

 - Focuses on total score of the measure (external criterion)
 - Steeper the slope (regression), the better the prediction

Test Structure: differences in internal test characteristics between groups

 - Focuses on test itself (internal criterion)
 - Can be empirical or theoretical

Question 3

Prediction Scatterplot

Answer 3

• Predictor (test score) is on x axis
  
  • If we shift x-axis, we are shifting base rate
• Criterion (job performance) is on y axis
  
  • If we shift y axis, we shift selection rate
• Regression Line (line of best fit)
  
  • Good = steep slope, ideally through the origin (0,0), with few FP and FN
  • Bad = shallow slope, equivalent datapoints in false and true quadrants

Question 4

Predictive Bias: Different Slopes

Answer 4

• Different slopes = differential validity
  
  • Meaning, different predictive validity for different groups (it is a better predictor for one group (steeper slope) than the other)

-Correction: use a different measure for the minority group for big slope differences

(for small slope differences, use within-group norming)

Question 5

Predictive Bias: Different Intercepts

Answer 5

• Different intercepts = systematic over- or under-estimation of group performance
  
  • Meaning, the same test score leads to different predictions for groups

Correction: add bonus points to the minority group

*NOTE: over- and under-estimation is counterintuitive in this graph! Underestimation = y intercept greater than 0.

Question 6

Predictive Bias: Different Slopes AND Intercepts

Answer 6

-Poor differential validity (does a poor job of predicting outcomes for one group; SLOPE) and over- or under-estimates one or both groups (INTERCEPT)

Correction: use a different test altogether

Question 7

Empirical Approaches to Test Structure Bias

Answer 7

1. Item x group tests (ANOVA) : examines whether differences between groups on the overall score matches comparisons among smaller item sets between groups
   
   • Used to rule out that items operate in different ways for different groups
2. IRT (difficulty/severity) -> Differential item functioning (between groups); items function differently between groups
   
   • If ICC shows group differences = construct validity variance between groups = BIAS
     
     • Difficulty: inflection, where middlepoint of sigmoid curve hits the x axis
     • Discrimination: slope; steeper = more discriminatory (how well the item distinguishes between those higher/lower on construct)
3. Confirmatory Factor Analysis: examines whether the factor structure of underlying variables is consistent across groups (tests of measurement invariance)
4. SEM: multiple predictors, multiple outcomes; identifies # of latent variables

Question 8

Theoretical Approach to Test Structure Bias

Answer 8

1. Facial Validity: lay person
2. Content Validity: expert
   - A construct may include content facets in one group, but may include different facets in another.

Question 9

Fairness

Answer 9

ACCURACY =/= FAIRNESS!

Adverse Impact: rejecting members of one group at a higher rate than another; cannot have group (B) selection be fewer than 80% of highest selected group (A)

Operationalizing fairness:

1. Equal outcomes: equal selection rate
2. Equal opportunity: equal sensitivity (classification errors)
3. Equal odds: equal sensitivity and specificity

Question 10

Score Adjustments to Correct for Bias

Answer 10

1. Bonus Points: add points to particular group; used to correct for intercept predictive bias
   - If group differences in SD, BP may not correct bias
2. Within-Group Norming: corrects for slop predictive bias
   - Group norms for relative functioning, common norms for absolute functioning
3. Separate Cutoffs (BP): per group
4. Top-Down Selection From Different Lists (WGN) : take best from each group
5. Banding (BP): band width is equivalent to StErMeas, give minority preference
6. Banding with BP: BP first, then bands
7. Sliding Band (BP): select all minority members in a band and JUST majority with top score; then repeat with next bands
8. Separate Tests (WGN): option for different slopes
9. Item elimination based on groups (WGN): if large group differences

Question 11

Non-score Adjustment Techniques to Correct for Bias

Answer 11

1. Use multiple predictors/tests
2. Change the criterion
3. Remove biased items
4. Resolve biased items (retain each item but alter parameters for different groups)
5. Use alternative modes of testing
6. Use work records
7. Increase time limit
8. Use motivation sets (face validity)
9. Use instructional sets (access to test prep)