Validity and fairness in testing (Sackett etl al., 2008)

Question 1

Do tests predict badly?

Answer 1

No, in contrast, taking range restriction and criterion unreliability into account the validity coefficients are quite strong

Question 2

How is % of variance accounted for calculated?

Answer 2

The squared correlation coefficient gives the percentage of variance accounted for

Question 3

Describe how range restriction leads to an underestimation of validity

Answer 3

Because there is usually lower correlation in the restricted sample than in the full applicant pool

e.g., r=.50 in unrestricted sample drops to .33 when only including the top 50% of the sample

Question 4

Given an example of how unreliable criterion measures result in an underestimation of validity

Answer 4

School performance - measured by averaging course grades without accounting for course difficulty

Question 5

Describe educational evidence that goes against the assertion that tests predict only short-term criteria

Answer 5

Test scores are predictive of a variety of long term indicators of academic and career success

1. Meta-analytic evidence that the SAT predicts 1st-4th year grades equally well, with only slight decrements over time
2. Meta-analytic evidence that GRE scores predict comprehensive examination performance, faculty ratings of student performance, and subsequent citation counts
3. LSAT scores are predictive of passing the bar exam, getting a doctoral degree, detting tenure, and getting patents in a gifted sample

Question 6

Describe military evidence that goes against the assertion that tests predict only short-term criteria

Answer 6

Armed Forces Qualification Test predicted soldier performance over 4 year in a longitundinal study

Question 7

Describe employment evidence that goes against the assertion that tests predict only short-term criteria

Answer 7

Test validity was constant throughout the 10 years that a study followed a sample of 6500 individuals in low complexity jobs and 2500 individuals in high complexity jobs

Question 8

Why is the assertion that tests do not predict earnings false?

Answer 8

National Longitudinal Study of Youth shows relationships between measures of developed abilities and earnings

Question 9

People argue that some relevant criteria are not predicted well by tests of developed abilities, what is wrong with this statement?

Answer 9

While it is true that there are criteria of interest to educational and professional organizations for which tests of developed abilities, there are other mechanisms in place to predict such criteria:

e.g., academic performance: personal statements and letters of recommendation for fit to the college’s social and cultural environments

Question 10

People argue that important determinents of traditional criteria are missed, what is wrong with this statement?

Answer 10

Yes there is evidence that constructs in the noncognitive domain have the potential for improved prediction (e.g., achievement motivation and academic self-efficacy have incremental validity over admissions tests and high school GPA)

BUT you cannot say a test is flawed because it does not measure all determinants of the criterion of interest. Tests are typically part of broader selection systems, which incorporate multiple measures.

Question 11

Evidence against the statement that tests do not predict above a minimum threshold

Answer 11

There is strong evidence that higher test scores are associated with higher criterion scores throughout the test score range

Question 12

Is it true that the appearance of validity is due to SES?

Answer 12

No - multiple sources of data (including meta-analyses, longitudinal studies, SAT dataset of >165,000 students) are at odds with this assetion

Question 13

Is it true that tests are readily coached?

Answer 13

No - coaching produces small mean improvements in test scores, but these are far smaller than those claimed by test preparation firms

Question 14

Do men or women tend to have higher GPAs?

Answer 14

Women

Question 15

Do motivational factors (such as stereotype threat) explain group differences in test performance?

Answer 15

No

Question 16

Is it true that while minority group members obtain lower mean scores on tests, they perform just as well as majority group members once admitted or hired?

Answer 16

May be true for gender, not true for race