2 of the assertions made about testing in Sackett et al's (2008) article Flashcards
X is one aspect of the field of psychology with which virtually the entire public comes into contact.
Testing
Broadly, what do Sackett and colleagues (2008) do in their paper?
Review criticisms commonly leveled against cognitively loaded tests used for employment and higher education admissions decisions, with a focus on large scale databases and meta-analytic evidence.
What are two of the assertions made about testing?
1) The appearance of validity is due to SES
2) Tests are readily coached
A common assertion made about testing is that test scores are influenced by socioeconomic factors, such as parents’ earnings and education. What does the stronger version of this criticism assert?
That tests measure nothing more than SES.
What is the criticism that tests measure nothing more than SES leveled at?
This criticism is primarily leveled at educational admission tests
What did Kohn (2001) claim?
That “the SAT merely measures the size of students’ houses” (Kohn, 2001)
In the fact of the claim that the appearance of test validity is due to SES, what do Sackett and colleagues (2008) highlight?
That much of the so-called empirically grounded evidence backing up this claim is based on research from 2001 that did not, in fact, focus on the validity of SAT I when controlling for SES, but rather on the incremental validity of SAT I scores over SAT II scores when controlling for SES (Geiser & Studley, 2001)
Sackett and colleagues (2008) describe research that addressed the issue of whether SES is responsible for the validity of tests of developed abilities in predicting academic performance. Who conducted this research? What did they do?
Sackett et al. (2007)
The researchers summarized results from 8 datasets, including a meta-analysis, and College Board SAT dataset on >165,000 students from 41 colleges, data on all individuals entering an accredited law school in 1991, and 3 large longitudinal studies following samples of high school students through college.
Using multiple large datasets the authors were able to partial out the effects of SES from the test-grade correlation.
Using multiple large datasets the Sackett et al (2007) were able to partial out the effects of SES from the test-grade correlation.
What did they find?
They found that SES is related to test scores, but not in the simple way asserted.
Using multiple large datasets the Sackett et al (2007) were able to partial out the effects of SES from the test-grade correlation.
In broad, unrestricted populations, what was the correlation?
In smaller, restricted samples, what was the correlation?
Quite substantial (i.e., r=.42 among the population of test takers).
Considerably smaller (ie. rs = .15-.20 among samples of students enrolled in a single institution)
Did Sackett et al (2007) find that test scores are predicted of academic performance as indexed by grades?
Yes
Did Sackett et al (2007) find that the test grade relationship was an artifact of common influences of SES on test scores and grades?
No
The assertion that the predicted power of tests disappears once the effects of SES are removed is at odds with the findings from these multiple sources of data.
Based on this evidence, Sackett and colleagues (2008) conclude that while the claim that tests measure “nothing but” SES is not supported, there is clearly…
a substantial relationship between SES and test scores.
However, they go on to say that the relationship between test scores and academic performance is affected only to a small degree by controlling for the effects of SES on both test scores and grades. Thus, it is not the case that test validity is an artifact of SES. Rather…
SES is linked to the development of abilities that are predictive of academic performance.
