2-7 tb Flashcards
What is evidence of validity based on test-criteria relationships?
?????
criterion
the measure of performance (independent behaviors, attitudes, events) that we correlate with test scores
the extent to which a measure is related to an outcome.
the measure of performance (independent behaviors, attitudes, events) that we correlate with test scores
the extent to which a measure is related to an outcome.
criterion
Examples of criterion measures in clinical setting? Educational setting?
Clinical: psychologists often use tests to diagnose mental disorders
Educational: Educators use admissions tests to forecast how successful an applicant will be in college or graduate school (GRE)
the predictive method
used when it is important to show a relationship between test scores and a future behavior
validity coefficient: the resulting correlation coefficient when two sets of scores are correlated–usually test scores (IV) and criterion score (DV)
used when it is important to show a relationship between test scores and a future behavior
the predictive method
validity coefficient: the resulting correlation coefficient when two sets of scores are correlated–usually test scores (IV) and criterion score (DV)
the concurrent method
when test administration and criterion measurement happen at the same time
when test administration and criterion measurement happen at the same time
the concurrent method
how does validity differ from reliability?
reliability is the consistency of the test
validity is the inferences based on test scores
test can be reliable but not valid (e.g. yardsticks are reliable for measuring distance, but not valid for measuring IQ)
what are the two types of criterion
objective and subjective
objective criterion
it is observable and measurable
e.g. number of accidents on the job
it is observable and measurable
objective criterion
subjective criterion
it is based on a person’s judgment
e.g. supervisor ratings
it is based on a person’s judgment
subjective criterion
Does the criterion measure what it is supposed to measure?
the criteria must be representative of events they are supposed to measure (content validity)
criteria measures fewer than what measured by test, it lacks evidence of content based validity
criteria measures more than the test, there is criterion contamination
tests of significance
The process of determining what the probability is that a study would have yielded the observed results simply by chance
“How likely is it that the correlation between the test and the criterion resulted from chance or sampling error?”
The process of determining what the probability is that a study would have yielded the observed results simply by chance
tests of significance
p-value
the probability of obtaining test results at least as extreme as the results actually observed during the test, assuming that the null hypothesis is correct
the probability of obtaining test results at least as extreme as the results actually observed during the test, assuming that the null hypothesis is correct
p-value
p < .05
means the likelihood a relationship was found by chance or as a result of sampling error was less than 5 chances out of 100
p < .01
means the likelihood a relationship was found by chance or as a result of sampling error was less than 1 chances out of 100
coefficient of determination
answers the question “what amount of variance do the test and criterion share?”
The amount of variance shared by two variables being correlated, such as a test and a criterion, obtained by squaring the validity coefficient.
The amount of variance shared by two variables being correlated, such as a test and a criterion, obtained by squaring the validity coefficient.
coefficient of determination
What does the coefficient of determination tell us about the test and its relationship with a criterion?
For example, if the correlation (r) between a test and a criterion is .30, the coefficient of determination (r2) is .09. This means that the test and the criterion have 9% of their variance in common.
How confident can we be about estimates of validity?
as long as the test is in use, need to keep validating it
use cross-validations and meta-analyses
Using validity information to make predictions: Linear Regression
statistical procedure for predicting performance on a criterion using one set of test scores
DV is often the criterion
intercept is the predictor variable and the constant
statistical procedure for predicting performance on a criterion using one set of test scores
Using validity information to make predictions: Linear Regression
Using validity information to make predictions: Multiple Regression
statistical procedure for predicting a criterion using more than one set of test scores
equation includes multiple terms for the predictor variables and the constant (intercept)
DV is often the criterion
statistical procedure for predicting a criterion using more than one set of test scores
Using validity information to make predictions: Multiple Regression
Ethical issues associated with test validation
Test Users: be aware of test bias, make sure it’s valid for minorities, no questions specific to cultural backgrounds, appropriate norm groups for minorities
Test Publishers: make test manuals and information available/accessible before purchase and refuse to give test material to people not qualified to administer it
