Validity Flashcards
It is a judgment based on evidence about the appropriateness of inferences drawn from test scores
Validity
Validation process if test users plan to alter format, instruction, language or content of the test
Local validation
Trinatarian view
Content, criterion, construct validity
Judgment concerning how relevant a test items appear to be
Face validity
How adequately a test samples behavior representative of the universe of behavior that the test was designed to sample; coverage, course syllabus, curricula
Content validity
A plan regarding the types of info to be covered by the items the numbers of items tapping each area of coverage, organization of items and so forth
Test blueprint
Manage creation of new items and the output of old items in a manner that is consistent with the test’s blueprint
item pool management
In quantifying content validity, this is used to gauge agreement among raters or judges regarding how essential a particular item is
Lawshe content validity ratio (CVR)
Negative=fewer than half the panelists indicate item is essential
Zero= exactly half the panelists indicate essential
Positive=more than half but not all panelists indicate essential item
Content Validity ratio
Judgment of how adequately a test core can be used to infer an individual’s most probable standing on some measure of interest
Criterion-related validity
A criterion must be ___, ___, and ___.
relevant, valid and uncontaminated
Happens when criterion overlap with predictor measures
Criterion contamination
Test scores are obtained about the same time criterion measures are obtained; also used to compare tests with different versions
Concurrent validity
How accurately scores on a test predict some criterion measure
predictive validity
Correlation coefficient that provides a measure of the relationship between test scores and scores on the criterion measure
Validity coefficient
Degree to which an additional predictor explains something about the criterion measure that is not explained by predictors already in use
Incremental validity
Provide info that can be used in evaluating criterion-related validity of a test
Expectancy data
Likelihood that testtaker will score within somwe interval of scores on a criterion measure (an interval that may be seen as “passing or acceptable”
Expectancy table
Provide an estimate of the percentage of employess by the use of a particular test who will be successful at their jobs.
Taylor-russel tables (comprises test’s validity, selection ratio, base rate)
Numerical value that reflects relationship between number of people to be hired and number of people available to be hired
Selection ratio
Percentage of people hired under the existing system for a particular position
Base rate
Limitations of the Taylor-Russel table
- cannot distinguish differences between successful from unsuccessful employees
- relationship between predictor and criterion must be linear
Obtain difference between the means of the selected and unselected groups to derive an index of what the test is adding to the already established procedures
Naylor-Shine table
Provide an estimate of the extent to which inclusion of a particular test in the selection system will actually improve selection
Taylor-Russel tables
Theory that explains the usefulness or practical value of tests
Test utility theory
Statistical rules for developing a sequential analysis of a problem that would lead to an optimal decision
Decision theory
Extent to which a particular trait or behavior exists in the population
Base rate
Proportion of people a test accurately idetifies as possessing or exhibiting a particular trait, behavior, characteristic
Hit rate
Proportion of people the test fails to identify as having or not having a particular characteristic or attribute
Miss rate
Employers are reluctant to use this theory because of the complexity of the application and the threat of legal challenges
Decision theory
One measure of the value of a test lies in the extent to which its use improves on the hit rate that exists without using ut
To remember
If a test is a valid measure of the construct, then high scorers and low scorers will behave as predicted by the theory
Construct validity
Prove validity by demonstrating that scores on the test vary in a predictable was as a function of membership in some group
Method of contrasted groups
Proof that a test provides scores similar to an older, established, and valid test of the same construct
Convergent evidence
No significant or little relationship between test scores and other variables should not be theoretically related to the construct
Discriminant evidence
Examine both convergent and discrimanant validity
Multitrait-multimethod matrix
Frequently employed as a data reduction method
factor analysis
Estimating number of factors, deciding how many to maintain
Exploratory factor analysis
Tested for its fit with the observed covariance structure of the measured variables
Confirmatory FA
The extent to which the factor determines the test scores
factor loading (low: provide discrimanant evidence; high: provide convergent evidence)
Inherent factor in a test that systematically prevents accurate measurement
Bias
When a test systematically under-predicts or over-predicts the performance of a particular group
intercept bias (regression line intercepts y-axis)
When the slope of one group’s regression line differs significantly from another’s
slope bias
solution: estimated true score transformations
Numerical or verbal judgment that places a person along a continuum
rating
Judgment resulting from intentional or unintentional misuse of a rating scale
RATING ERROR
How to overcome the 3 rating errors (leniency, severity, central tendency)
Thru RANKINGS (measure individuals against one another insetad of against an absolute scale)
Tendency to give a particular ratee a higher rating than he or she objectively deserves beacuse of the rater;s failure to discriminate among conceptually distinct and potentially independent aspects of ratee’s behavior; the ratee “can do no wrong”
Halo effect
Extent to which a test is used in an impartial, just, and equitable way; more rooted in issues involving values
fairness
