Selection - Validity

Question 1

define validity

Answer 1

the degree to which available evidence supports inferences made from scores on selection procedures; how well are you measuring what you’re claiming to measure?

Question 2

how is validity applied in the context of selection?

Answer 2

we want to know how well a predictor (i.e., test) is related to criteria (i.e., performance)

Question 3

Discuss the relationship between reliability and validity

Answer 3

• it is possible to have a measure that is reliable yet does not measure what we want for selection.
• However, we cannot have high validity if we do not have high reliability. High re- liability is a necessary but not a sufficient condition for high validity.
• calculated validity can’t be higher than the maximum possible validity because both the predictor and criterion scores contain random error. Random error is uncorrelated, so the more random error within scores from a predictor or criterion then the more likely it is that the maximum possible validity will fall.
• if reliability of either test X or criterion Y were lower, maximum possible validity would be lower as well.
• If either our test or criterion were completely unreliable (that is, reliability 0.00), then the two variables would be unrelated, and empirical validity would be zero. (We discuss the meaning of a validity coefficient later in this chapter.)
• Thus reliability or unreliability limits, or puts a ceiling on, possible empirical validity.
• Practically speaking, to enhance maximum possible validity, reliability should be as high as possible for our predictors and our criteria.

Question 4

list sources of validity evidence

Answer 4

Content
response processes
internal structure
relations w/ other variables
decision consequences

Question 5

content validity: describe what it is

Answer 5

Content validity is demonstrated to the extent that the content of the assessment process reflects the important performance domains of the job. Validity is thus built into the assessment procedures.

Content validation methods focus on content relevance and content representation. Content relevance is the extent to which the tasks of the test or assessment are relevant to the target domain. Representativeness refers to the extent to which the test items are proportional to the facets of the domain. Content relevance and representativeness are commonly assessed using subject matter expert ratings.

many companies rely heavily on content validation for various reasons.

Question 6

describe the steps in content validity

Answer 6

1. job analysis

2. examination plan

Question 7

challenges with conducting criterion related validity studies

Answer 7

sample size: small sample sizes are often due to many job classifications having only a small number of employees

range restriction: the selection process has already restricted the org’s workforce to a certain level of performance (the scores of hired people don’t reflect scores of entire applicant group)

criterion measures: good criterion measures are often not available or just plain bad

Question 8

advantages of content validity

Answer 8

• feasible
• viewed as fair
• helps ID best candidates
• practical to meet legal standards when supplemented by supporting data
• easy to explain to courts and candidates
• acceptable under UG and The Principles

Question 9

disadvantages of content validity

Answer 9

• not easy: time, resources, expertise, documentation
• some people don’t believe in it
• difficult for entry level jobs where there isn’t any specific prior preparation required
• not appropriate when job requirements change frequently or aren’t well defined
• more resources to develop job specific test vs. use of GMA tests

Question 10

content validity: direct vs indirect measures

Answer 10

for content validity, there is a hierarchy of assessment evidence. we can think of this evidence as being higher when direct methods are used and lower when indirect methods are used. For example, in measuring keyboard ability, a highly content valid keyboarding test would replicate the most important job tasks (text entry and data entry). The test would be a direct measure of keyboarding ability. Two indirect measures or indicators of keyboarding ability are completion of a high school keyboarding course, and having keyboarding work experience. The indirect measures do not inform us about the current keyboarding proficiency of the subject.

Question 11

List and describe the guidelines for interpretting correlations according to the U.S. Dept. of Labor (note that setting an arbitrary value of a validity coefficient for determining whether a selection procedure is useful is not a wise practice).

Answer 11

above .35 = very beneficial
.21-.35 = likely to be useful
.11-.20 = depends on circumstance
less than .11 = unlikely to be useful

Question 12

validity of work sample tests

Answer 12

.33

Roth, Bobko, & McFarland, 2005
Correlated w/ supervisory ratings of job performance.

Question 13

validity of structured interviews

Answer 13

.51

Not sure where this is from.

Question 14

validity of unstructured interviews

Answer 14

.38

Not sure where this is from.

Question 15

validity of job knowledge tests

Answer 15

.48. job knowledge tests which have high job specificity, have higher levels of criterion-related validity. Job knowledge tests can not be used for entry-level jobs. They are not appropriate for use with jobs where no prior experience is required or where no prior job-specific training is required

Not sure where this is from.

Question 16

validity of behavioral consistency T&E methods

Answer 16

.45; This method is based on the principle that the best predictor of future performance is past performance. Applicants describe their past achievements and the achievements are rated by subject matter experts.

McDaniel et al 1988

Question 17

validity of self report T&E methods

Answer 17

.15-.20; few studies available

Not sure where this is from.

Question 18

validity of years of experience

Answer 18

.18; Years of experience are a very indirect measure of ability.

Not sure where this is from.

Question 19

assessment tests vs. GMA

Answer 19

Review of the meta-analysis results, and comparison to the list of direct and indirect assessment methods, leads to the conclusion that, except for general ability tests, the predictive value of assessment methods reflects the extent to which they more directly assess applicant competencies.

Question 20

the most direct assessment methods are?

Answer 20

work sample tests, job knowledge tests, and structured interviews

Question 21

how does content validity relate to criterion validity ?

Answer 21

The data indicate that direct assessment methods have higher levels of criterion-related validity than indirect assessment methods. This is evidence that the stronger the content validity evidence supporting an assessment method, the more likely it is that the assessment method will have a high level of criterion-related validity. In the author’s view, the meta-analysis criterion-related validity data provides support for the content validation model.

LOOK INTO/FIND CITATION

Question 22

how to minimize adverse impact while maximizing validity

Answer 22

assess full range of KSAs

make sure predictor test verbal requirements dont exceed verbal requirements of the job

both of these things can be done with job knowledge tests

Question 23

Research designs for validity studies

Answer 23

concurrent: Test given to group of employees already on job, then correlated with measure of employees’ performance
Weaker than predictive but more practical, because of the homogeneity in performance scores; Problem: Range restriction

Predictive:
Test administered to group of job applicants who are going to be hired and then compared with future measure of job performance

Question 24

validity generalization

Answer 24

Research indicates a valid test for job in one organization is also valid for the same job in another organization

Two building blocks for validity generalization

• Meta analysis: determine average validity of specific types of tests for a variety of jobs
• Job analysis: show that job in question is similar to those used in the meta analysis

Question 25

describe synthetic validity

Answer 25

Based on assumption that tests that predict a particular component (e.g., customer service) of one job should predict performance on the same job component for a different job

Question 26

discuss construct validity

Answer 26

Most theoretical of validity types
Extent to which a test actually measures the construct that it purports to measure
Concerned with inferences about test scores rather than test construction (like content validity)

Question 27

describe how to determining construct validity

Answer 27

Correlating scores on test with scores from other tests

Convergent vs. discriminant validity

Question 28

face validity

Answer 28

Not included as primary way in Uniform Guidelines but still important beacuse it affects perceptions of test takers
It helps in court because it’s easy to understand and explain
To increase face validity you can inform the applicants about how the test relates to job performance and giving honest feedback about their performance
Face validity doesn’t mean a test is accurate or useful though; not enough by itself

Question 29

things to consider when choosing ways to measure valdiity

Answer 29

cost efficiency

consider goals of validity study

Question 30

Taylor Russel Tables

Answer 30

Designed to estimate the percentage of future employees who will be successful on the job if an org uses a particular test; need 3 pieces of info: test’s criterion validity coefficient, selection ratio, base rate of current performance (% of successful employees currently)

Question 31

list ways to determine the usefulness of a selection device

Answer 31

Taylor Russel Tables

Utility formula

Question 32

determining fairness of a test: types of bias

Answer 32

measurement bias

predictive bias

Question 33

determining fairness of a test: what is fairness

Answer 33

Disagreement in the field on what fairness actually is, but most consider race, gender, disability in content (measurement bias) and prediction (predictive bias)

Question 34

measurement bias

Answer 34

• Refers to technical aspects of a test
• Measurement bias = group differences (sex, race, age) in test scores that are unrelated to construct being measured
• Adverse Impact: Differences in test scores result in one group being selected at a significantly higher rate than another

Question 35

predictive bias

Answer 35

-Situations in which predicted level of job success falsely favors one group over another

Single Group Validity
-Predicts performance for one group but not others

Differential Validity
-Test valid for two groups but more valid for one than for the other

Question 36

things that affect applicant perceptions of fairness

Answer 36

test difficulty 
time allowed to complete
face validity 
how hiring decisions are made
policies about test retakes

Question 37

list the ways to make hiring decisons

Answer 37

unadjusted top down
Rule of Three
Passing/Cutoff Scores
Banding

Question 38

discuss unadjusted top down selection

Answer 38

• Applicants rank ordered on basis of test scores
• Highest scorer is hired then moves down until all openings filled
• Most utility (Schmidt, 1991) but can result in high levels of adverse impact and reduces org’s ability to use nontest factors like references or org fit

Compensatory approach
-If multiple test scores are used, the relationship between a low score on one test can be compensated for by a high score on another

-Who will perform best in the future?

Question 39

Discuss the rule of three method for selection

Answer 39

• Names of top three scorers are given to person making hiring decision and then they can choose any of the three based on needs
• Provides more choice, but subjectivity involved

Question 40

Discuss the use of passing/cutoff scores method for selection

Answer 40

• Can reduce adverse impact and increase flexibility to look at other factors
• Org determines lowest score on a test that can be accepted/predict success
• Who will perform at an acceptable level in the future?
• Is good when needing to reach affirmative action goals
• But performance scores lower than top down (Schmidt, 1991)
• Legal problems can occur when unsuccessful applicants challenge the validity of the passing score

Methods
-Angoff & Nedelsky require SMEs to read each item on a test and provide an estimation for each question

-Multiple Cutoff/Hurdle Approach
When there is more than one test for which we have passing scores
Used when one score cannot compensate for another or when relationship between selection test and performance is not linear
When you fail one test, you are not considered further
Saves money

Question 41

usefulness of synthetic validity; why hasn’t it been used often?

Answer 41

could be useful in the context of selection; could even be the future of selection, but it’s not as widely practiced as you would think and has had relatively little application in the past few decades. it became seen as obsolete by the time methods were developed thet addressed some of the challenges of conducting syntactic validity studies. It was introduced in order to help smaller organizations conduct validity studies, since number of incumbents to do so was likely small, making local validity studies near impossible. Synthetic validity allows us to draw conclusions about selection test validity without much other than a general description of the Job. Not as much local knowledge, but rather general knowledge.

Question 42

outline some general steps you might take to create a content valid measure

Answer 42

1. Job Analysis: this will serve to define the job content domain, which consists of work activities, KSAs, etc. needed to perform the job. Specifically, JA includes task descriptions, task importance, KSAs required, and linkage of KSAs to the job tasks.
2. Determine SMEs to participate: expert judgments should be obtained from job incumbents and/or supervisors to provide information regarding job tasks, KSAs, etc. in order to develop the content of the selection measure.
3. Specification of Selection Measure Content: This will take place after the job tasks and KSAs have been identified through the previous steps. This includes domain sampling, where items and test content are chosen.
4. Assessment of Selection Measure and Job Content Relevance: After the test has been developed, SMEs judge the degree to which KSAs identified from the JA are needed to answer the test questions. This usually involves incumbents taking the test and rating the extent of overlap. A formula can then be used to produce a content validity ratio (CVR). These can be averaged to produce a content validity index (CVI) for the test, which indicates how well the test overlaps with the ability to perform on the job.

Question 43

How viable are criterion validation studies as you ascend the organizational hierarchy?

Answer 43

As you move up the staff functions hierarchy, conducting criterion validation studies are going to become increasingly expensive, and therefore less viable. The consequence of error increases as you move up in the hierarchy.
This is because the jobs become more abstract, more construct oriented, and involve less observable KSAOs. More judgement calls involved with higher up positions.
Establishing appropriate criterion (i.e., job performance) becomes more difficult as a function of the increasing complexity of the job, especially because jobs are changing so quickly.

Question 44

describe general steps for construct validation

Answer 44

Make a case that the construct actually exists
Could be observations
Create a measurement for the construct (and piloting)
If the construct exists in varying amounts, the measure should pick who is in and who is out
Measure should be able to predict
Items could be “content valid” if you are a SME
The construct should be able to predict Behaviors -> explanation
Gather data – examine characteristics within the dataset
How do the items correlate?
Clean up the scale
Administer scale to criterion groups
CFA
Should have discriminance with things that should be conceptually different
Evaluate how well the construct is working

Question 45

content validation: examination plan

Answer 45

is the next step after conducting a JA. The JA information is used to construct this plan, which includes
-What will be assessed
- The type(s) of assessment methods to be used, including the evidence and
rationale supporting these decisions
- Linkages of tasks, KSA’s, and test parts
- Test weighting based upon the job analysis data
- The appropriate method of use of each assessment
- A plan for test development or test selection and test review
- A plan for the sequencing, standardized administration and objective scoring of
the assessments
- A plan for establishing passing scores
- Evaluation of test effectiveness by a study of test reliability, and
- Statistical analysis of the test

Question 46

how positive manifold affects validity

Answer 46

Selection tests often show a pattern of positive correlations among tests and between tests and criteria. Positive manifold does not require that these relationships be large, but that they are in the same direction. When selection tests have positive manifold, the match or mismatch between test content and job content will have little impact on criterion related validity of those tests. The same applicants who do well on the tests that match job content will do well on tests that do not match job content but that measure the same construct. This is basically making the argument that content validity does not always increase criterion related validity. This is especially true for GMA tests which are often criticized for not having face validity. The less positive manifold a set of predictors has, the more important content validity then becomes in relation to criterion related validity.