Reliability and Validity

Question 1

What is the most important property of a personnel assessment method?

Answer 1

Predictive validity: the ability to predict future job performance, job-related learning and other criteria.

Schmidt and Hunter, 1998

Question 2

What impact does using hiring methods with higher predictive validity have for an org?

Answer 2

Use of hiring methods with higher predictive validity leads to substantial increases in employee performance as measured in % increases in output and value of that output. This means orgs can increase their productivity and output by using procedures that work well and not using those that don’t.

Schmidt and Hunter, 1998

Question 3

Why does cognitive ability (General Mental Ability- GMA) hold a special place among all selection measures?

Answer 3

1) It has the highest validity and lowest application cost (compared to all procedures that can be used for all jobs, entry level or advanced). Doesn’t require job knowledge when others do.
2) It has the strongest validity research than any other method (thousands of studies).
3) Shown to be best predictor of learning on the job.
4) Theoretical foundation stronger than any other. (Intelligence theories make the construct more clear than what’s measured by interviews).

Schmidt and Hunter, 1998

Question 4

How valid are work samples?

Answer 4

Work samples are slightly more valid than GMA, actually, but they are more costly and can only be used with applicants who already know the job or have had training.

Schmidt and Hunter, 1998

Question 5

What is the relation of job performance validity and on the job learning/training performance validity for GMA?

Answer 5

Very high predictive validity of GMA for training performance.

When an employer uses GMA to select employees who
will have a high level of performance on the job, that employer
is also selecting those who will learn the most from job training
programs and will acquire job knowledge faster from experience
on the job.

Schmidt and Hunter, 1998

Question 6

What selection/promotion methods are best for people who already work in an org?

Answer 6

• past performance!
• work sample
• job knowledge
• peer ratings - but can be biased

Schmidt and Hunter, 1998

Question 7

How is past performance used to predict future performance in a selection/promotion context?

Answer 7

“Past performance is best predictor of future performance”

Steps:
1) First step is to determine what achievement dimensions best separate top performers from low. Ask experienced supervisors - critical incident technique.

2) Structured interviews based on those behaviors. Ask them to describe their past achievements that best illustrate their ability to perform these functions at a high level (e.g., organizing and getting work done through others).
The answers are scored using scales that are anchored (BARS) by specific scaled achievements that serve as anchors or illustrative examples.
The questions can also ask about experiences in SIMILAR jobs, so can be used for folks who don’t have experience with exact job. Can be achievements in other life sectors (work, volunteerism, etc). May not work super well with very entry level people who haven’t had the chance to demonstrate the capacity.

This is called the behavioral consistency method.

It’s time consuming and thus costly to develop, but might be well worth the effort for higher level jobs especially.

No validity evidence available for it.

Hunter and Schmidt, 1998

Question 8

Why does cognitive ability predict performance?

Answer 8

The major direct causal impact of mental ability has been found
to be on the acquisition of job knowledge. That is, the major
reason more intelligent people have higher job performance is
that they acquire job knowledge more rapidly and acquire more of it; and it is this knowledge of how to perform the job that
causes their job performance to be higher. There is also a direct effect of GMA on performance but it’s smaller.

The same applies to job knowledge - those who have more of it have had more practice and opps to learn and acquire job knowledge.

Same to conscientiousness - they are more on task and exert more effort on the job so they acquire more job knowledge.

Hunter and Schmidt, 1998

Question 9

What is validity generalization?

Answer 9

The application of meta-analysis
to validity data became known
as validity generalization, and a test was
argued to show validity generalization
when a large majority (typically 90%) of
population validities were above 0.

Even though the Uniform Guidelines say that local validity studies should occur,
Some scholars argue that meta-analytic evidence largely eliminates
the need for local validity studies.

HOWEVER this could be detrimental to diversity and it could be applied inappropriately as a validation defense.

McDaniel et al. 2011

Question 10

What is synthetic validity?

Answer 10

Breaking down multiple jobs into requirements and collapsing across roles and validating the selection tools for the requirements. i.e., both secretary and paralegal need typing skills, collapse their data together just for purpose of validation.

Good for small sample sizes (i.e., when hiring fewer people into a role, or in small business).

C&A, 2018; Gatewood et al., 2019

Question 11

What is test transportability?

Answer 11

Refers to using tests already validated by another organization, for a similar role. Test publishers are a good example of this.

C&A, 2018; Gatewood et al., 2019

Question 12

What is validity generalization and who is it good for?

Answer 12

This refers to using meta-analytic research on selection tools as validity evidence. These studies are also called VG studies (validity generalization). These are good for instances when don’t have large sample sizes for a local validation study (i.e., small and even some medium sized businesses).

C&A, 2018; Gatewood et al., 2019

Question 13

How large does a sample size need to be for a local validation study?

Answer 13

Some say at least 500, others say even more.

Gatewood et al., 2019

Question 14

What is the definition of validation?

Answer 14

The process of gathering validity evidence across multiple studies.

Gatewood et al., 2019

Question 15

What is the definition of validity?

Answer 15

The degree to which a measure measures what it is designed to measure.

What a test measures (trait/construct) and how well it measures it.

Gatewood et al., 2019

Question 16

Why is validation important in selection? (2 reasons)

Answer 16

1. fundamental for useful and competent HR practice
2. must provide validity evidence if procedure has adverse impact on a protected group

Gatewood et al., 2019

Question 17

Define reliability.

Answer 17

The degree of consistency/ agreement between two sets of scores that were collected independently (estimates precision of a measurement tool as a correlation)

Gatewood et al., 2019

Question 18

What are three types of reliability and how are they are performed?

Answer 18

TEST-RETEST
(Most simple and direct)
Same form of test given to same people on 2 different occasions (ideally a few months apart). Correlation.

PARALLEL FORMS:
Each test has the same # and difficulty of items.
Correlate them to see how equivalent they are, and that correlation is the reliability. Ideally at same time or w/in a few days of each other. Sampling error inevitable, which lowers correlation so it’s a conservative estimate.

INTERNAL CONSISTENCY:
Diff from others -this is about the degree to which items on the test correlate with each other. Cronbach’s alpha. Factor analysis is used.

Gatewood et al., 2019

Question 19

What is inter-rater reliability?

Answer 19

The degree to which raters’ ratings agree (the degree they are free from unsystematic error variance)

Gatewood et al., 2019

Question 20

What are three methods of calculating interrater reliability?

Answer 20

1. Interrater agreement - agreement of their ratings, using % or kappa
2. Interclass correlation: when 2 raters are rating multiple people
3. Intraclass correlation: how much differences are due to diffs in people vs error

Gatewood et al., 2019

Question 21

Describe the three primary kinds of validity evidence.

Answer 21

1. Content validity:
   degree to which a test samples the content domain of the job associated with successful performance. i.e., does the test battery for a typist position cover all the critical KSAs needed to perform well as a typist?
2. Criterion validity: degree to which the test can predict future performance on the job. Can be predictive or concurrent. Predictive: uses data from target people with time in between; concurrent uses data from current employees. Gold standard but not always possible due to small sample size (i.e., small business).
3. Construct validity: demonstrates and tests relationships between measures and constructs. i.e. ensuring the construct we’re measuring is what we intend to and isn’t measuring something else we don’t. (Discriminant and convergent)

Gatewood et al., 2019

Question 22

How is predictive validity evidence gathered and what’s the tradeoff?

Answer 22

Predictive validity (criterion) is collected by gathering data from applicants using the new selection tools, but then setting that data aside and not using it for the selection process (if you do, you risk greater range restriction later). Then 3-6 months later, assess the new hires for that role in terms of performance, and then analyze their scores to see if there is a relationship with the selection tools you didn’t use. The tradeoff is that there is a time delay and often managers won’t want to wait to use the new selection tools.

Gatewood et al., 2019

Question 23

How is concurrent validity evidence gathered and what is the tradeoff?

Answer 23

Concurrent validity evidence is gathered by testing the selection tools on current employees and using their existing performance appraisal data to test the relationship. Pro: faster; can use them in selection sooner. Cons: it’s with existing employees who already went through selection process (with other methods) so there is more range restriction. Can statistically correct.

Works well for things like cognitive ability tests (equivalent to predictive)

Gatewood et al., 2019