Selection Flashcards
Newman & Lyon (2009)
“• Examined combination of recruiting parameters and its effects on adverse impact ratio and performance
• Found that focusing on cognitive ability and conscientiousness recruiting increased adverse impact ratio and improved (did not harm) job perf
o Particularly, if targeted recruitment of these charac within minority groups
• Org recruitment strategy that focused emphasized results-orientation attracted more conscientious applicants
• Blacks who were conscientious more attracted (prob of applying) to innovation-based org
• Job postings that are worded in certain ways may help/deter recruitment of particular demographic groups”
Finch et al (2009)
“• Trade-off problem – mean perf levels (-)ly related to diversity in workforce
• Monte carlo study on multistage selection process and its effects on AI
• Found 9 predictor combinations that had no AI and met 4/5ths rule
• All included integrity tests (except 1), mostly biodata Qs and conscientiousness – NONE had cognitive ability tests as predictors
• Multistage selection strategies outperformed singlestage strategies in terms of having lower AI and almost equivalent mean perf
• Commonly held assumption of including cog ability tests in the first stage resulted in higher AI and lower mean perf”
Highhouse (2008)
“• Reviewed why intuition in selection decisions is so popular
• Because of the widespread belief that 1) it is possible to obtain a near-perfect precision in predicting human performance and 2) experience/expertise increase ability to predict human perf
• Its impossible to expect near perfect prediction of human performance and research does not support expertise and better skill at predicting human perf in unstructured interviews
• Common responses to limitations of intuition-based selection:
o Analytical selection decision aids do not take into account “broken-leg” incidents/rare events
o Need to evaluate candidate based on configuration of traits and not just 1 trait”
Kuncel (2008)
“• Ways to overcome overreliance on intuition in personnel decision-making by increasing appeal of effective predictors, incorporating human judgments into mechanical systems, improve acceptance of more effective data combination methods
• Present prescreened applicants to experts and let them make decisions
• Emphasize select-out or select-in methods
• Tell more narrative stories based on data
• Train raters to avoid irrelevant information
• Use data combination methods that allow human input
• Present research findings in other metrics besides correlations
• Use more high-fidelity decision aids e.g. cognitive ass
• Pit experts against each other and provide hard-to-avoid feedback making experts aware of their shortfall”
Colarelli & Thompson (2008)
“• Evolutionary view of why humans prefer intuitive decision-making as opposed to using rational decision aids
• Humans rely on face-to-face interaction a lot bcoz of evolutionary past
• Situation isn’t as bad as Highhouse (2008) described because ppl do use decision aids when making decisions about ppl they don’t have to interact with or for personal advantage
• Given our understanding of human evolutionary tendencies, we need to adapt our selection decision aids accordingly”
Klimoski & Jones (2008)
“• Accused Highhouse of neglecting the org/work context surrounding personnel decisions – usually done in a group setting
• Shift the problem from individual to systems level to address the need to further understand org contextual factors that might influence personnel decisions”
“• Most businesses in Europe composed of small and moderate enterprises – less than 10 ees each
• HR functions and tools that are proposed by IO psychologists aren’t relevant and applicable to these companies
o Might explain why less use of selection decision aids
• In order to entice these companies into using more objective tools, use the language of money – conduct utility analysis”
Martin (2008)
• Managers might overrely on obj tests too when faced with threat of hiring potentially unqualified candidate, when hiring for higher level positions, and when interviewer/decision maker is at a senior level in the org
Guion (1998): ch. 1
“Personality: a mixture of values, temperament, coping strategies, character and motivation
o State = temporary condition or mood vs. Trait = habitual way of thinking or doing in a variety of situations
Five Factor Model = Extraversion. Agreeableness, Conscientiousness, Emotional Stability, Openness to Experience
o 2 meta-analyses had positive finding on the criterion related validities for the five dimensions (Barrick & Mount, 1991; Tett et al., 1991; Ones, Schmidt & Viswesvaran, 1994).
o Criticisms of the big 5 exist
Non-Cognitive Predictors: Physical Characteristics, Physical Abilities, Psychomotor Abilities, Experience, education and training, Person-Situation Interaction, Predictors for team selection:
Developing Predictive Hypotheses: - Linking predictors to criteria is a local hypothesis based on local job and need analysis, criterion constructs are dictated by organizational needs”
Guion (1998) Ch 4
“Title VII: Employers may not fail or refuse to hire based on race, color, religion, sex, national origin
Office of Federal Contract Compliance Testing Order 1968: Contractors should validate tests in selection decisions if they had an adverse impact on protected group
• Adverse Impact = discrimination affects different groups differently; evidence that a group as a whole is less likely to be hired – requires justification of business necessity
• Disparate Treatment = evidence that a candidate from a protected group is treated differently from other candidates in employment process
Validation Requirements According to Guidelines: Criterion related, content, and construct validation
1991 Amendment to Civil Rights Act: prohibited race norming, quotas, defined business necessity and job relatedness
Affirmative Action: to reduce effects of prior discrimination; can be voluntary or required
Age Discrimination Act of 1967: prohibits discrimination against anyone 40 years or older – decisions should be based on ability NOT age
Americans with Disabilities Act of 1990: prohibits discrimination against qualified people who have disabilities”
Guion (1998) Ch 5
“Classical test theory = any measure is the sum of a true score plus an error score
Reliability is the extent to which a set of measurements is free from random error variance
Reliability coefficient –correlate 2 sets of measures of the same thing from the same people
Validity = the extent to which systematic sources of variance are relevant to the purpose of measure
Accuracy = measure must be highly correlated with a standard measure of the same thing and the relationship must be linear (x=y)
Sources of unreliability in measures - Thorndike (1949) pointed out that reliability depends on the reasons for individual differences in test performance
Reliability is computed by correlating 2 sets of measures presumably measuring the same thing in the same people in the same way
• 2 systematic scores are expected to be the same, whereas random error lowers the correlation
• If the source of variance is consistent in 2 measures it is treated as systematic
Different methods of operationalizing reliability make different procedural and mathematical assumptions and define error differently – estimates should make sense for circumstance (coefficients of stability, equivalence, internal consistency, interrater agreement)
Standard error of measurement – reliability of individual score may be important in selection
• Use to determine: if 2 people’s scores differ significantly, if a person’s score differs significantly from hypothetical true score, if scores discriminate differently in different groups of people
Psychometric Validity Evidence: Evidence based on: test development, reliability, patterns of correlates, outcomes”
Guion (1998) Ch 7
“Criterion related validity: relationship between predictor scores and the criterion used in the analysis
Validation as hypothesis testing - Criterion related validation tests the hypothesis that Y is a function of X
Regression: permits prediction (not always linear)
Contingency Tables: often gross categorical predictions can be sufficient for some organizational purposes
• Practical compromise between dichotomous, pass-fail use of predictors and statistical regression
Correlation: describes how closely 2 variables are relate, correlations permit inferences about the degree of prediction error based on regression function
• Residual = difference between the observed Y and the Y predicted by the regression
• Pearson/Product Moment Coefficient: influenced by Nonlinearity, Homoscedacity, Correlated error, Unreliability, Reduced variance, Extreme skewness, Group heterogeneity, Outliers, Unknown error
Statistical significance: Probability that r differs from 0 by chance
• Null hypothesis is ALWAYS false in real world
• Type I Error = null is actually true but it is rejected
• Type II Error = null is false but you fail to reject it
• Statistical power is the probability that a statistical test will lead to the rejection of the null and is a function of N, effect size in the population, alpha level chosen”
Guion 1998 C 8
“Linear additive model – scores are summed to form a composite often with different weights for different variables
• Summing scores is compensatory because a person’s strength in one trait may compensate for weakness in another
• Multiple regression finds optimal weights for the several predictors to get the best correlation with the criterion
• Suppressor: A valid predictor may contain an invalid, contaminating variance component and a variable that does not predict criterion but is correlated with contamination may improve prediction
Noncompensatory Prediction Models
• Mutiple cut scores/multiple hurdles – uses a cut score fore each of 2 or more tests; used when each trait is so vital to performance that other strengths can not compensate or if their variance is too low to yield correlation
• Conjunctive model – based decision on predictor that minimizes the estimated criterion value; no other score in the set no matter how high can compensate for an unsatisfactory prediction based on minimum score
• Disjunctive model – based on predictor with the highest estimated criterion value; no matter how poorly one performs on some other variable the decision is based on candidate’s strength
• Multiple regression requires cross validation (but samples may be independent so systematic error is different)
Synthetic Validity - Inferring validity of the test in a specific situation from a logical analysis of jobs into their elements, a determination of test validity for these elements, and a combination of elemental validities into a whole
“Causal” Research: Quasi-experimental research – causal inference is desired but not assured because no random assignment “
Guion (1998) Ch 11
“• Performance tests - used to assess proficiency, skill or knowledge at the time of testing, tests for maximal performance and may be inappropriate to infer actual performance
• Work samples and simulations: High criterion related validity, sample of a job content domain taken under standard conditions, simulations imitate actual work but omit its trivial, time consuming, dangerous or expensive aspects – high and low fidelity simulations are used
• Developing work samples begins with job analysis, uses only frequent tasks
• Non-cognitive Performance: Physical abilities, Fitness testing, Sensory and Psychomotor Proficiencies
• Assessment of Basic Competencies
o Competency = here and now performance
o Ability = aptitude for future performance
o Basic competency = acceptable performance of simple things a person must do on a job”
Gotman (2003)
“Civil Rights Act of 1991 - adverse impact unlawful if the defendant “fails to demonstrate that the challenged practice is job related and consistent with business necessity
• Adverse Impact light (Beazer): When the job clearly requires a high degree of skill and the economic and human risks are great, the employer bears a lighter burden to show that employment criteria are job related.
• Adverse Impact Moderate (Griggs-Albermarle):
o Duke Power relied on Title VII language making it legal to use professionally developed ability tests.
o EEOC guidelines: a test must fairly measure the knowledge or skills required by the particular job which the applicant seeks, or which fairly affords the employer a chance to measure the applicant’s ability to perform a particular job or class of jobs.
• Adverse Impact Heavy (Dothard):
o Bona fide occupational qualification: 1) qualification is reasonably necessary and either 2) all individuals excluded are in fact disqualified or 3) that some of the individuals so excluded possess a disqualifying trait that cannot be ascertained except by relevance to the BFOQ”
“• Meta analysis: applicants who hold positive perceptions about selection are more likely to view organization positively and report stronger intentions to accept job offers and recommend employer to others
• Org justice theory - applicants view selection procedures in terms of the 4 facets of org justice and these influence reactions
• Applicant perceptions include: view about dimensions of org justice, thoughts and feelings about testing, test anxiety and broader attitudes about tests and selection in general
• Applicant reactions are related to a number of org outcomes, companies that promote fairness and use job related selection tools may be less likely to become targets of employment discrimination lawsuits”
Hogan & Holland (2003)
Hulin (2002)
“• Work provides a source of: identity, relationships outside the family, obligatory activity, autonomy, opportunities to develop skills and creativity, purpose in life, feelings of self worth and self esteem, income and security and it gives other activities such as leisure time meaning
• Most frequent reason given for delayed beginnings of families or marriage is work
• We spend more time engaged in work activities than any other single activity
• There is evidence that events at work create emotions and emotional reactions that spill over into nonwork behaviors and health; little evidence that the spillover goes in the opposite direction
Contributions of I/O Psychology to the Lives of Employees
o If appropriately done selection and training provide better person-job fit than that provided by a random selection system and unstructured on the job training
• Theories of work behavior should be a foundation for general theories of behavior
• Theories of behavior have emerged from I/O research (goal setting theory, theories of affect and evaluations)”
Lievens et al. (2005)
Mael et al. (1996)
“Biodata - good validity, but evidence that perceived invasiveness of selection measures leads to a more global negative perception of the organization
Attributes of invasive items: Verifiable, Controllable, Negative, Transparent, Personal
Individual Difference Characteristics examined: Introversion, Self disclosure, Need for privacy, Gender, Age, Education level, Attitudes/experience
• Two studies: 1) social scientists and undergrads asked to rate 24 items 2) Army officers, students and SMEs rated 60 items (including original 24)
• Verifiable and transparent items seen as less invasive and personal items seen as more invasive; negative items seen as no more invasive than positive items
• Relatively weak contribution of individual difference variables is encouraging
• 4 categories of objections to invasive items emerged:
o Trauma – caused persons to reflect on and re-experience traumatic events
o Stigma – self incriminating items
o Religion and politics – included claims that items were biased for or against religious persons
o Intimacy –reawaken traumatic memories or could open respondent to unwanted attention “
Motowidlo (2003)
“• Job Performance: total expected value to the organization of the discrete behavioral episodes that an individual carries out over a standard period of time (refers to behavior that can impact organizational goal accomplishment)
• Campbell’s Multifactor Model: defines 8 behavioral dimensions of performance:
• Job-specific task proficiency, Non-job specific task proficiency, Written and oral communication, Demonstrating effort, Maintaining personal discipline, Facilitating team and peer performance, Supervision, Management and administration
• Task (activities that usually appear on formal job descriptions) vs. Contextual (behavior that contributes to organizational effectiveness through its effects on the psychological, social, and organizational context of work
• Organizational Citizenship Behavior: Altruism, Conscientiousness, Sportsmanship, Courtesy, Civic virtue
• Counterproductive Behavior: any intentional behavior on the part of the org. member viewed by the organization as contrary to its legitimate interests; must be both intentional AND contrary to organizational interests
• The General Performance Factor: Explains 49% of total variance in supervisory ratings of performance (Viswesvaran, Schmidt, & Ones, 1996). Probably determined by general mental ability and conscientiousness
• Antecedents of Job Performance: personality, ability, experience, knowledge, skills, performance
Significant paths: 1) extraversion, ability, and experience to knowledge 2) ability, experience, neuroticism, and knowledge to skill and 3) skill to performance”
Murphy & DeShon (2000)
“• Parallel test model is not useful for evaluating the psychometric characteristics of ratings; Generalizability theory should be used instead.
• Use of a parallel test model to estimate the reliability of ratings requires two assumptions:
a) Variability in ratings can be broken down into that due to true scores and that due to error (classical test theory isn’t wrong, but it applies to few situations)
b) Raters represent parallel measures (raters can rarely be thought of as parallel tests)
Reliability and validity are closely related concepts
• There are many systematic variables that lead raters to disagree when providing ratings of the same target
• If the errors are due in part to the systematic effects of variables other than the target construct then part of the error variance is due to sources of invalidity
• Error variance - indexed by the reliability coefficient - is in part an assessment of the validity (or invalidity) of the measure
The meaning of True Scores
• ““True scores”” have little to do with ““true performance””
• One of the many purposes of generalizability theory was to try and disentangle the many sources of variance that are lumped together under the heading of true score in the classic theory”
Sacco et al. (2003)
Schleicher (2002)
“• Examined different types of evidence to demonstrate the effectiveness of frame of reference training in improving the construct and criterion-related validity of ACs
• Frame-of-reference (FOR) training is an intervention designed to improve the validity of trait judgments in the performance appraisal context by eliminating idiosyncratic standards held by raters and replace them with a common frame of reference for rating (Bernadin & Buckley, 1981)
• Results showed that FOR training improved the reliability and the accuracy of AC ratings
• There was also improved discriminant validity associated with the FOR assessment ratings in the form of smaller heterotrait-monomethod and heterotrait-heteromethod correlations and somewhat improved convergent validity in the form of larger correlations with external measures of the same and similar constructs
• FOR training significantly improved the criterion-related validity of the current AC for predicting supervisors’ ratings of job performance.
• The results suggest that FOR training should be useful in AC practice given that it is no more lengthy or expensive than control training, enhances the legal defensibility of the AC, and appears to enhance AC validity”
Shippmann et al. (2000)
“• Growing popularity of competency modeling, due to restructuring of work and concerns that traditional job analysis are unable to play a central role in the modern human resource management
• Core competency- KSAOs that are associated with high performance
• SMEs – job analysis is more work- and task-focused (“what” is accomplished) and competency modeling is worker-focused (“how” work is accomplished)
• Competency - successful performance of a certain task or activity, or adequate knowledge of a certain domain of knowledge or skill
• Competency modeling more for training and development vs. Job analysis more for selection, performance appraisal, and other human resource decision making applications
• No single type of descriptor content is best suited for all purposes; most useful model has the right level of detail for the required use”
Werner & Bolino (1997)
Borman et al. (2002)
“• Overview of the steps taken in Project A job analysis which became an established part of personnel management procedures after World War II
• Measurement issues: 1) Context specific (job specific) versus General 2) Unit of description: Job-oriented (tasks) vs. Person-oriented (KSAs), 3) Level of specificity/generality at which content of work is described
Project A Task Analysis for Entry Level Position – purpose: to support job performance criterion development efforts (assumption that job performance is multidimensional)
• Steps in the job analysis procedure used in Project A: 1) Specify the task domain: synthesized existing job descriptions provided by Soldier’s manuals and the Army Occupational Survey Program (AOSP), 2) Subject matter expert judgments, 3) Selecting tasks for measurement
Critical Incidents Analysis - alternative method to identify critical dimensions of performance for the jobs
• Identify job-specific performance dimensions, Identify army-wide performance dimensions
Job analyses for Second-Tour (upper level) positions
• Used procedures similar to those used for entry level positions, but focused on supervisory/leadershipcomponents of these higher level jobs (used Supervisory Responsibility Questionnaire and Leader Requirements Survey to understand what type of leadership tasks needed for these positions)
• Resulted in 9 supervisory dimensions”
Sackett et al. (2003)
Schmidt & Oswald (2006)
Clevenger et al (2001)
“• Advantages of SJIs (situational judgment inventories):
• Relatively easy and inexpensive to develop and display at least moderate validity
• Differences in mean scores between racial subgroups typically smaller than those for ability tests
• Perceived to be face-valid and have job relevance
• Measured cog ability, conscientiousness, job experience, situational judgment, job knowledge, and job performance in 3 samples
• SJI appears to be a valuable alternative to other measures, and also an important source of incremental value in a test battery (SJIs were not very highly correlated with the other predictors)
• SJI displayed less adverse impact than all measures except conscientiousness”
Gottfredson (1994)
Guion (1998) 1
“• Cumulative effects of hiring decisions can result in substantial increases in mean performance levels and productivity
• Successive hurdles: partial assessment of all candidates, those who pass first cut are assessed further
• Simultaneous assessment: all assessment completed at roughly same time
• Five Assumptions in Personnel decisions: 1) people have abilities, 2) people differ in any given ability, 3)relative differences in ability remain about the same even after training, 4) different jobs require different abilities, and 5) required abilities can be measured
• Many orgs make initial selection decision based on potential for growth”
