Exam 3 Flashcards
Be able to calculate, understand, and interpret z scores.
Z-Score = (raw score – mean) / standard deviations
- The z-score is essentially a calculation of how many standard deviations a raw score is away from the mean
Why are measurement and statistics so important to staffing?
Because talent analytics are based on scores which are obtained from measures
What are the four scales of measurement?
- Nominal
- Ordinal
- Interval
- Ratio
Nominal scale
exist when the scores represent simple categories but provide no additional information
- Ex. Gender, race, or country
- Scores only tell us which category something is from
Ordinal scale
provide slightly more information, telling us about category membership and rank order
- Ex. Forced-ranking performance assessment programs, school grades, or “top ten cities”
- These scales don’t tell us whether the differences between each rank are the same
Interval scale
account for both rank order and ensures the differences between each rank order are the same
- Ex. Common 5-point or 7-point scales used in surveys and performance management systems
- These scales do not have a true zero
Ratio scale
possess all the characteristics of an ordinal scale, but also include a true zero point
- Ex. Money, temperature, or weight
Understand why scales that provide more information are more valuable for staffing.
the more information provided by a measure, the more insight that we have into the concept, the better distinctions we can make, and the more sophisticated analytical models can be used to make predictions
Describe how correlations are used in selection to estimate criterion-related validity?
- correlation measures the relationship between 2 variables and is used to assess validity to evaluate how accurately a test measures the outcome it was designed to measure (between a predictor and outcome)
- concurrent = present
- predictive = future
What is selection?
refers to the strategies, processes, methods, and practices used to assess applicants on the job- and organizationally related KSAOs
Why is selection strategically valuable? (S.I.C.E.E)
- it shapes the nature of talent resources
- implements the firm’s strategy to drive competitive advantage
- changes or reinforces the company culture
- effects of selection are difficult and costly to duplicate
- enhances analytics and quantitative insights
What is the basic logic of selection?
A process to assess those applicants in order to determine who meets or exceeds qualification standards and thus should receive job offers
How is “selection” a bet on the future?
Based on the incomplete and imperfect information available today
What is validity?
Accuracy- are you measuring what you’re actually trying to measure?
What are the 3 types of validity?
- criterion-related
- content
- construct
Criterion-related validity
- Empirical relationship between predictor and outcome (criterion) variables. Usually estimated using correlation/regression
- Ex. Wanting to know if a college entrance exam will predict future academic performance… first semester GPA would serve as a criterion-related variable since it is an accepted example of academic performance.
Content validity
- Expert judgment about the conceptual overlap between predictors and outcomes
- how well a test covers all of the relevant parts of the construct it aims to measure
- Ex. Taking a driving test to obtain your driver’s license
Construct validity
- Whether scores on some assessments represent the underlying attribute they are intended to represent.
- Ex. A measurement of the human brain; to measure intelligence
What are predictors?
(selection methods)
- Provide scores about latent talent used for predicting performance outcomes
- Ex. Sign and samples
What is criteria?
Standards to measure candidates to determine whether or not they have the qualifications to perform the job
How are predictors and criteria related?
To determine the quality of hire
What are the 7 steps of selection?
1- identify desired talent
2- ensure compliance w/ legal and political guidelines
3. determine cultural influences
4. determine selection methods
5. structure the selection process to balance trade-offs
6. make selection decisions
7. evaluate the selection process
What are the 2 predictor methods?
- sign; homogeneous
- sample; heterogeneous
Sign KSAOs
homogeneous
- cognitive ability
- knowledge and skill
- personality
- work interests, styles, and values
- biodata
- fit
- situational judgment test
Sample KSAOs
heterogeneous
- situational judgment test
- interviews
- assessment centers
- work samples
- simulations
Why do signs and samples matter?
Provides information on an associated dependent (Y-axis) variable regarding a particular outcome
Sign= usually self-reported surveys
Sample= behavioral simulations
What does it mean to say the best predictor of future behavior is past behavior?
Suggests that a person’s previous actions and behaviors are a strong indicator of how they are likely to behave in the future
What types of predictors are based on “future behavior is past behavior” principle?
Interviews: (behavioral and situational questions)
Simulations: (situational judgment tests, assessment centers, work samples)
What is cognitive ability?
Refers to how well individuals can manipulate, process and remember data
What does cognitive ability predict?
- Job performance
- Occupational “prestige”
- Salary
- Promotion rate
- Basically, every desirable criterion of interest!
Cognitive ability is among the highest validities of anything we use, so what is the problem with using them?
- Cognitive ability often produces larger demographic mean differences than nearly any other test
- Using the “most valid” test will result in lower diversity
What is personality and how is it used in selection?
Consistency in behavior and people’s personalities are set
Why is personality related to performance?
Adverse impact tends to be low
What is the five-factor model? Be sure to describe and give examples of each trait.
- Neuroticism (emotional stability) - refers to one’s level of emotional instability, anxiety, and negative emotionality
- Extraversion - This trait refers to one’s level of sociability, assertiveness, and positive emotionality
- Agreeableness - refers to one’s level of cooperativeness, kindness, and empathy
- Openness to experience – willing to try new experiences
- Conscientiousness - refers to one’s level of organization, self-discipline, and responsibility.
What is the underlying logic of the interview?
- Past behavior is the best predictor of future behavior.
- If interviews ask Qs to gauge applicant’s level of competence on job-related KSAOs (from job analysis), then this maximizes the capture of job-relevant info and minimizes time spent on irrelevant Qs
Why are interviews so frequently of poor quality?
- Not properly developed and administered to be structured and streamlined across candidates, such as basing interview Qs on job analysis and asking the same Qs of all applicants.
- Interviewers are not trained and process fails to use multiple interviews and interviewers
What is interview structure?
- ask same Qs for all – minimal deviations from the Qs
- minimize use of prior info (such as interviewer knowing the details of schools attended, experience, socioeconomic factors, etc)
- Use behavioral rating scales
- Interviewers are trained
How does Structure interviews relate to validity, diversity, etc.?
the most valid predictor method, also most accurate, consistent, legally defensible, and able to reduce bias and discrimination
Behavioral interview questions
(experience-based), ask Qs about past behavior, validity .30+
Example: Tell me about a time you… (teamwork, problem solving, etc.)
Situational interview questions
ask Qs about a hypothetical behavior, validity .25+
Example: Imagine you were in a situation where…How would you handle it?
Job knowledge interview questions
ask Qs about specific job skills, validity .25+
Example – How do you develop a presentation using x software? Talk about processes, program-specific knowledge, etc.
What do interviews measure?
- general cognitive ability
- knowledge and skills
- personality, social skills
- interests/preferences
- org fit
- physical abilities
What kinds of KSAOs should we use interviews to measure?
Interviews should be used to measure JOB-RELEVANT KSAOs (from job analysis)
How strongly correlated are the different types of interview questions?
Almost zero correlation! (between situational and behavioral)
We know behavioral interviews predict better
What kinds of interviews do applicants/managers tend to prefer? Why?
tend to prefer UNstructured, so it is important to explain why a structured interview is being used
Why they prefer unstructured – feel like they can’t get to know people on a personal level, very boring to ask the same Qs over and over to different candidates
Know (very well!) the guidelines for conducting effective interviews.
5 best practices:
- Base Qs on job analysis
- Ask same Qs
- Train interviewers
- Use Systematic & Behavioral scoring
- Multiple Interviewers & Interviews
Describe the basic purpose and logic of simulations.
tests are SIGNS of behavior, simulations are SAMPLES of behavior.
What is fidelity?
applicants complete similar/identical tasks to those required on the job
Higher fidelity = higher validity
How does fidelity factor into our discussion of simulations?
lower fidelity = situational/judge
higher fidelity = work sample
What is a situational judgment test (SJT)? Do you understand the key characteristics of SJTs, validity, etc?
SJT – present applicant with work-related situation, ask how they will handle it
Example: Coworkers are not doing certain tasks that are needed to get done. Would you: A) Perform tasks yourself, B) Tell your boss, C) Convince coworkers to split up the tasks. Correct answer would get +1 points, others would get 0 or -1
What is an assessment center?
applicants participate in multiple exercises (avg of 5), each exercise measures multiple competences (avg of 10). Raters evaluate the competences across exercises.
Give some examples of different assessment center methods.
- In-basket
- Leaderless group discussion
- Roleplay
- Mock presentation
- Business game
What is the distinction between exercises and dimensions?
- exercises ARE the methods to elicit info about the candidate’s level of the desired behaviors/competencies
- dimensions ARE the KSAOs/competencies
What are the most common exercises used in selection?
- work samples
- gamification
What are the most common dimensions measured in assessment centers?
- interpersonal skills
- communication (oral & written)
- motivation & achievement
- persuasion & influence
- organization & planning
- problem solving
What is the validity, diversity, legal issues, etc., of assessment centers?
Pros: high validity 0.30+, adverse impact is lower, candidate experience is favorable
Cons: can be expensive to develop/administer, can look dated very quickly
Be familiar with the guidelines for effective use of assessment centers/simulations.
6 best practices
1-Predictor method gives scores that are construct valid and reliable;
2-Predictor method gives scores that are job-related;
3-Use multiple Qs/items that are the same across applicants;
4- Qs are understandable and instructions are clear;
5-When assessors are used they are: trained, multiple, avoid prior info, held accountable for ratings;
6-Continually monitor scores of assessments for reliability and validity
Explain compensatory
best used when high scores on one predictor are the most important factor and can trump or make up for low scores on another predictor
Advantages: minimizes errors, optimal predictor combo bc can use regression, very flexible
Disadvantages: must have large sample size, must assess all applicants on all predictors
Explain noncompensatory
(two types: multiple cutoffs or multiple hurdles)
Multiple cutoffs – best used when you must cut people who score too low on certain predictors due to legal/safety reasons
- Adv: narrows applicant pool to only those minimally qualified, easy to use and explain, low cost/efficient, sometimes ADA compliance
- Disadv: only identifies those with minimal qualifications, must assess all applicants on all predictors
Multiple hurdles – best used when KSAOs are not compensatory or training is infeasible
- Adv: Same as multiple cutoffs but even less costly, and only assess candidates who pass earlier hurdles
- Disadv: need established validity of each predictor and justify cutoff for each, increased time required for implementation
Understand the benefits of using regression for selection decision making. What are the limitations or potential risks with using regression?
Benefits – it is the best way to combine the result of multiple predictors to predict performance (better than human experts!); can help to identify the most important predictors out of a set
Limitations/risks of regression – bc it’s only based in statistics, it doesn’t account for possible adverse impact or cost of using certain predictors
Why is the best approach a hybrid of rational (expert judgment) and empirical (regression) approaches?
- Uses regression first to identify the most important predictors
- and then use expert judgement to identify which of those chosen predictors have issues that make them less desirable to use
Understand the logic of cut scores, and how raising/lowering a cut score will affect the number of true positives, true negatives, false positives, and false negatives.
- Setting cut scores means there is a certain minimum score applicant much reach to be considered for the job
- Pro of raising the cut score: decreases false positives (people who would be hired but can’t do the job)
- Con of raising the cut score: increases false negatives (people who you don’t hire but could have done the job)
Why should we generally avoid setting cut scores?
- Cut scores are a major concern for legal reasons – adverse impact
When are cut scores sometimes necessary?
- Cut scores should be used when there are minimal qualifications for safety or legal reasons, or when there are large # of applicants who will never be hired
