Assessment Centers, SJTs, Simulations Flashcards
What three things determine the predictive validity of an assessment center?
- The validity of the predictors: Even if you have a good assessment center, if the predictors are not valid, the scores for predicting performance won’t be valid either. For example, psychologists with more training may provide more validity
- The intercorrelations among the predictors
- The way in which data are combined: For example, combined clinical data doesn’t have as much predictive validity as mechanical combinations
If you wanted to create an assessment center with very high level of validity what three predictors would you chose and why?
Cognitive ability tests, personality tests, and structured interviews. All of these are strong predictors.
- Cognitive ability tests have been shown to have a linear relationship with performance.
- Personality tests can predict work behaviors and traits such as conscientiousness, which relates to motivation in the workplace.
- Structured interviews can be used alongside a job analysis to find the best questions that would relate to the job, making sure that all questions contain all relevant information without omitting anything.
List and describe three different types of simulations.
Oral Presentation: The person is given a certain amount of time to prepare a presentation based on given information. Usually, this would be related to presenting financial results or making business decisions based on information.
Role Play: The person can be asked to act out a certain scenario. We would look at how the person would handle the situation. The situation may be pitching a sale or dealing with a subordinate.
In-Basket: The person is given a series of papers to simulate a typical in-basket; the applicant may be expected to respond to messages as he/she/they would on the job. Modern versions may include papers, voice mails and e-mails.
The Gaugler et al (1987) meta-analysis yielded an overall validity for job performance of r = .37. Explain why this number may be misleading.
This is misleading because the overall validity applies for assessment centers that we know nothing about, such as the as the measures and the data combination methods. Gaugler did found that the validity of assessment centers was variable and strongly moderated. However, this number represents an average validity, so there should be a considerable standard deviation. Therefore, the validity of an AC is dependent on how it is constructed
What is one explanation for the low validity of the assessment center?
One explanation is that many simulations in assessment centers are not based on a good job analysis. They are not carefully designed, and are subject to low reliability. Also, many assessment centers to do not use cognitive ability tests, personality tests, and structured interviews, all of which are strong predictors.
Lack of training from predictors can lead to low validity, a fewer number of predictors would mean more collective bias in their scores.
What are some of the elements of a traditional assessment center?
First, assessment centers are a series of realistic simulations used to evaluate the fundamental skills necessary for a job.
Second, traditional ACs use multiple simulations, evaluate multiple dimensions, use multiple observers and they combine information with a group process.
Third, simulations are focused on the most important behavioral areas of the job.
Fourth, the goal of a traditional AC is to produce a high-fidelity simulation for the job
Job knowledge and work sample tests can have substantial predictive validity. In what situation would we expect them to be of less use? Why?
In a situation where you’re measuring modern complex jobs because all of their core tasks cannot be measured in a sample.
Or in situations where people have no knowledge of the job tasks. Here, the test would then be inefficient.
Or in situations where you’re hiring people with the expectation of training them. Here the tests would be not be useful because work samples assume applicants already have some necessary skills
Or in situations where jobs are easily trained, as no experience would mean tests will not be useful because these jobs expect that people can be trained easily without any job knowledge.
What is a situational judgment test? Are they good for anything?
Situational Judgement Tests (SJTs) present situations to people and ask them to judge the quality of different responses to a situation. SJTs can be built for any job and are mostly specific to a job; they are also built from critical incidents or job analysis.
SJTs have reasonable predictive power of about 0.20-0.25. They can provide useful information about KSAOs including job knowledge, interpersonal skills, and integrity. They can be expensive to develop but are less expensive to use than simulations.
What is the predictive evidence for situational judgment tests and assessment centers?
For SJTs, The predictive evidence for situational judgements tests is 0.20-0.25, a reasonable predictive power. An evidence of this is the correlations between situational judgment tests and performance, and their use in decision making to improve interpersonal and intrapersonal skills for medical doctors, and medical school selection in UK and Canada. Here, performance grew stronger with each year of medical school because in later years, interpersonal and intrapersonal skills became more important.
For Assessment Centers, Gaugler found the average corrected validity to be 0.37. This validity of AC is variable and strongly moderated. Assessment Centers have the potential to combine the best predictors into one great predictor. As evidence, Kuncel and Sackett found that aggregating post-exercise dimension ratings (PEDR) can create validity greater than its parts. Predictive validity increases with good predictors, mechanical combinations, and simulations based on job analysis.
What is a work sample? What are they used for? How useful are they? What is a limitation of their use?
Work samples are direct samples of critical work tasks that are scored in a standardized way or subjectively scored by raters. They have a strong predictive power regarding performance, and they assume applicants have some of the necessary skills. Some of the limitations involve situations in which work samples are of less use. For example, in situation where you’re measuring modern complex jobs. This is because all the core tasks cannot be measured in a sample, so its easy to have a deficient measure
