Combining Multiple Predictors Flashcards
What is the difference between a mechanical synthesis and a clinical synthesis?
A mechanical method involves methods that use an equation or formula to combine information. This can be as simple as an average or as complex as a configural weighting machine. Mechanical methods combine information the same way every time.
- In the synthesis, the set of predictors are given to an expert to combine into a rating
- The rating (clinical judgment) is mechanically combined with other information to produce a final rating.
- This final rating is the mechanical synthesis
- The idea is to obtain insight of the clinical or expert judge and the structure of the mechanical combination
A clinical method relies on a human judge to combine the information using judgment.
- The set of predictors is combined using a mechanical method (like an average) to create a rating.
- The rating is given to the clinical who combines all of the information into a final rating
- The idea is to give the clinical a solid foundation for making a decision.
Sawyer (1966) made an important distinction between two things in his discussion of decision-making. What is the distinction he made and why is it important?
Sawyer distinguished between the method of measurement and the method of data combination.
Th method of measurement is between having a person evaluate someone vs. using an instrument that was evaluated in a mechanical way.
The method of data combination is between having a person combine multiple pieces to make a decision or use an equation to combine the information.
This is important because it separates methods for collecting information and methods that combine information. People are good at gathering data but not combining information together.
What is the overall critical finding from research on mechanical versus clinical decision making?
The most important finding is that its better to combine data mechanically and use more ‘mechanical’ structured methods. . A mechanical synthesis (clinical data combined mechanically) is better than clinical synthesis (mechanical data combined clinically)
People are generally good at collecting information, however, the selection system is most effective when the information is combined mechanically. Mechanical data combination methods are most effective and this is a consistent finding in psychology.
List and describe two different methods for mechanically combining data.
- Expert Weights: For expert judgment-based weights, we do a study of experts and ask them how they weigh the information. We ask them how they do it, write it down, and aggregate the information across the sample of experts This is also called policy capturing study, and the approach is only as good as the experts chosen.
- Meta-Analytic Weights: Use the information from a meta-analysis. This has the advantage of being based on a more stable estimate. Meta-analytic weights can be used in two ways. We can use the predictive validity as a weight, or use the predictive validity and the information about its correlation with other predictors.
These two methods are differential weights
What are two explanations for why clinical data combination tends to results in poor predictions?
First, some people can develop incorrect rules for making judgments and are resistant to changing them, and they are inconsistent about how they apply those rules.
Second, people can be swayed by unusual information that appears to them and they also have decision making biases
What did Tetlock find about the predictive judgment of experts? What tends to beat them?
Tetlock examined about 28,000 predictions regarding politics, economics, and public policy and found that they were only slightly more accurate than chance and a lot less accurate than simple algorithmic models. He argued this was because formulas were more consistent with judgment and not easily swayed by unusual information.
What are these foxes and hedgehogs Tetlock discusses in his research on expert judgment? Why are they important?
Tetlock discussed two types of forecasters.
Foxes tend to be less certain and less likely to have a metatheory that explains everything. Hedgehogs tend to see the distant past as deterministic, meaning they don’t see that past events are due in large part to chance, but they view the present and recent past as flexible and contingent. They also have a metatheory about how the world works.
This is important because different world views can impact forecaster’s predictive power. Foxes are slightly more predictive than hedgehogs because they’re more open- minded, cautious and able to integrate complexity and update their beliefs. Hedgehogs have extreme beliefs and hind-sight bias.
Kahneman, excerpts from “Thinking Fast and Slow”
1) List two key characteristics each for “hedgehogs” and “foxes.” Which of the two (that is, “hedgehogs” and “foxes”) is typically better at forecasting outcomes?
Hedgehogs have a one extreme theory about the world and they are impatient when others don’t see things their way – they are opinionated. Foxes are complex thinkers, they don’t seek one big theory but rather see the interaction of different forces including blind luck. The foxes have better forecasting outcomes, likely because they are more open-minded, cautious and better at updating beliefs than hedgehogs.
Describe two pieces of evidence for the accuracy of clinical versus mechanical data combination.
First, an economist created a formula to predict the value of wine based on rainfall and temperature. This formula was found to be more accurate than the judgement of experts: professional wine tasters.
Second, a study by Kuncel found that equations were better predictors of performance success versus expert judgment, particularly with strong predictive power for GPA and overall academic performance, training success, advancement, and job performance
Mechanical combination of predictor data has been shown to work better for 3 work related outcomes. What are they? Does this mean the expert or psychologist has No useful role in personnel selection? Why or why not?
Mechanical combination works better for predicting training success, advancement, and job performance (work related outcomes).
However, an expert or psychologist is still useful in collecting information and linking the experience of the applicant to the company. They are not as useful in combining the data to make a decision. Psychologists also play a role in how to mechanically combine predictors by deciding weights. They help collect data and make the formula, so if incorrect weights are used based on rapid observations of people, they can make unnecessary emphasis of certain information leading to a bad combination of data.
The “Framing” effect is one of the most powerful methods for manipulating people’s judgments. What is it and what effect does it have on people’s judgments?
The framing effect is when you ‘frame’ decisions as either a loss or a gain. In prospect theory, you find that when you frame losses, people are more risk-seeking – they are hoping for a large gain. In contrast, when framed as gains, people are more risk aversive – they are afraid of disappointment. Since people behave differently when specific gains or losses are imminent, its possible to control or manipulate their decision making.
List and describe one decision making bias (other than Framing). Why is it important?
Availability
People will make judgments about the likelihood of the occurrence of events or the frequency of things happening based on how easily they can access examples in their memory. This leads to distorted perceptions of risks and of people or groups due to publicity in the media.
What are the characteristics (either behaviors or individual differences) of Superforecasters? (List 4.)
- They are more actively open minded
- They may be more scope sensitive than others
- Their scores on Cognitive reflection tests correlate with their accuracy
- Their scores on Raven’s progressive matrices and abstraction tests correlated with their accuracy as well as their Year 2 political knowledge
What is the illusory correlation and why would it matter in I-O psychology (give a specific example)?
Illusory correlation is the strong tendency to generate beliefs about perceived causal explanations for events or relationships between phenomena. These beliefs are resistant to change. This is important because business decisions can be made looking at relationships between phenomena, and false correlations from observations or data can lead to biased decisions. Illusory correlations also contribute to stereotypes and implicit theories.
What is the sunk cost bias? What effect could it have on organizational decision making?
Sunk costs are when people are very aversive to abandoning a project after investing time or money in it, especially if its financial. They explain that further investment is warranted because resources already invested will be lost, not taking into consideration overall losses. This could affect an organization decision making because having for example invested in the training of a worker that is performing poorly with several CWBs, the company may want to keep the employee because of the investment, but this may lead to further losses.
