LECTURE 6 NOTES Flashcards
Two views of clinical prediction
- clinical intuition
- statistical prediction
Effect on prediction
- expertise
- feedback
Prediction
a judgment made about an outcome before the outcome knowledge is known
Examples of Predictions
Will Connor bedard become an elite skate in the NHL?
Will this person stuggle with clinical depression?
Profiling
Creating or writing an outline or article that describes a person
Clinical Intuition
“people with experience and expertise in the domain make intuitive prediction for individual cases” (Kim, 2017, p105)
Clinical Intuition pt 2
*Used by clinicians with a lot of experience
profiling cases
*Interview patient about life-story
*Consider important facts and make decision *Use experience and intuition to understand the
interactions between descriptive statements of the case
How useful is clinical intuition for successful diagnosis of disorders
Statistical Prediction
“prediction about a particular case are made solely on the basis of empirical evidence and/or a statistical comparison to data drawn from a large sample” (Kim, 2017, p105)
Statistical Prediction pt. 2
*Based on Empirical data set.
*Data set is usually large
*Predictions based on comparison of scores from
a patient with scores from data set
Statistical Prediction Approach In A Nutshell
Example: Minnesota Multiphasic Personality Inventory (MMPI)
*Standardized test of psychopathology *Used to diagnose personality disorders *Tested and replicated with large samples
STATISTICAL PREDICTION APPROACH IN A NUTSHELL II
Does “Mark” suffer from depression?
*Administer the MMPI to “Mark”
*“Mark” responds to items on the MMPI *Apply statistical/Mathematical formulae to compare to a standard
*Will a typical person with “Mark’s” scores on the
MMPI be clinically diagnosed with depression?
A DISCUSSION: HOW USEFUL IS STATISTICAL PREDICTION FOR SUCCESSFUL DIAGNOSIS OF DISORDERS?
Blind Empiricism
States conditions when question is useful:
*A question is useful if people’s responses to a question can predict behaviors or disorders *As long as question can make accurate predictions, question is useful even if not based on reason or on theory
Clinical vs. Statistical Prediction
Clinical
- Based on interviews
- Follow-up Questions can be asked based on responses provided by patient
- Qualitatively rich source of information
Statistical
- Based on direct reports or self- reports of patient
- Statistical comparison of patients reports to large data set
- Prediction about diagnosis based on statistical comparison with responses from large data set
WHICH IS MORE ACCURATE AT MAKING CLINICAL DIAGNOSIS:
CLINICAL INTUITION VS. STATISTICAL PREDICTION
“Mechanical prediction is typically as accurate or more accurate than clinical prediction.”
“Even though outlier studies can be found, we identified no systematic exceptions to the general superiority ( or at least material equivalence) of mechanical prediction.”
(Grove et al, 2000, p25)
The evidence suggests that mechanical prediction is superior
In the outlier studies, can any characteristics be identified that suggest an advantage for using clinical intuition over mechanical prediction?
“In seven of the eight studies, the clinicians received more data than the mechanical prediction.”
However, for most of the reported studies, “clinical predictions were outperformed by a substantially greater margin when such data was available to the clinician.”
(Grove et al, 2000, p25)
Grove et al (2000) Question
Which is more accurate: Intuition or prediction?
“mechanical predictions of human behaviors are equal or superior to clinical predictions (intuition) methods for a wide range of circumstances.” (Grove et al, 2000, p19) see abstract
Meta analysis
An analysis of several analyses that answers a specific question
How to do a meta-analysis
*Perform a literature search and identify all articles that address the question (i.e., judgment based on intuition vs. judgment based on prediction)
*Be aware of any statistical/methodological issues and apply statistical technique to compensate
*Perform the comparison test(s)
*Identified 136 studies that were suitable for the meta-analysis
*Performed the meta-analysis to identify whether intuition or prediction was superior.
Why is clinical intuition inferior to mechanical prediction
- Ignore base rates and prior probabilities
- Overreliance on the representativeness heuristic
- Overreliance on the availability heuristic
- Assign nonoptimal weights to cues
Incorrect Decisions in Predictions
Background: Hypothesis Testing
Type 1 Error
- theory is not true
- decide theory Is true
Correct Decision for Type 2
- Theory is not true
- decide theory is not true
Correct Decision for Type 1
- decide theory is true
-theory is true
Type 2 error
- theory is true
- decide theory is not true
Making Predictions with a Single Case
Type 1 Error
John is not guilty of shoplifting
Verdict = guilty
Correct decision type 1
John is guilty of shoplifting
Verdict = guilty
Type 2 Error
John is guilty of shoplifting
Verdict = not guilty
Correct decision for type 2
John is not guilty of shoplifting
Verdict = not guilty
How can we reduce errors when making predictions?
Expertise
Feedback
Question: Does expertise have any effect on validity of judgments?
*Compared judgments of experienced clinicians with judgments of graduate trainees and inexperienced clinicians
*Found no difference in judgments
*Result replicated for risks of mental disorder and
likelihood of future criminal behavior
There is no evidence that expertise has an effect on validity of judgments
In the domain of clinical/medical diagnosis is feedback useful?
Effects of Feedback in Clinical Diagnosis
Definition: Feedback
*Information about a person’s performance to a
task
*Information is used as a basis for improvement.
*The goal is to improve correct decision **In clinical diagnosis, the outcome of the decision is seldom known with certainty
Predictor
*Information that people use as a signal to lead them to a decision
*The outcome of the decision is not known
Validity
*Information that is correlated with the accuracy of a judgment or decision
*Using a valid predictor to make a decision is more likely to lead you to a correct decision than using an invalid predictor.
Einhorn & Hogarth (1978)
What kind of feedback is useful?
*Diagnosis based on personal experiences and observation
*Personal theory of valid predictors
ANSWER:
*Corrective feedback on validity of predictors
*Direct attention away from invalid predictors
*Direct attention toward valid predictors
Frequent, consistent and immediate feedback
- Weather forecasts
- Sports predictions in a regular season
- Clinical Diagnosis
- Verdicts in the justice system
Stewart, Roebber, and Bosart (1997) How accurate are predictions of weather forecasters compared to computational or statistical predictions?
Weather forecasters are at least just as accurate as statistical models
study for weather predictions
Task
Predict the following:
*Minimum temperature *Maximum temperature *Precipitation in a 12-hour period *Precipitation in a 24-hour period
forecasts from three sources
*Faculty (experts) and student forecasters from a department of atmopheric sciences
*Multiple regression model
*Operational forecasting model (Model output
statistics or MOS)
Results
*Human forecasters were more accurate than predictions of the operational forecasting model
*Human forecasters were about as accurate as the multiple regression model
*High reliability across predictions made by human forecasters
*Weather forecasts are more accurate than clinical diagnosis (Faust, 1986)
WHY ARE WEATHER FORECASTERS MORE ACCURATE THAN CLINICIANS AT MAKING
PREDICTIONS?
*Consistent and immediate feedback about predictors
*Practice effects
*Easy-access to base-rate information
WHY ARE WEATHER FORECASTERS MORE ACCURATE AT PREDICTIONS THAN CLINICAL
PSYCHOLOGISTS?
task characteristics
*Consistent and immediate feedback about
predictors
*Task predictability based on predictors *High and precise quality of predictors *Model that provides accurate forecasts
