Topic 6: Risk Assessment Flashcards
What is a practical problem of risk assessment?
forensic psychologists are routinely called upon to include some statement about offender risk
treatment summaries, section 34 assessments, probation referrals, school board questions
not a “threat assessment”
how is this best done?
What is first generation of risk assessment tools?
expert opinion
usually a psychologist or psychiatrist
results in: “clinical impression”, “subjective assessment”, “professional judgement”
collection of information on individual and her/his situation
usually involves interview and file review
the nature of information gathered, and its interpretation is subject to personal discretion
What are the pros of first generation risk assessment?
always available to clinician
incorporates clinician’s experience
clinician has freedom to consider any data seen as relevant
may have some short term validity
What are the cons of first generation risk assessment?
questionable accountability and fairness
difficult to explicate decision rules as they vary from clinician to clinician
poor reliability and validity for medium and long term predictions
What are second generation risk assessments?
examples: VRAG, PCL-R, Static-99
basic approach is to identify variables empirically associated with risk and assign values, or weight, to each
may include dynamic/clinical variables, but often do not
designed to predict a particular outcome: approach employed by insurance companies, likelihood of violence (or other target) expressed as a relative probability
What are the pros of second generation risk assessments?
objective, scientific, lend themselves well to appropriate levels of scientific and legal scrutiny
affords consistency (reliability) and accuracy (validity)
less dependent on rater’s clinical experience
What are the cons of second generation risk assessments?
can be insensitive to changes (if only static variables are included)
optimization can restrict generalization
What is the Violence Risk Appraisal Guide (VRAG)?
assigns patients into one of nine categories or bins
each bin is associated with a specified probability of recidivism
How is the VRAG calculated?
- which is the highest correlation?
- take variance out; recalculate
- which next highest correlation?
- repeat through whole test
What is specified when reporting risk estimates?
risk cohort
type of recidivism
time frame / opportunity: years of opportunity, can’t use time when they had no opportunity, i.e., jail
underestimate?: how close are they to the standardization sample? is there something about them that changes their risk that isn’t measured (injury or sickness)
What information is not considered in a risk assessment statement?
this statement is about how likely a crime is to occur in a specified time frame
it is silent on: type of crime, the severity of the crime, what risk exists in the short term, when in that interval (or under what conditions) recidivism is most likely, or what might reduce the overall risk
ethical issue: if you conduct an assessment an argument can be made between if the psychologist should design treatment plan or just do risk assessment; usually person who performs treatment didn’t do risk assessment
What is the underlying risk model in second generation risk assessment?
the individual case could be higher or lower than the reference population
not comparing to general population
What are third generation risk assessments?
risk-needs assessment
arose with a “risk management” orientation
premise: certain needs are linked to criminal behavior, therefore need to alter needs, change likelihood of criminal behavior
criminogenic needs: dynamic risk predictors
emphasis on repeated assessments
example: HCR-20 Webster, Douglas, Eaves & Hart (1997)
three categories are: historical (static), clinical, and risk
What is the underlying risk model in third generation risk assessment?
risk = static risk + dynamic risk
if threshold is exceeded, offending becomes highly likely
since you change the static factors, must focus on dynamic
every individual has offending threshold
What are the comparisons between second and third generation risk assessments?
actuarial (2nd gen) approaches are good for long term prediction and stratifying offenders by risk level
SPJ (3rd gen) approaches are also about stratification, but are better at identifying the circumstances under which further offending will take place; periodic re-assessment is expected
don’t think of it as a competition between the two
What are the pros of third generation risk assessment?
imposes formal structure on evaluation but flexibility around decision-making
can be sensitive to changes
better than unaided clinical judgment
What are the cons of third generation risk assessment?
less evidence on reliability
less evidence on validity
no empirical optimization of item selection or weighting
accuracy unclear when scores adjusted by clinicans
How is the accuracy of assessments measured?
correlation coefficients: “r” scores
r2 = percentage of the variance accounted for by the correlation
there is a serious problem evaluating instruments on the basis of hit rate (correct classification); it is known as the base rate problem
base rate is the proportion of a reference population that exhibits has/undergoes a target attribute or event
What does a base rate of 50% say about the accuracy of a risk assessment?
let’s assume a BR of 50% recidivism in a particular group - say YO’s
we have a risk assessment instrument that is 90% accurate; i.e., 90% of individuals predicted to recidivate will, and 10% will not; also, 90% predicted not to recidivate will not, and 10% will
assume we have a sample of 1000 individuals we want to assess
true positive: 450 (90%)
false positive: 50 (10%)
false negative: 50 (10%)
true negative: 450 (90%)
does not over predict one thing, errors are made equally often
but BR is seldom 50%; is usually lower or higher
What does a base rate of 10% say about the accuracy of a risk assessment?
actually recidivate: 100
do not recidivate: 900
true positive: 90 (100 x 0.90)
false positive: 90 (900 x 0.10)
false negative: 10 (100 x 0.10)
true negative: 810 (900 x 0.90)
How does base rate impact the accuracy of a risk assessment?
as BR decreased false positives increase
as BR is raised false negatives increase
note that the test is still 90% accurate in both cases
which groups (in court) are most likely to raise this concern?
What are sensitivity and specificity in a risk assessment measure?
it’s often useful to describe an instrument’s ability to make correct predictions in terms of sensitivity and specificity
sensitivity = TP / (TP + FN)
specificity = TN / (TN + FP)
ideally, we’d like to maximize both
in practice, one usually comes at the cost of the other
What is sensitivity?
how good is it at finding what we’re looking for?
sensitivity = number of predicted recidivists divided by the total number of recidivists
probability of correctly classifying a future recidivist
What is specificity?
how good is it at not picking up on contradicting things, or things we are not interested in?
specificity = number of predicted non-recidivists divided by the total number of non-recidivists
probability of correctly classifying a future non-recidivist
In what way is sensitivity and specificity dependent on cut off scores?
a good risk assessment will:
1. produce a larger mean score difference for the two groups
2. produce reasonably small variance so as to minimize variance
What are AUCs (area under the curve)?
area under the curve of the receiver operator characteristic (AUC / ROC)
insensitive to base rates
measure to comparison
0.5 = no better than chance
0.60 - 0.65 = some accuracy
0.65 - 0.69 = better accuracy
0.7+ = much better accuracy
What does the AUC value reflect?
the proportion of non-recidivists that would have a lower test score than a randomly chosen recidivist
measured some time after the risk assessment was completed
What was the Grover (2000) study on actuarial versus clinical prediction?
human judges vs. “mechanical-prediction schemes”
actuarial prediction comparable to or better than clinical prediction: 128
clinical prediction better than mechanical: 8
more information to clinicians did not alter predictive accuracy
clinicians with interview data may decrease predictive accuracy
Why is there a debate between actuarial and clinical prediction?
why the debate? –> there’s lots of evidence favoring actuarial assessment
What was the Monahan et al. (2001) study on actuarial versus clinical prediction?
“the general superiority of statistical over clinical risk assessment in the behavioral sciences has been known for almost half a century”
“that horse was already dead”
actuarial significantly better at predicting risk over the long-term (r = ~0.45)
unguided clinical assessment little better than chance for long-term risk; better for short-term (r = ~0.1)
What was the Harris, Rice, & Cormier (2002) study on actuarial versus clinical prediction?
prospective replication, forensic patients, 8 year follow up
VRAG vs. clinician (team) judgments
clinical r = 0.23 (p < 0.001)
VRAG r = 0.42 (p < 0.001)
clinical correlation was significantly lower than VRAG (p < 0.05)
Wouldn’t aging offenders pose progressively lower risk over time for physical reasons?
after all, we’re talking about prediction intervals as long as 10 years in some cases
Barbaree (2003): decrease in recidivism (by 10% of the last year, per year) in older age
Harris and Rice: accuracy of VRAG maintained when controlled for age and time at risk
What are the comparisons between all three generations of risk assessments?
actuarial assessments (2nd generation) dramatically outperform clinical judgment, at least over the long term
clinical judgment may still have some sort term utility
the addition of dynamic variable may make a modest contribution to predictive accuracy, but that’s not always the case
actuarials, while accurate, provide little specific information and are insensitive to change
What are two foci that have been added in 3rd generation instruments?
relevance: risk factors may be more or less influential based on the individual
case formulation: movement away from using RA instruments purely as means of estimating risk, and more to detail a range of possible outcomes
What are pitfalls to avoid in risk assessment?
failing to stick to the publisher’s scoring criteria: none of the research regarding an available instrument applies if standardization has been broken
inappropriate instrument selection: offender characteristics, target events
vague reporting: failing to describe instrument, not specifying reference groups, etc.
What is a risk?
ultimately reflects one’s “tendency to offend”
What is a threat?
includes consideration of: target availability, vulnerabilities, offender risk
threat = risk x target availability x vulnerabilities
What are vulnerabilities?
refers to lapses in security measures and/or capacity to respond
