Expertise (2) Flashcards
What is an expert? expertise? competence?
Def.
= professional in certain domain
–> sometimes referes to a person with expertise
Level of Expertise = individual performance
–> high expertise = “good expert”
Competence= performance across experts
(Thomas & Lawrence)
LG: Are consensus and consitency measures of expertise? (1)
Measure of competence:
• high consistency (low intra-rater reliability) and good consensus(inter-rater reliability)
—> consistency necessary for consensus
–> they can also be used as Proxy-measures of expertise!
- But, it also depends on the domain…
(Thomas & Lawrence)
LG: What makes a “good expert”?
- confidence?
- -> confidence is not a GOOD predictor for accuracy!!! but has some predictive value
1) Oskamp (1965) showed in a case study that only in the beginning did confidence have some predictive value of diagnostic accuracy –> however, the more information they got the more confidence they became, but not much more accurate (CEILING) - 32 psychologist (some clinicians, some graduates)
- got a case in sections
- after each section had make judgments about personality and give their confidence
- limitation: not super representative sample (only some clinicians) and maybe not representative case
2) Miller et al. (2015) did a meta-analysis of 36 studies which look at confidence and accuracy for psychological diagnosis and judgments
–> against what previously was expected their is a small but statistically significant correlation at least for cases where the base-rate is high or there are repeated judgments
BUT that does not mean that some increase in confidence means an increase in accuracy
AND there is still overconfidence bias is very much a thing
–> SO, still confidence is not a good proxy for accuracy
What makes a “good expert”?
- experience?
Skvortsova et al., 2016
- Aim: investigate differences in preferred reasoning styles of novice and experienced clinical psychologists as possible explanation of this surprising phenomenon.
- Method: investigated clinical and control decisions of novice (n = 20) and experienced (n = 20) clinical psychologists as well as age-matched controls (n = 20 and n =20 respectively) by using vignettes and MouselabWeb matrices
➔Reasoning style was assessed using the Rational-Experiential Inventory - Results: experiences and novice clinical psychologists did not differ in diagnostic accuracy and experiences psychologists had a higher preference for rational thinking than novices
➔in experienced psychologists a stronger preference for deliberation was associated with greater accuracy, and in novice psychologists a stronger preference for intuitive reasoning was associated with less accurate decisions
➔Maybe its not a question of more experience but of deliberation about the task that could help clinicians perform more accurately!!!
Limitations:
- there is a questionable relationship between self-report of thinking and actual strategy use
- tasks employed were quite easy
- Small sample size
What makes a “good expert”?
- practise?
Thomas, & Lawrence, (2018)
Theory of deliberate practice = extreme nurture view 10.000 hours of practice make an expert
- however this is not true –> it is a combination of nature and nurture (like always)
BESIDES:
- usually practice works because one has a feedback, but as a psychologist you do not get a feedback for diagnosis because there is no ground truth!!!
- -> so practice does not really work in this field
What makes a “good expert”?
- knowlegde?
–> in theory experiential (and theoretical) knowledge is what makes an expert
- in the parallel constrain satisfaction model: accumulated knowledge and experience leads to a greater interconnected network and a better mental representation
(- in theory it also lowers the threshold for deliberate thinking but that is not always the case –> Skortsova et al., 2016)
- just having a lot theoretical knowledge is still not a good proxy but also no hinderance for expertise (Thomas & Lawrence)
LG: Are consensus and consitency measures of expertise? (2)
- the different domains
- Why do domains differ?
- Experts in different domains vary in degree of competence
o Stockbrokersand clinical psychologists – low consistency
o Weather forecasters and auditors – high consistency - Task Characteristics are very different:
o Challenges to Validating Predictions –> human behaviour is difficult to predict, while for e.g weather outcomes are measurable with very little error
o Gold Standards = high concept validity –> substituting inadequate gold standards is easier if you have a usable outcome (weather)
o Usable Outcome Feedback = immediate, accurate, consecutive –> can be used as feedback!
=> GROUND TRUTH
o Access to Accurate decision aids
LG: When should you consider something as abnormal when diagnosing? Binder et al. - old definition - problem with the old definition - Literature review
- Depending on the criterion (liberat vs. conservative) the cut-off was 1 or 2 SDs below the mean
- -> low = abnormal - Healthy adults also sometimes score low!
- Binder et al. is a literature review, which aims to examine and reveal the base rates of large discrepancy scores between measures and the prevalence of low scores of neuropsychological test (like IQ tests)
Key psychometric principles:
- Obtaining some low scores from a battery of test is the rule, not the exception
- The more tests that are given, the more likely the person is to have a large spread between high and low scores
- People with fewer years of education and/or lower levels of intellectual abilities are expected to have more low scores compared with those with more years of education and/or higher intellectual abilities
- People with more years of education and/or higher intelligence obtain some low scores
LG: Why do experts disagree?
EVERSON: FORENSIC CHILD SEXUAL ABUSE EVALUATIONS: ASSESSING SUBJECTIVITY AND BIAS IN PROFESSIONAL JUDGMENTS
- aim
- methods
- results
- limitation
- conclusion
–> evaluators examining the same evidence often arrive at different conclusions in forensic assessments of child sexual abuse (CSA) BUT WHY?
• identifying subjective factors contributing to disagreements (interventions can then lead to greater reliability)
• 1106 professionals in field of child maltreatment (different professions)
• completed the Child Forensic Attitude Scale (CFAS) - 28-item survey assessing 3 forensic attitudes:
1. emphasis-on-sensitivity(i.e., a focus on minimizing false negatives or errors of under calling abuse)
2. emphasis-on-specificity(i.e., a focus on minimizing false positives or errors of overcalling abuse)
3. scepticism toward child and adolescent reports of CSA
• 605 professionals also participated in 1 of 3 diverse decision exercises to assess the influence of the 3 forensic attitudes on ratings of case credibility
- consistent with professional roles, victim advocates ranked highest among professional groups on the 2 emphasis-on-sensitivity subscales and lowest on the specificity and scepticisms subscales
- in contrast for child protective service workers leaned more towards scepticism and specificity (not consistent with their role!)
- women generally emphasized sensitivity, men specificity
➔substantial variation in the impact of attitude combinations
➔ attitude profiles are more predictive of case decisions than individual attitudes
- sample was not randomly selected
- the study relies on a newly developed, untested attitude scale of modest scope and design
- it is possible that this study seriously underestimates the contribution of subjective factors to evaluator disagreements
- disagreements about CSA allegations can be partly explained by individual differences in attitudes
- attitudes operate as predispositions or biases toward viewing CSA allegations as likely true or likely false
- can inform interventions and forming of strategies
What would be helpful to increase accuracy of experts?
also realte Task 6
- using the diagnostic cycle (which is leaning on the scientific method of formulating a hypothesis and testing it)
- bias training
- IMPROVE DOMAIN COMPETENCE
Thomas, & Lawrence, (2018)
> decision aids and proper models
> measuring expertise with discrimination-to-inconsistency criteria
> Bootstrapping: involves creating a model to mimic an expert’s evaluative policy, used if no performance measurement criteria is available
LG: How often do experts disagree?
1) Gowensmith et al., 2017
- looked at 240 felony cases and 720 reports (about competency to stand trial, sanity looking at mental functioning, and psychological health)
–> perfect agreement only 20% (for psychotic alone it was only 45%)
–> some illnesses are easier to find an agreement => personality disorders
–> some are harder => developmental disorders/mental retardation (low base rate) and substance abuse
–> consensus on the most pertinent diagnostic categories (e.g. schizophrenia etc.) for pretrial evaluations is reached in fewer than 1/2 of the case
=> consensus is better then chance level but it is not that high
2) Thomas, & Lawrence, (2018)
consistency of clinical psychologists: 0.44
consensus of clinical psychologist: 0.40
–> low professional competence
Parallel Constrain Herbig & Glöckner, 2008
PCS model of decision making General idea of constrain satisfaction School schedule example From automatic to deliberate Mental representations
Expertise
= assumes a primacy of automatic processes and ascribes deliberate processes (only) a supporting part for the optimization of decisions
- for every decision you have a number of variables with a number of choices (domain) and you have several constrains
- variables: different classes, different class rooms, different time slots
- constrains: no class can use the same room at the same time, the younger classes can only have school until noon etc.
- -> going through all the different options until no constrains are violated, but it happens parallel so very quickly and automatic
- if the automatic process takes too long deliberate thinking starts
- this happens because the given situation is not consistent enough (threshold) with our mental representation
Why do experts disagree?
- Gowensmith –> profession (psychologist vs. psychiatrist) , location (prison vs. clinic) = quite important, time (might be evaluated too quickly)
- Skvortsova –> differnt reasoning style applied: deliberate :) vs. intuitive :(
- Everson –> difference in attitudes, therefore criteria
