Validity and reliability Flashcards
Issue of validity and reliability
When diagnosing patients, its very important to consider how valid and reliable the diagnosis may be.
For example, a reliable but invalid diagnosis is dangerous bc it could lead to trust between clinicians bc they got the same diagnosis, but misdiagnosis could lead to ethical issues, ie stigmatization and ineffective treatments
An unrealiable but valid diagnosis is paradoxical, bc what isnt consistent (over time, or accross clinicians) by definition cant be valid
As there might be an inverse relationship between validity and reliability, we need to be able to find a good balance of both
This is bc u need to understand nature of disorder to interpret behavor and experience (increase validity), but this could also lead to more subjectivity and disagreement between clinicians (decreasing reliability)
Or, u might make diagnostic criteria v standardised (increase reliability), but itll become more artificial (invalid)
when is diagnosis considered reliable
If its consistent across clinicians
They should use same classification system and arrive at the same diagnosis
This is INTER-RATER RELIABILITY
How to establish interrater reliability?
Audio/ vid recording method- one clinician does interview w patient while recording, and a diff clinician uses recording to diagnose as well
GOOD: same stimulus, same quesitons, same reaction from patient
BAD: its more artificial. If second clinician did interview theyd have slightly diff answers from patient
2 clincinas do spearate interviews
GOOD: not artificial
BAD: time- pateints symptos could change over time, so there could be diff in diagnosis which is due to either genuine changes in disorder or inconsistnecies between clinicians
Order effects
If ur mind is spinning/ not consistent, answers would be inconsistent
TEST-RETEST REALIABILITY
The longer the time span between clinicians, the more likely inconsitencies are due to genuine changes of symptoms
when is diagnosis valid
The diagnosis corresponds to the actual disorder/ is accurate
PREDICTIVE VALIDITY
The ability to predict how the disorder will respond to treatment
E.g if u were given antidepressants based on diagnosis and they actually help u reduce symptoms, then it means diagnosis had high predictive validity
How to establish validity?
Heterogeneity of clinical presentation- same disorder can manifest itelf differently in diff people. Diagnostic criteria made kinda flexible to accout for this
Classification based on symptoms, not etiology- bc our knowledge of mental disorders is vague, we cant focus on causes of disorders. but lots of symptoms overlap for diff disorders
Issues w comorbidity- how do we draw boundaries between diff disorders if they occur frequently together (could they have same cause? Are they diff disorders or just diff symptoms?)
Stability of symptoms- symptoms need to be stable over time to prove validity, so as to make sure symptoms observed werent bc of some other one time thing
Cut-off point- not sure what is or isnt clinically significant
Selecting treatment- type of treatment depends on disorder, so need to identify prob correctly
Reiger
-summarised the results of field trials of the DSM-V in the USA and Canada for 23 diagnostic categories
Theory:
According to DSM-5, it is possible to identify mental illness through a structured test. However, the test is not always reliable between individual interpretations. In order to measure reliability between each psychiatrist, inter-rater reliability data is gathered. The more consistently a psychiatrist diagnoses the same disorder as another for the same patient, the more reliable the test.
Procedure
Patients were randomly assigned to two different clinicians for diagnostic interview and were blind to any previous diagnosis.
Trials examining a total of 33 separate diagnosis
All the field trials used the test-retest method, two clinicians independently interviewed the same patient on different occasion in a clinical setting.
Results
Of the 23 diagnosis, 5 were in the strong range, 9 in the moderate range and 6 in the weak range of kappa
3/23 diagnoses were in the ‘unacceptable’ range
MDD diagnosis was weak
PTSD diagnosis was very strong
Mixed anxiety-depressive disorder was unacceptable
Conclusion:
Results seemed to be mixed
While more than half of diagnostic categories ranked as strong or moderate reliability, others such as for MDD (which is widely diagnosed) were alarming
Itd be wrong to say reliability of diagnosis for these disorders was decreased, bc field trials for DSM 5 used more realistic estimates (test retest designs and random samples of patients)
For older field studies for dsm 4 and 3, results couldve been biased bc they used audio/video recording methods (this didnt reflect irl)
Usual clinical interview methods have good to very good reliability of diagnosing the representative sample
Most DSM disorders studied with their proper revisions have demonstrated effective communication between clinicians about patients diagnosis, with adequate precision
Some diagnosis that were revised to a broader spectrum of symptom expression were tested in the good to very good range.
Comorbidity - having one or more additional disorders at the same time as another
Reliability of certain disorders may be affected by comorbidity.
Chmieleweski
talks about method of how to test reliability.
Suggest an explanation for low reliability for DSM 5 compared to DSM 4 as they claim that reliability to a large extent depends on the method with which it was established. Point out that prior to DSM 5 the overwhelming majority of research studies that investigated inter-rater reliability used the audio/video recording method.
METHOD
Chmielewski et al reviewed past studies and concluded that most of the old trials to establish reliability of DSM-III were conducted using a combination of audio-/video-recording and test retest methods, most of the old trials for DSM-IV used recordings, while old trials for DSM-5 used exclusively the test-retest method.
the differences in reliability estimates might be a reflection of these different methods.
In their own study they recruited 339 patients and compared reliability estimates for DSM-IV categories using both methods.
Ppt were interviewed using the SCID. The interview was recorded. These audio tapes were later assessed independently by a second interviewer.
One week later, the participants were invited back and interviewed by a different clinician.
Results
Results confirmed the initial expectations. The mean kappa coefficient across all the disorders for the audio-recording method was 0.80, considered “almost perfect”.
The mean kappa coefficient for the test-retest method was 0.47, considered only “moderate”.
To ensure that the disagreement between clinicians over the one-week interval was not due to the true changes in the patients’ symptoms, researchers compared these results to self-reports. Patients’ self-reports of their symptoms indicated very little change over this short time period.
COnclusion
lower consistency of diagnosis in the test-retest condition is the result of the method used, not actual changes that patients underwent between the two interviews.
These findings are important as they suggest patients may not receive the same diagnosis across clinics or studies
