11 Assessing Validity and Reliability Flashcards
Reliability
The degree of stability exhibited when a measurement is repeated under identical conditions
If you do something over and over again, will you get the same result?
Validity
A measurement
An expression of the degree to which a measurement measures what it is trying to measure
Precision and accuracy
- Gold standard
- Silver standard
- Off-base model
- Hit or miss model
- Both
- Good accuracy, poor precision
- Good precision, poor accuracy
- Poor both
Screen versus diagnose
Screen: no preset notion of having a disease (it is done in an apparently healthy population - applying a test to a general population)
Diagnose: trying to find a disease in a particular person
Screening definition
The identification of unrecognized disease or defect through application of tests, examinations or other procedures in apparently healthy people
No concern in that person about the disease being screened for
No diagnostic intent
Very low to low disease prevalence
Diagnosis definition
Confirmation of presence or absence of disease in someone who is suspected of having it
People with symptoms of disease
Diagnostic intent
Low to high disease prevalence
In order to have a screening program, what is necessary?
A pre-clinical phase
Primordial prevention
Alter societal structures and thereby underlying determinants
Preventing the risk factors themselves
Primary prevention
Alter exposures that lead to disease
Prevent the disease from occurring
Secondary prevention
Detect and treat pathological process at an earlier stage when treatment can be more effective
Prevent symptomatic manifestation
Screening is here
Tertiary prevention
Prevent relapses and further deterioration via follow up care and rehabilitation
Attempt to minimize the adverse impact of the diseases
How do we decide if a screening program is worthwhile?
Disease must be important Early treatment must be better than late Prevalence of preclinical disease has to be high enough to justify costs Simple to administer Those being screened should be likely to comply Should provide a true measure Reproducible results Cost-effective
Sensitivity
The test correctly identifying those who have the disease
True positive rate
a/(a+c)
Specificity
The test is negative in those who do not have the disease
True negative rate
d/(b+d)
Positive predictive value
Percentage of correct results in those with a positive test result
a/(a+b)
True positives / (true positives + false positives)
Negative predictive value
Percentage of true negative results in those with a negative test result
d/(d + c)
True negatives/ (true negatives + false negatives)
Pre-test probability
Same as the prevalence, but applied to an individual
Post-test probability measurements
The probability of the disease in an individual after a given test result
Lead-time bias
Apparent increased survival duration because of screening
Screen detected cases survive longer without benefit of early treatment (no prolongation of life)
Selection bias
Volunteers or compliers are better educated and more health conscious, thus they they have a better prognosis
Length-time bias
Screening preferentially identifies slower growing or less progressive cases that have a better prognosis
At one point in time, more people are likely to have slow growing cancers compared to aggressive cancers
Overdiagnosis bias
Too many false positives
How can we properly evaluate if a screening diagnostic program or test is effective?
Randomized control trial
