EBP week 7 CPR’s Flashcards
Clinical prediction rule is
A model that quantifies the
contributions of a set of
variables to:
✓Diagnosis
✓Prognosis
✓Likely response to treatment
for an individual patient
A tool designed to assist clinicians to
improve decision-making based on
predictable variables from history &
physical examination in terms of:
✓making a particular diagnosis, or
✓establishing a prognosis, or
✓matching patients to optimal interventions
CPRs Relevant to Clinical Practice
Ottawa Ankle Rules (OARs) - diagnosis
✓Should a patient with acute ankle pain be evaluated with an X-ray?
* Rule for Shoulder Pain – prognosis
✓What is the risk of persistent symptoms in a patient with shoulder
pain?
* Rule for Neck Pain - response to treatment
✓Which patients with neck pain will benefit from thoracic spine thrust
manipulation?
✓Note: this is different from effect of intervention
OAR, ottawa ankle rules
Example for Dx: Ottawa Ankle Rules (OARs)
CPR for ankle and foot injuries to rule out
fracture (and reduce need for X-ray)
Clinical Prediction Rule (CPR)
* Combinations of clinical findings that are
✓systematically derived or developed
✓statistically tested
* Stepwise logistic regression is often used to develop
the most economic model
* Which combination of predictors (variables) will provide
most accurate diagnosis?
Validating CPRs: a three-step process
Step 1: Deriving the Model
Step 2: Validation of the Rule
Step 3: Impact Analysis
Some studies may include all three steps, but most will be
published separately.
step 1. derivation
identification of factors with predictive power
step 2. validation evidence of reproducible accuracy
narrow validation
application of rule in a similar clinical setting and population as in step 1
broad validation
application of rule in multiple clinical settings with varying prevalence and outcomes of disease
step 3. impact analysis
evidence that rule changes physician behavior and improves patient outcomes and/or reduces coses
CPR process levesl of evidence
4 lowest
3
2
1 highest
Validating CPRs
▪ Deriving the model
* Look for factors that potentially contribute to diagnosis
* Sn and Sp: good for ruling dx in or out?
✓ OARs: Sn = 95%; Sp = 50% (better for ruling out fx)
* Analyzed using stepwise logistic regression to use the least number
of predictors that can most accurately predict the outcome. (Refer
to concept of ROC curve)
▪ Validation of the rule
* Confirm accuracy on different samples in different settings to show
that the model is robust
▪ Impact analysis
* Has the CPR been widely adopted, and has it improved diagnosis,
and has it been shown to reduce cost or risk?
✓ OARs have significantly reduced number of X-rays
Important questions to ask pertaining to research validity of the articles
about the Derivation / rule development
- did investigators operationally define the sample in their study
- were the subjects representative of the population from which they were drawn?
- did the investigators include all relevant predictive factors in the development process?
- were the predictive factors operationally defined?
- did the investigators include a large enough sample size to accommodate the number of predictive factors used to derive the clinical predication rule?
- were the outcomes of interest operationally defined?
- were those responsible for measuring the outcomes masked to the status of the predictive factors (and vice versa)?
- did the investigators collect outcome data from all of the subjects enrolled in the study?
Assessment of Study Credibility – CPR Derivation
1. Did investigators operationally define the sample
in their study?
- Address the issue of sample homogeneity and its
potential to achieve the outcome of interest
✓Use of Inclusion/Exclusion Criteria: age, unilateral pain
Assessment of Study Credibility –
CPR Derivation
2. Were the subjects representative of the population from which they were drawn?
Address the issue of sample bias in order to ensure the accuracy and utility of the prediction rule
Assessment of Study Credibility –
CPR Derivation
3. Did investigators include all relevant predictive factors in the development process?
Address the methods for factor identification and selection
✓This involves a selective/culling process;
✓Need to perform statistical tests e.g., association with and compared
to pre-established criteria (i.e., radiographic reference) to ensure
maintaining the most predictive factors and eliminating all without.
Assessment of Study Credibility – CPR Derivation
4. Were the predictive factors operationally defined?
Address the validity of the measures used to identify and
quantify the predictive factors
✓A clear definition of these variables is necessary to correctly identify which subjects have these characteristics and to optimize the accuracy of the prediction rule.
Assessment of Study Credibility –
CPR Derivation
5. Did the investigators include a large enough sample size to accommodate the number of predictive factors used to derive the CPR?
Address the issue of statistical power and model under
specification
Assessment of Study Credibility –
CPR Derivation
6. Were the outcomes of interest operationally defined?
Address the validity of the measures used to capture the
outcomes
✓A clear definition of outcomes is necessary to avoid misidentification.
Assessment of Study Credibility –
CPR Derivation
7. Were those responsible for measuring the outcomes masked to the status of the predictive factors?
Address the issue of tester/assessor bias to improve the
accuracy of the prediction rules; masking tester/assessor
reduces an opportunity for bias
Assessment of Study Credibility –
CPR Derivation
8. Did the investigators collect outcome data from all of the subjects enrolled in the study?
Address the issue of statistical power and inaccuracy of
prognostic estimates
✓The ability to capture outcomes from all subjects is important because the attrition may lead to a skewed representation of which
outcomes occurred;
Validating CPRs
Deriving the model (Done!)
* Look for factors that potentially contribute to diagnosis
* Sn and Sp: good for ruling dx in or out?
✓ OARs: Sn = 95%; Sp = 50% (better for ruling out fx)
* Analyzed using stepwise logistic regression to use the least number
of predictors that can most accurately predict the outcome. (Refer
to concept of ROC curve)
▪ Validation of the rule
* Confirm accuracy on different samples in different settings to show
that the model is robust
▪ Impact analysis
* Has the CPR been widely adopted, and has it improved diagnosis, and has it been shown to reduce cost or risk?
✓ OARs have significantly reduced number of X-rays
Questions to assess the Validity of clinical prediction rules (Jewell 2018)
- did the investigators compare results from the clinical prediction rule to results from an established criterion test or measure of the outcomes of interest?
- were all subjects evaluated with the criterion test or measure of the outcome the rule is intended to predict.
- were the individuals interpreting the results of the clinical prediction rule unaware of the results of the criterion measure and vice versa?
- did the investigators confirm their findings with a new set of subjects?
- has the clinical prediction rule been validated on a population other than the one for which it originally was designed?
Study Results
Diagnostic categories
✓Sensitivity, specificity, predictive values,
positive/negative likelihood ratios (LR+, LR-)
* Prognostic estimates
✓Odds ratios, risk ratios, hazard ratios
* Response to treatment
✓Some combination of above
* Likelihood ratios are used for assessing the value of performing a diagnostic test.
* They use the sensitivity and specificity of the test to determine whether a test result usefully changes the probability that a condition exists.
Compare and contrast:
* Sensitivity & specificity refer to groups of people whereas likelihood
ratios (LRs) can be applied to individual patients
Should You Use this Evidence?
Is the study high quality (e.g., does the
design minimize bias)?
* Are the results important enough to use?
* Is your patient/client represented in the
study sample?
Patient/client values and preferences regarding:
✓Desire to know the future regardless of its nature
✓Belief in the value of scientific evidence
✓Previous experiences
type of evidence
level 1
at least one prospective validation in a different population AND one impact analysis demonstrating change in clinician behavior with beneficial consequences
Application?
wide use of the CPR in a variety of settings, influencing clinician behavior and improving patient outcomes
type of evidence
level 2
accuracy in ONE LARGE PROPECTIVE STUDY with a broad spectrum of patients and clinicians. OR validation in several smaller settings that differ from one another
application?
CPRS that can be used in various settings with confidence in their accuracy
type of evidence
level 3
validation in only ONE NARROW PROSPECTIVE sample
application?
CPRs that may be used with caution and only if patients in the study are similar to those in the clinicians setting
type of evidence
level 4
rules have been derived but not validated or validated only in split samples, large retrospective databases or by statistical techniques.
Application?
rules need further evaluation before they can be applied
LEVELS of evidence of CPRS
level 1
at least one prospective validation in a different population AND one impact analysis demonstrating change in clinician behavior with beneficial consequences
level 2
accuracy in ONE LARGE PROPECTIVE STUDY with a broad spectrum of patients and clinicians. OR validation in several smaller settings that differ from one another
level 3
validation in only ONE NARROW PROSPECTIVE sample
level 4
rules have been derived but not validated or validated only in split samples, large retrospective databases or by statistical techniques.
