Part II Flashcards
What are ways of calculating value?
Expected value - of gambling: (1/80) x $1000 = $12.50 Expected utility - subjective assessment / person preferences
EXAM What is Blois’ Funnel?
The breadth of diagnostic considerations are refined, restricted over interaction between patient, physician
EXAM What are some common heuristics that people employ?
There are 8 major ones: availability representativeness ascertainment bias confirmation bias diagnosis momentum anchoring premature closure value-induced bias
What is availability bias?
Overestimating probability of unusual events because of recent / memorable instances
What is representativeness bias?
Overestimating rare diseases by matching patients to “typical picture” of the disease
What is ascertainment bias?
thinking is shaped by prior expectations
What is confirmation bias?
tendency to look for confirming evidence and not disconfirming evidence
Diagnosis momentum
Things that are initially diagnostic consideration are sticky
Anchoring
Failure to adjust probability of a disease or outcome based on new information
Premature closure
A tendency to accept a diagnosis before it is fully confirmed
Value-induced bias
Overestimate the probability of an outcome based on the value associated with that outcome
What are some ways of defending against cognitive bias?
(11) Develop insights / awareness, consider alternatives, metacognition, decrease reliance on memory, specific training, simulation, cognitive forcing, make task easy, minimize time pressure, establish accountability, feeback
What are EHR / CDS considerations for avoiding bias?
(3) Decrease reliance on memory, cognitive forcing strategies, make task easier
Describe Tree Decision Analysis
Sum of prob = 1 per scenario conditional prob: P(HIV | IV) sequential events: describe on a tree
What notation is used for a tree diagram?
Decision node - square Chance node - circle Outcome node - triangle
Describe the sample tree example
Expected Yes branch = 0.18 Expected No branch = 0.15
What is sensitivity analysis?
Do a “what if” analysis across a range of values
Cost effectiveness analysis
The value of outcome nodes become units instead of binary values
What is utility?
Perceived utility to patient: 1. Standard gamble 2. Time trade-off 3. Visual analogue
What is a QALY?
TTO = (# years perfect health) / (# years in current health)
Can be calculated using Time Trade Off (TTO) x yrs = QALY
What is “cost effectiveness”?
Operating with constrained resources; NICE uses QALY; can also calculate ICER
What is ICER?
Incremental cost/effectiveness ratio (ICER) = compare “willingness to pay” to determine if therapy cost effective
ICER = (C1-C2)/(E1-E2)
E1, E2 = effectiveness = QALY
What is a Markov Model
Chain of events each with known, fixed prob of transition in defined time period, STOCHASTIC, and 1st order is memoryless (next state not depend on prior)
Worked Markov Model example
Slides 27-29 Slides 34-36
What is a Monte Carlo Simulation?
Mathematical model can be deterministic / stochastic: 1. deterministic - variable states determined by parm. 2. probabilistic / stochastic - variable states determined by prob. distrib.
Describe the 2x2 table?
Rows = test results Columns = disease state
Calculate P(Disease), P(no disease), P(test+), P(test-)
Use 2x2 table
Describe TPR, FPR, TNR, FPR
TPR = A/A+C FNR = C/A+C TNR = D/B+D FPR = B/B+D
Describe sensitive tests
Good at ruling out disease, good for screening tests
What is PPV?
A / A+B = P(disease+ | test+) Aka Precision
What is Precision?
PPV
What is Recall?
TPR
What is specificity?
TNR
UTI worked example
Slide 58
Spam worked example
Slide 65
What is a ROC?
Receiver Operating Characteristic curve Y-axis: Sensitivity (TPR) X-axis: 1-specificity (FPR)
What is Relative Risk?
RR = P(disease | exposure) / P(disease / no exposure) Levels: weak (1.1-1.5), mod (1.5-3), strong (3-7), very strong (>7)
Relative Risk example
Slide 72
What is Bayes Theorem?
P(A|B) = (P(B|A) * P(A)) / P(B)
What is a likelihood ratio?
Positive Likelihood ratio = (LR+) = sens/1-spec = TPR/FPR (LR-) = 1-sens/spec = FNR/TNR
What is post-test odds?
pre-test odds x likelihood ratio (positive)
LR Example
Slide 75-76
What is a Fagan Nomogram?
Can determine post-test probability given pre-test probability (prevalence) and LR+
What is clinical decision support?
• Most restrictive: an electronic system that provides structured guidance based on patient-specific inputs – Expert systems – Conditional alerts • Less restrictive: any electronic tool that reduces the cognitive burden of patient care in an EHR – Order sets & corollary orders – Data visualization techniques, visual design standards • Least restrictive: “Not all decision support is electronic decision support”
What are key components to CDS?
Knowledge base
Patient-specific information
Mode of communication => CDS intervention
What are the key functions for CDSS?
Alerting Highlighting out-of-range laboratory values
Reminding Reminding the clinician to schedule a mammogram
Critiquing Rejecting an electronic order
Interpreting Interpreting the echocardiogram
Predicting Predicting risk of mortality from a severity-of-illness score
Diagnosing Listing a differential diagnosis fora patient with chest pain
Assisting Tailoring the antibiotic choices for liver transplantation and renal failure
Suggesting Generating suggestions for adjusting the mechanical ventilator
What are CDS design considerations?
Level of Control – Pre-emptive – Suppressible – Hard-stop – Interruptive
What are degrees of CDS interruptiveness?
• On demand – Link to formulary from within order • In-Line or Background (modeless) – “Unread lab result” indicator on toolbar – Optional reminder for health maintenance • Popup or Interruptive (modal) – Alerts – Reminders requiring acknowledgement
What are specific categories of CDS?
• Therapeutic duplication • Single & cumulative dose limits • Allergies & cross allergies • Contraindicated route of administration • Drug-drug and drug-food interactions • Corollary orders • Cost of care • Nuisance
EXAM - What are the 10 Commandments for Effective CDS?
- Speed is everything – expect sub-second latency
- Anticipate needs and deliver in real time – e.g. showing relevant labs with med orders
- Fit into the user’s workflow – external tools not as good as those at POC
- Little things can make a big difference – “usability matters – a lot”, “make it easy to do the right thing”
- Physicians resist stopping – don’t tell docs to not do something without offering an alternative
- Changing direction is easier than stopping
- Simple interventions work best – try to fit guidelines onto a single screen
- Ask for additional information only when you really need it – “likelihood of success is inversely proportional to the number of extra data elements needed”
- Monitor impact, get feedback, and respond
- Manage and maintain your knowledge-based systems
What are the 5 Rights of CDS?
- Right Information – quality of knowledge base
- Right Person – target of CDS
- Right Format – implementation of CDS (speed, ease of use, comprehensibility)
- Right Channel – mode of CDS
- Right Time – workflow integration
How do you evaluate CDS?
Literature - not representative, few RCTs, insufficient HCI research, etc.
What are limitations of current implementations?
– For most organizations, implementing and maintaining an EHR is hard enough
– Difficult to implement and evaluate CDS with constrained resources
What CDSS have evidence?
– Chronic disease management
– Acute care management
– Therapeutic drug monitoring and dosing
– Drug prescribing and management
– Diagnostic test ordering behavior
– Primary preventative care
What is the CDS evidence for chronic disease management?
A small majority (just over half) of CCDSSs improved care processes in chronic disease management and some improved patient health. 55 trials considered – 87% (n=48) measured impact on care process • 52% of these (n=25) showed significant improvement – 65% (n=35) measured impact on surrogate outcomes • 31% (n=11) showed benefits
What is the CDS evidence for acute care management?
The majority of CCDSSs demonstrated improvements in process of care, but patient outcomes were less likely to be evaluated and far less likely to show positive results 35 trials considered – 63% (n=22) showed improved process of care • 64% of med dosing assistants (9 of 14) • 82% management assistants with alerts/reminders (9 of 11) • 38% (3 of 8) guidelines / algorithms • 67% (2 of 3) diagnostic assistants – 20 studies looked at patient outcomes, but only 3 showed improvement
What is the CDS evidence for Drug Monitoring & Dosing?
“[S]tudies were small and generally of modest quality, and effects on patient outcomes were uncertain, with no convincing benefit in the largest studies. At present, no firm recommendation for specific systems can be given. • 76% were standalone systems, 85% were to be used by physicians • 60% showed improved process, 21% showed improved outcome • Insulin (in all studies) and Vitamin K (in meta-analysis) showed significant improvement
What is the CDS evidence for Drug Prescribing & Management?
“CCDSSs inconsistently improved process of care measures and seldomly improved patient outcomes. Lack of clear patient benefit and lack of data on harms and costs preclude a recommendation to adopt CCDSSs for drug therapy management.” 65 studies considered – Process of care improved in 37 of 59 (64%) – Outcomes improved in 6 of 29 (21%)
What is the CDS evidence for Diagnostic Test Ordering?
“Some CCDSSs can modify practitioner test-ordering behavior…[S]tudiesshould describe in more detail potentially important factors such as system design, user interface, local context, implementation strategy, and evaluate impact on user satisfaction and workflow, costs, and unintended consequences.” 35 studies identified – quality improved after 2000 – 55% improved testing behavior (18 of 33) – 5 of 6 diagnostic testing – 5 of 8 treatment monitoring – 6 of 17 disease monitoring – 4 of 4 designed to reduce test ordering rates – Cost, user satisfaction, and workflow rarely measured or reported
What is the CDS evidence for Preventative Care?
“Evidence supports the effectiveness of CCDSSs for screening and treatment of dyslipidaemia in primary care with less consistent evidence for CCDSSs used in screening for cancer and mental healthrelated conditions, vaccinations, and other preventive care. CCDSS effects on patient outcomes, safety, costs of care, and provider satisfaction remain poorly supported.” 41 RCTs considered – Improved process of care in 63% (25 of 40)
What are the 4 predictors of improved practice?
- Provision of CDS as part of clinical workflow
- Provision of recommendations, not just assessments
- Provision of CDS at time/location of decision
- Computer based decision support
What are unintended consequences of CDS?
19.8 - More/new work for clinicians 17.6 - Workflow issues 14.8 - Never ending system demands 10.8 - Paper persistence 10.1 - Changes in communication …
What does the evidence show for medication alerts & CPOE
Clear benefits in reducing prescribing errors Less clear CPOE can prevent adverse drug events
What is the curly braces problem?
When you have a code like {get blood pressure} it’s different each time from computer system to computer system
What is Arden Syntax?
Data: systolic_blood_pressure := read last {get systolic blood pressure}; /* the value in braces is specific to your runtime environment */ systolic_pressure_threshold := 140; stdout_dest := destination {stdout}; ;;
How are guidelines modelled?
CPG recommendations
Axis I - ambiguity (syntactic, semantic, pragmatic), vagueness (passive voice, strength qual, underspec)
Axis II - deliberate, inadvertent
Axis III - condition, action, explanation
Describe what syntactic, semantic, and pragmatic issues
– Syntactic – “A or B and C” (missing parentheses?) – Semantic – “I will meet you at the bank” (which bank?) – Pragmatic (conflicting recommendations)
What are some requirements for computer interpretable guidelines?
- The guidelines must first lend themselves to computation
- The representation format must allow for clinical expressivity – Temporal dependencies – Complex rules – Strength of evidence – Imperative / optional actions
- Standard vocabularies and semantics
- Interoperable
- Portable
What are some guideline modeling frameworks?
GLIF Protege Arden Syntax GEM SEBASTIEN
Describe Knowledge Maintenance
• Reliance on EHR patient data • Guideline authorship, review, update cycle • Review patterns of use – process measures, override rates, sentinel events, and other measures of CDS effectiveness • Role for service-oriented architecture for “plug-andplay” CDS systems
What is OpenCDS?
Decision support service Uses the May 2011 HL7 standard specification for a Decision Support Service • Built using open source software tools • Robust authoring environment for rules • Integration with standard terminologies (ICD10, SNOMED, LOINC, RxNORM) • Can be integrated with other types of CDS tools, such as the HL7 Infobuttons standard
What is SMART on FHIR?
2009: NEJM article “No small change for the health information economy” by Mandl & Kohane suggested that EHRs should be an extensible platform, like an iPhone™ – Liquidity of data –reduce impediment to data transfer – Substitutability of applications – modular and interoperable – Built to open standards for open-and closed-source developers – Development of an ecosystem of apps, free marketplace of ideas
What is FHIR?
2010: SMART = Substitutable Medical Applications and Reusable Technologies – 1st Gen: HTML, JavaScript, OAuth, Resource Description Framework (RDF) for metadata, and common terminologies like LOINC, RxNorm. – Lacked a standard for sharing granular clinical data – Poor initial uptake of “SMART Classic” by EHR vendors
- 2011: HL7 community concerned that HL7 V3 was not gaining traction – Led to emergence of Resources for Health –> Fast Healthcare Interoperability Resources (FHIR®)
- 2013: SMART team adopts FHIR® standard
Describe SMART on FHIR format
Can be represented as XML or JSON - Javascript Object Notation JSON is the only way to serialize a software object RESTful applications - REST = representational state transfer URI = universal resource identifier HTTP methods = PUT, GET, POST, DELETE
What are CDS hooks?
Developed by SMART project - specifies how EHR triggers can invoke external CDS services Addresses major barrier to computable, shareable decision support
What is alert fatigue?
Refers to state of user resistance to guidance provided by alerts, even those that might offer possible benefit or reduce harm, presumably because they are overwhelmed by unimportant alerts • Difficult to measure – Literature typically uses alert override rates as proxy for “low utility” – EHR systems offer different alert designs for Drug-Drug interactions, custom CDS, and other alert types. Studies may be comparing apples to oranges. • Difficult to define – What is an “appropriate” override rate? • Counterintuitive results – Reducing alert burden dramatically does not dramatically reduce override rate – In fact, EHR override rates have remained flat or perhaps increased in the past decade Overall override rates INCREASED between 2004 - 2013
What are the recommendations for alerts?
- Classify alerts in to 3 levels – minor, moderate, severe
- Develop a core set of critical drug drug interactions
- Classify alerts into active and passive, only make critical alerts active (interruptive)
- Conduct training on new improvements
- Develop systems with automated feedback/learning to identify and move alerts from active/interruptive to passive/non-interruptive
Describe Knowledge Generation (Hersh, 2009)
Original research Write up, Submit publication Peer review, publish Secondary publication, relinquish copyright, public repository…
What is knowledge acquisition (Hersh)?
Start with all literature => possibly relevant literature => definitely relevant literature => structured knowledge (divided into information retrieval, information extraction / text mining)
What are 4 basic approaches to knowledge modeling and representation?
- Clinical algorithms
- Bayesian statistics
- Production rules
- Scoring and heuristics
What is a clinical algorithm?
Follow path through “flow chart”
Elements in chart are nodes - data is gathered at information nodes (squares); decisions are made at decision nodes (diamonds)
Benefits – Knowledge is explicit – Knowledge is easy to encode
• Limitations – No accounting for prior results – Inability to pursue new etiologies, treatments, etc. – New knowledge difficult to generate • Forerunner of modern clinical practice guidelines
Describe Bayesian statistics
• Based on Bayes’ theorem, which calculates probability based on prior probability and new information • Assumptions of Bayes’ theorem – Conditional independence of findings – no relationship between different findings for a given disease – Mutual exclusivity of conditions – one finding can only explain one disease
What is Bayes’ general theorem?
Probability of disease i in the face of evidence E, out of a set of possible j diseases is: P(Di|E) = (P(Di) P(E|Di)) / (Σ P(Dj) P(E|Dj) ) • Translation of formula: probability of a disease given one or more findings can be calculated from – The prior probability of the disease – sometimes can be estimated from prevalence of disease – The probability of findings occurring in the disease
What is the implementation and limit of the Bayesian approach?
- Leeds Abdominal Pain System (de Dombal, 1975) – Most successful implementation, used in diagnosis of acute abdominal pain – Performed better than physicians – accuracy 92% vs. clinicians 65-80%, better in 6 of 7 disease categories – But difficult to use and not transportable to other locations (Berg, 1997)
- Limitations of Bayesian statistics – Findings in a disease are usually not conditionally independent – Diseases themselves may not be mutually exclusive – When multiple findings important in diagnosis, reaches high computational complexity quickly
