Critical Appraisal Flashcards
what % of a sample is within one standard deviation from the mean
68%
what % of a sample is within 2 standard deviations from the mean
95%
equivalent to 95% confidence interval
what % of a sample is within 3 standard deviations from the mean
99%
what is “critical appraisal”
process of carefully and systematically assessing the outcome of scientific research to judge its trustworthiness, value and relevance in a particular clinical context
think about:
–> how SOLID is the RATIONALE for the research question
–> how IMPORTANT is the research question
–> what is the potential IMPACT of answering it
–> can i TRUST the RESULTS of this research
what does “reliability” refer to in the context of critical appraisal
precision/replicability
what does “validity” refer to in the context of critical appraisal
is the degree to which a measurement is concordant with the “true value”
many different types of validity, but all relates to HOW a test MEASURE what it PURPORTS to actually measure
what does “responsiveness” refer to in the context of critical appraisal
SENSITIVITY of the measurement to a CHANGE in the patient’s condition
what does “interpretability” refer to in the context of critical appraisal
what is the MEANING of a given score–> i.e above 10 on a PHQ 9 means depression
what are the 4 types of validity
- face validity
- content validity
- construct validity
- criterion validity
what it “face validity”
does it seem like it makes sense
what is “content validity”
the extent to which a measurement includes ALL of the concepts of the INTENDED CONSTRUCT but NOTHING MORE
–> i.e a PHQ 9 need to include all the criteria for depression, but shouldn’t include criteria for autism
what is “construct validity”
is the measurement related COHERENTLY to other RELATED but not observable constructs
i.e if a new scale for depression was completely unrelated to a scale that measures energy and concentration levels you would be concerned about the validity of the new scale (since energy and concentration are two of the criteria for depression)
what is “criterion validity”
the extent to which the measures PREDICT readily observable phenomena
i.e is the score on the pain scale related to how much pain medication the patient requests
which concepts look at the relationship between a diagnostic test and the actual presence of the disease?
specificity
sensitivity
PPV
NPV
what is the “null hypothesis”
the default outcome of every study assumes that there is NO statistical significance between the two variables that you are looking at in your study (i.e no relationship between smoking and risk of lung cancer)
in most studies, researcher is trying to disprove the null hypothesis –> i.e trying to prove that there is in fact a connection between smoking and lung cancer
what are the two types of errors in studies related to rejecting or failing to reject the null hypothesis
type I error and type II error
what is type I error
FALSE POSITIVE
when the investigator REJECTS a null hypothesis that is actually TRUE in the population
so it would be like saying there is a connection between smoking and lung cancer, when really there isnt
(example often given is telling a man he is pregnant)
what it type II error
FALSE NEGATIVE
when an investigator FAILS TO REJECT a null hypothesis that is ACTUALLY FALSE in the population
so it would be like saying there is no connection between smoking and lung cancer, when there actually is
(example often given is telling a very obviously pregnant woman that she is not pregnant)
what are some ways to remember type I vs type II error
what are some ways to remember type I vs type II error
can the sensitivity or specificity of a test ever change?
no, they are FIXED properties of a test
what is the “sensitivity” of a test (as a concept)
the TRUE POSITIVE RATE
you want to know how many people who HAVE the disease you are able to IDENTIFY with a test
determine this by comparing the number of patients who were identified by your test as having the disease to the number of patients who ACTUALLY have the disease
a highly sensitive test RARELY MISSES people with the disease and rarely has false negatives
what is the utility of a highly sensitive test
a highly sensitive test, when negative, RULES OUT a disease
–> its good at picking up positives, so if the test is negative, you can be pretty certain you dont have the disease (SNout)
–> you want high sensitivity tests for diseases that are really bad to miss, like brain cancer
what is the “specificity” of a test (as a concept)
the TRUE NEGATIVE rate
here the focus is on patients who do NOT have the disease –> focusing on the proportion of people WITHOUT the disease who have a NEGATIVE test
a test with high specificity will NOT identify HEALTHY people as having the disease –> we expect lots of true negatives and a very small amount of false positives
when do you want a highly specific test?
when a false positive might be harmful to the patient–> i.e mammography in young women might not be highly specific enough for breast cancer and place these patients at greater risk for invasive procedures without benefit
what is the utility of a highly specific test?
when a highly specific test is POSITIVE it RULES IN having the disease (SPin)
so if you take a highly specific test and it is positive, you can be fairly certain you have the disease–> if you take a highly sensitive test and it is negative, you can be fairly certain you do not have the disease
when it comes to diagnosing a low prevalence disease, what would be the ideal combination of tests used to pickup and diagnose the disease
you would want a high sensitivity screening test that would pick up all possible cases of the disease (with some false positives) and then a highly specific confirmatory test to rule in those who actually have the disease (and discard those who dont actually have it)
how do you measure sensitivity
sensitivity = true positive / (true positive + false negative)
how many people your test said had the disease / the number of people in the sample who actually have the disease total
how do you calculate specificity
specificity = true negative / (true negative + false positive)
how many people are negative for the disease according to your test / the number of people who are actually negative for the disease in the sample regardless of test result
what is the PPV
the probability that someone who has the POSITIVE TEST result actually HAS THE DISEASE
the PPV varies directly with what other value/concept
PPV varies directly with the patients PRE TEST PROBABILITY of having the disease (i.e their baseline risk)
what is NPV
the probability that someone who has a NEGATIVE test result actually DOES NOT HAVE the disease
NPV varies INVERSELY with that other value/concept
NPV varies INVERSELY with the prevalence of the disease or with patients pretest probability of having the disease
how do you measure PPV
PPV = true positive / (true positive + false positive)
basically, the proportion of all positive tests that are true positives
how do you measure NPV
NPV = true negative / (false negative + true negative)
basically, the proportion of all negative tests that are actually negative
how do you calculate the accuracy of a test
accuracy = (TP + TN) / (TP + TN + FP + FN)
basically, the proportion of all tests that are actually accurate
what is the “prevalence” of a disease
looks at ALL EXISTING cases of an illness or disease
i.e looking at a photo of 1000 people and asking how many of these people have black hair
how do you calculate prevalence
prevalence = (TP + FN) / (TP + FN + FP + TN)
basically, all of the cases of a disease in the population
what are the 3 ways to express prevalence
- point prevalence
- period prevalence
- lifetime prevalence
what is point prevalence
the number of cases at a certain time–> i.e a survey on Dec 2020 asking if you are actively smoking
what is period prevalence
the number of cases over a certain time frame–> usually 12 months
what is lifetime prevalence
the number of cases over one’s total lifetime
i.e a survey asking you if you have ever smoked in your life
what is incidence?
incidence looks at the number of NEW cases over a PERIOD of TIME
i.e if in population of 1000 people over two years, 50 people were dx lung cancer then the incidence is 50 cases per 1000 people in that period or 25 cases per 1000-person years (INCIDENCE RATE)
are the sensitivity or specificity of a test affected by prevalence of the disease in a population
NO
but PPV and NPV are
what are 5 types of data that are often analyzed in medical literature
continuous
dichotomous
categorical
time to event
time trends
what is an example of continuous data
weight, age
what is an example of dichotomous data
a yes/no diagnosis or yes/no medications
what is an example of categorical data
i.e type of housing (apartment, condo, house)
what is an example of time to event data
i.e time until death or time until rehospitalization
what is an example of time trends data
i.e rate of hospitalizations over time or number of calls per year
what is univariate analysis
compares only TWO variables or different categories of one variable
i.e depressed vs not depressed
what is multivariate analysis
more than one variable is included in the analysis
allows for risk adjustment
what is the dependent variable in the multivariate analysis
the outcome variable
what are independent variables in multivariate analysis
predictive factors or variables that need to be accounted for i.e to avoid confounding the data
what is the “efficacy” of an intervention
it is the extent to which the intervention does more good than harm under IDEAL circumstances
“efficacy is a DELICACY”–> i.e remember that efficacy is measured under strict, fancy, expensively run, randomized clinical trial conditions–> a “delicacy” situation
what is the “effectiveness” of an intervention
the extent to which an intervention does more harm than good when provided under USUAL circumstances of healthcare practice–> “real world setting”
effectiveness studies = “pragmatic trials”