9 - Prognosis and Risk

Question 1

define risk

Answer 1

• a chance or possibility of disease
• ## ie don’t have the disease starting out but trying to predict who will get the disease

Question 2

what is the purpose of a risk factor study

Answer 2

• to estimate the probability of disease
• to understand the mechanism of disease
• to identify high risk populations
• to inform lifestyle decisions
• to inform the design of other studies

Question 3

describe the risk study design

Answer 3

start with the disease free population then see if they are exposed to risk 1, 2, 3 (extent to which they are exposed to each risk) etc and see if they have an outcome or no outcome

• like a cohort study but this time comparing risk/prognostic factors instead of comparing 2 therapies
• think about it as a cohort study - need discriminative measures
• can do prospectively but also maybe sometimes you cant (ie have to wait for the outcome/disease to occur which may take a while)
• can also be done retrospectively (risk factor prognosis factors)
• all the things we were worried about for cohort studies/therapies apply here

Question 4

risk factor is synonymous with what terms?

Answer 4

predictor or independent variable

Question 5

define prognosis

Answer 5

• an advanced indication for the course of the disease, a prediction

Question 6

what is the purpose of a prognostic study?

Answer 6

• to inform patients about what the future holds
• to understand the course of the disease
• to examine possible outcomes
• to estimate the probability of each outcome
• to inform treatment decisions
• to inform the design of other studies

Question 7

describe a prognosis study

Answer 7

• they already have the disease, now want to know what will happen to them
• just like the risk factor study but now we have an inception cohort (which is a population at a uniform and early stage in the disease) then look at prognostic factors and who ends up w what outcomes
• check out pic on slide 5 and example on slide 8

Question 8

how do prognostic factors/therapy studies relate

Answer 8

• prognostic studies and risk factor studies will inform our therapy studies and vice versa
• ie taking a certain therapy can affect your risk factor or prognosis (for example taking a baby aspirin is a therapy which can reduce the risk of heart attack - ie therapy and a prognostic factor)

Question 9

what is specificity and sensitivity again?

Answer 9

• couldnt find in notes, but according to wiki
• sensitivity = amount of true positives
• specificity = amount of true negatives

Question 10

how do you determine if a risk/prognosis study is internally valid? (4)

Answer 10

• was an inception cohort assembled?
• was the sample representative? (ie is the model robust?)
• was follow up complete?
• were objective/unbiased outcome criteria defined?

Question 11

was an inception cohort assembled?

Answer 11

• ie are included patients in a prog study at similar points in the course of their disease?
• who was not included/why
• think about potential for over/under-estimation of true likelihood of outcome

Question 12

was the sample representative?

Answer 12

• if interested in generalizability, need to know id the sample is representative of the population
• are there systematic differences btw the study sample and the population of interest?
• was the referral pattern described?

Question 13

what is a popularity bias?

Answer 13

• for sample representativeness

- experts select or follow more interesting cases (non-experts get more routine cases)

Question 14

what is a referral filter bias?

Answer 14

• for sample representativeness
• populations at tertiary centres much different than general population (ie most severe cases have been filtered out already or treated - not rep of population)

Question 15

was follow-up complete?

Answer 15

• all members of the inception cohort should be accounted for at the end of the study and their clinical status should be known
• assess the numbers lost to follow-up and their rate of outcome - lost data is usually not random! therefore can affect outcome
• how does likelihood of outcome change if we input data using worst-case (having outcome) vs best-case scenario (not having outcome) - ie if risk factors change depending on wc/bc now little certainty associated w study
• is there likely to be a difference btw complete and incomplete patients? loss of representative sample
• larger sample and fewer missing data = the more certain you will be

Question 16

were objective/unbiased outcome criteria defined?

Answer 16

• measurement issues
• has the criteria for diagnosis been clearly defined (explicit/objective criteria)?
• ie for risk factor study criteria is for whether the person is disease free and for prognostic study criteria is for whether a person is at the beginning of disease
• were the outcomes assessed in a consistent manner (all patients, same diagnostic test, same interval, same frequency, all assessed at study end - there is more control over this for a prospective study than retrospective)?
• is the outcome assessor aware of concomitant prognostic factors (other features of the patient)? - again blinding of patient and practitioner is important (person could recall differently if they know what the study is about)

Question 17

what is diagnostic suspicion bias?

Answer 17

• for validity of outcome criteria

- assessments more frequently or carefully bc of knowledge of other features

Question 18

what is expectation bias?

Answer 18

• for validity of outcome criteria

- interpretation of the diagnostic test is influenced by knowledge of other features

Question 19

what is our goal when trying to predict for prognostic testing? what is the weight?

Answer 19

• to define the magnitude of the contribution of each predictor (weight or Beta) so that our model fits the population, not just the sample
• something we measure that isn’t predictive might have a weight of 0, whereas those that are predictive have a larger weight
• oppositely predictive has a negative value

Question 20

what is the formula for outcome?

Answer 20

outcome = weight1 x predictor + weight2 x predictor + … + error

Question 21

what is overfitting?

Answer 21

• producing a model that fits the sample but not the population
• important for predictive models, similar to random sampling error (for therapy)
• see example: slide 18

Question 22

how do you determine if the prognostic study is robust? - ie evidence that it fits the population and not just the sample?

Answer 22

• have we seen the model come up in more than 1 study?
• see if the model is data driven or hypothesis driven
• adjustment for extraneous prognostic factors (ie model should include things that are well-established as predictors)

Question 23

what is a data driven model?

Answer 23

• uses a regression approach to narrow down or identify predictors
• this is a first-step efficacy-type approach
• have a bunch of data from sample and let computer do the work (univariate, stepwise, back/forward)
• good bc computers are error-free, but can’t think about what makes sense from a clinicians perspective so we get a rep of the sample, but not the population

Question 24

what is a hypothesis driven model?

Answer 24

• predictors to be included in the model are defined apriori as a result fo clinical expertise or existing literature
• when you have more experience about what is predictive (ie from literature etc)
• this is a more pragmatic approach (ie figuring out whether it actually applies

Question 25

what is univariate testing?

Answer 25

• like a t-test or chi square
• looking at whether one prognostic factor is statistically different in those who have the outcome vs those who don’t and then one by one conducting all these statistical tests to see whether there is a statistically significant difference in the risk factors for those who do and do not have the outcome
• similar to our issue with multiple comparisons (worried about spending our degrees of freedom or wearing out our data)
• so if we keep doing these tests we will eventually find a model that fits, but not going to apply to our population

Question 26

what is stepwise backward/forward testing?

Answer 26

• basically a button you push on the computer
• stepwise will bring a risk factor in, look at how good it is, bring another one in, take something out, bring another one in, until it comes up with a model
• sometimes you aren’t even aware of how many times it has brought something in or ou
• these tests are problematic in terms of spending degrees of freedom
• similar to our issue with multiple comparisons (worried about spending our degrees of freedom or wearing out our data)
• so if we keep doing these tests we will eventually find a model that fits, but not going to apply to our population

Question 27

what is a test and training sample?

Answer 27

• only a sample of data is used to construct the model and the rest is used to confirm the model
• trying to show consistency btw the 2 models

Question 28

describe how to deal with power when determining the robustness of a prognostic study - dichotomous vs continuous

Answer 28

• the more of the population we sample the more likely we are to have a population that is representative and the less likely you are to have a random sampling error
• for dichotomous need at least 10 events for every factor (the lower the event rate the larger the study you need)
• for continuous need 10-15 participants per factor (continuous outcomes will be smaller studies bc all of the data is going to contribute to that prediction)

Question 29

describe whether one should dichotomize continuous predictors

Answer 29

• should avoid doing so bc it spends our degrees of freedom!
• if we are trying to predict outcome which is on a continuous scale (say 0-100) and we have a risk factor also on a scale (from 0-100) there will be some error for both of these but if one is predictive of the other, we will be able to place it on the other one
• if we start dichotomizing based on the error, they might not fit as well or have to fit much better in order to see he relationship, so it doesn’t help us out to dichotomize our predictors (might help us in the clinic though) - can be more accurate in saying which level or how much is more predictive

Question 30

describe whether one should combine predictors

Answer 30

• if two things are predictive of each other, it makes sense to combine them (instead of getting more predictors) given that the number of predictors will increase or sample size –always aim for less predictors

Question 31

explain pre-weighted/well-known predictors

Answer 31

• If we have predictors that are already well known, we would probably pre-assign that weight instead of having computer assign the weight for us, so not part of that putting it in and taking it out computer thing

Question 32

explain adjusted R2 (shrinkage)

Answer 32

• Adjusted R2: takes into consideration some of the possibility around the random sampling error
• so if we make a pie chart of all the variability associated with the outcome, R2 is telling us how well this model predicts variability
• R2 of 1 perfectly predicts variability of the outcome (ie all variability in our outcome is perfectly explained by our model)
• as you get closer to 0, less of our pie is being explained by our model, which is not good
• so our adjusted R2 will possibly reduce/be a smaller number than the unadjusted R2 because the adjusted R2 has taken into account some of that risk of random sampling error

Question 33

look at example of “predictor” rule of thumb

Answer 33

• slide 24
• 10 events/predictor is fine (when we get to 10, R2 varies by little even with increased predictors), but more is better
• note that it is broader (more possible R2 values) with fewer predictors

Question 34

for prognostic studies, how do you determine how precise the estimates are?

Answer 34

• look at the 95% CI around the weights and the R2 (look at slide 19 and 24)
• want to see narrow CIs which will happen with sufficient power
• does the range within the CI provide certainty about what to expect?