UNCERTAINTY Flashcards
Definition of Uncertainty?
Probability that they will happen and will not happen … intrinsic to economic evaluation (impossible to remove)
How does uncertainty arise?
- methodological disagreement (amongst analysts and commentators) when it comes to comparing results of studies that have employed different methods
- data requirements of the study in terms of the estimated resources and health outcomes
- the need to extrapolate (generalise) over time/region
- patients do not act the same way as consumers
Definition of Parameter Uncertainty?
Uncertainty about the true numerical values of the parameters used as inputs’ in an economic evaluation model
Definition of Methodological Uncertainty?
Model Structure uncertainty = concerns how the elements of the model are fitted together
Modelling Process uncertainty = results from decisions made by the analyst
Definition of Sensitivity Analysis?
A set of techniques that seek to analyse how sensitive results are to uncertainty … how sensitive results are to changes in a model
What is Sensitivity Analysis?
Involves systematically examining the influence of uncertainties in the variables and the assumptions employed, on the estimated results
- varying parameter estimates across a range … seeing how this impacts on the results
What are the two types of models?
- DETERMINISTIC MODEL: a model where discrete changes are made to the model … The parameters are deterministic (i.e. they have one value within an analysis)
- e.g. one-way or multi-way sensitivity analyses - STOCHASTIC MODEL: a model with a probabilistic sensitivity analysis … The parameters are stochastic (i.e. there is uncertainty about their value within an analysis)
- Based on distributions
- All of parameters in model have to have distribution attached to them … decide which distribution attached to the parameters … find suitable standard error … put them into the model
Definition of One-Way Sensitivity Analysis?
LOOKING AT HOW SENSITIVE RESULTS ARE TO CHANGES IN ONE PARAMETER … impact of changing one parameter on the results
- how robust are the results to changes in one of the variables?
How does One-Way Sensitivity Analysis work?
Choose a variable → change that variable → look at the results
Which parameters do we change in One-Way Sensitivity Analysis?
Only relevant or very uncertain parameters are usually explored … those parameters which could have an impact on the results
The alternative values can be:
- Highest to lowest if a range of estimates is available
- Confidence intervals of the parameter (if reported)
- Plausible range: e.g 20% higher / lower
- Informed by expert opinion
Advantages of One-Way Sensitivity Analysis?
May be helpful to isolate important model parameters – gives insight into factors influencing the results
Can provide a validity check to assess what happens when particular variables take extreme values
Simple to understand and undertake
- “One-way sensitivity analysis has its greatest value in developing and reviewing a model” (Drummond et al., 2015)
Disadvantages of One-Way Sensitivity Analysis?
Likely to underestimate uncertainty … only changing one parameter
Doesn’t take account correlation between the other parameters in the model
Cannot indicate what parameter contributes the most to uncertainty
Definition of Multi-Way Sensitivity Analysis?
LOOKING AT HOW SENSITIVE RESULTS ARE TO CHANGES IN MULTIPLE PARAMETERS … impact of changing multiple parameters on the results
ADDING CORRELATION BETWEEN PARAMETERS
How does Multi-Way Sensitivity Analysis work?
Can look at combinations of parameters
- changes to a combination of parameters might make results sensitive
Which parameters do we change in Multi-Way Sensitivity Analysis?
E.g. setting all parameter estimates at their highest or lowest bounds simultaneously giving the most pessimistic or optimistic scenarios
Advantages of Multi-Way Sensitivity Analysis?
Can look at combinations of parameters … more likely to have an effect on results
Disadvantages of Multi-Way Sensitivity Analysis?
Difficult to interpret correctly … require judgement about the joint probability of two parameters taking values greater than their threshold values simultaneously
What are the limitations to One-Way and Multi-Way Sensitivity Analysis?
- Selection bias … choice of parameters is discretionary
- Interpretation arbitrary … no guidelines as to what degree of variation in results is acceptable (Drummond et al, 2005)
- Impossible or time consuming to use effectively in large/complex models
Definition of Threshold Sensitivity Analysis?
Values that parameters would need to take for cost and effects to change sufficiently to alter the decision about which alternative offered the highest expected net benefits
When is Threshold Sensitivity Analysis useful?
For variables such as cost or effectiveness of a drug
- Up or down to where can take the price to keep it cost-effective
Definition of Extreme Sensitivity Analysis?
Each parameter is set at extreme (but plausible) upper and lower bounds … the difference in costs and effect over this range is recorded
How do we judge the quality of Sensitivity Analysis?
- Identify the uncertain parameters
- Specify the plausible range … review the literature, get expert opinion, use CI around the mean
- When reading literature, identify if an appropriate form of sensitivity analysis was applied
Definition of Probabilistic Sensitivity Analysis?
Changes all parameters in the model at the same time to account for correlation between model parameters
STOCHASTIC MODEL: The parameters are stochastic (i.e. there is uncertainty about their value within an analysis)
What do all the parameters in a Probabilistic Sensitivity Analysis have?
All of parameters in model have to have PROBABILITY DISTRIBUTION attached to them
How do we choose the distributions in PSA?
The distributions are based on the values of the observed data, literature review, and experts’ opinion
How does Probabilistic Sensitivity Analysis work?
MONTE CARLO SIMULATION
- Decide which distribution to attach to the parameters
- Find suitable standard error
- Randomly draw one value of each parameter at the same time from a probability distribution using Monte Carlo Simulation
- Put them into the model
- With every random draw we can estimate the model results and record them … every time you have a different ICER
- Each set of samples from all of the inputs generate a single estimate of expected costs and expected effects - Repeating this multiple times builds up an empirical distribution of ICERs
- The final ICER is then calculated based on the mean of the ICER values from all of the model iterations
- Compare the final (mean) simulation ICER with the threshold line on the distribution of cost-effectiveness
Definition of Monte Carlo Simulation?
Simulation which repeats the PSA 10,000 (ish) times to create an ICER distribution
What do we need for a Monte Carlo Simulation?
- Distribution to be assumed for each of the model’s parameters
- Samples to be taken from those distributions from which the ICER is calculated
Advantages of Probabilistic Sensitivity Analysis?
Values are drawn randomly from the probability distribution … deals with selection bias
Can incorporate uncertainty for all parameters at one time
- allows the joint uncertainty across all parameters in an economic model to be assessed at the same time
Provides a more accurate characterisation of the uncertainty inherent in any decision making process
∴ Create a distribution of cost-effectiveness result and provides some certainty in the model
Disadvantages of Probabilistic Sensitivity Analysis?
Can be computationally expensive and timely
Data can be difficult to find
Have to ensure you choose the right parameters and right standard errors
How can PSA results be represented?
- in terms of ICERs
- Scatter Plot on cost-effectiveness plane
- Cost-effectiveness acceptability curves (CEACs)
Definition of Scatter Plot on cost-effectiveness plane?
Plot each simulated estimate of the expected incremental costs and effects … each dot represents the runs on the PSA … ∴ should be over 10,000 dots
How do we interpret the PSA results on a Scatter Plot?
Shows us the general trend of the model
- Count number of dots which fall to the right of the threshold line (cost-effective)
- Can count this proportion and then get probability of intervention being cost effective
- Depends on the slope of the threshold line … that will impact the probability of the intervention being cost-effective
Definition of Cost-effectiveness acceptability curves (CEACs)?
CEACs show the probability that an intervention will be the most-likely to be cost-effective at a given cost effectiveness threshold [compared with the alternative]
- at the given threshold … there is %probability that the intervention is cost-effective and will be adopted
How do Cost-effectiveness acceptability curves work?
Each possible level of the ceiling ratio is plotted against its associated probability … gives us the CEAC
AIM: Want cost-effective ICER and high probability that intervention is cost-effective
How do Cost-effectiveness acceptability curves work with two interventions/treatments?
Point where the two CEACs cross is where they are indifferent … decision maker is indifferent between the two interventions
