PRESENTING UNCERTAINTY AROUND COST-
EFFECTIVENESS ESTIMATES TO DECISION MAKERS: HOW
SURE ARE WE?Mohan Mohan BalaBala, PhD., PhD.
CentocorCentocor research & development, Incresearch & development, Inc..Josephine Josephine MauskopfMauskopf
RTI Health SolutionsRTI Health Solutions
Gary Gary ZarkinZarkinRTI InternationalRTI International
Overview of workshop
Present a brief review of the different Present a brief review of the different methods used to incorporate uncertainty methods used to incorporate uncertainty into economic analysis into economic analysis Examine issues with current approachesExamine issues with current approachesExplore alternate approaches to present Explore alternate approaches to present uncertainty to decision makersuncertainty to decision makers
Economic Analysis in Health Care
CostCost--effectiveness analysis often used by decision effectiveness analysis often used by decision makers to understand whether an intervention makers to understand whether an intervention provides good value for moneyprovides good value for moneyIncremental CostIncremental Cost--effectiveness Ratio (ICER) is effectiveness Ratio (ICER) is the metric commonly used to make decisions the metric commonly used to make decisions regarding costregarding cost--effectivenesseffectivenessNet Health Benefit (NHB) and Net benefit (NB) Net Health Benefit (NHB) and Net benefit (NB) are alternate metrics that have been proposedare alternate metrics that have been proposed
Definitions
ICERICER
NHBNHB
NB
C C –– CostCostQ Q –– QALYSQALYSV V –– Societal Societal
WTP WTP per per QALYQALY
∆∆CCICER =ICER =
NHB = NHB = ∆∆Q Q ––∆∆CC
VV
NB NB = V X NB = V X ∆∆Q Q –– ∆∆CC
CEA Decision Criterion
We will pay at most $V (often $50,000) for We will pay at most $V (often $50,000) for each additional QALYeach additional QALYSo we will choose A compared to B (A So we will choose A compared to B (A being more costly and more effective) ifbeing more costly and more effective) if
ICERICERABAB < V< VNHBNHBAB AB > 0> 0NBNBABAB > 0> 0
Estimating ICEREffectiveness (efficacy) estimates obtained Effectiveness (efficacy) estimates obtained from clinical trialsfrom clinical trials
These data commonly used as input to These data commonly used as input to models that provide QALY estimates models that provide QALY estimates May require extrapolation beyond trial May require extrapolation beyond trial periodperiod
Costs obtained from either clinical trials or Costs obtained from either clinical trials or secondary data sourcessecondary data sources
LongLong--term costs and cost offsets are term costs and cost offsets are modeledmodeled
Uncertainty around the cost and effectiveness Uncertainty around the cost and effectiveness estimates result in uncertainty around the ICERestimates result in uncertainty around the ICER
Sources of Uncertainty
Uncertainty around model inputs Uncertainty around model inputs (Parameter Uncertainty)(Parameter Uncertainty)
Utility associated with health statesUtility associated with health statesCost associated with health statesCost associated with health statesTransition probabilitiesTransition probabilities
Uncertainty around model structure (Model Uncertainty around model structure (Model Uncertainty)Uncertainty)
Not commonly accounted forNot commonly accounted for
Options for Incorporating Uncertainty
(Deterministic) Sensitivity Analysis(Deterministic) Sensitivity AnalysisVary 1 or 2 inputs at a timeVary 1 or 2 inputs at a timeScenario analysisScenario analysis
Confidence IntervalsConfidence IntervalsCostCost--effectiveness Acceptability Curveseffectiveness Acceptability CurvesStochastic league tablesStochastic league tables
Sensitivity Analysis - Tornado Diagram
Base Case
Efficacy (± 10%)
Drug Cost (± 10%)
Utility (± 10%)
Discount rate (0% - 10%)
ICER
Issues with One/Two-Way Sensitivity Analysis
No standards to determine which No standards to determine which parameter should be varied and how muchparameter should be varied and how muchSensitivity analysis involving more than Sensitivity analysis involving more than two parameters difficult to presenttwo parameters difficult to presentStill, these analyses are easy for Decision Still, these analyses are easy for Decision Makers to comprehend and can be Makers to comprehend and can be valuable in some casesvaluable in some cases
Scenario Analysis
Best or Worst case scenariosBest or Worst case scenariosAlternate realistic scenariosAlternate realistic scenarios
Accounts for correlations between input Accounts for correlations between input parameter valuesparameter valuesMay be easier for decision makers to May be easier for decision makers to understandunderstand
Confidence Interval
A 95% confidence interval for a parameter, based A 95% confidence interval for a parameter, based on sample data, has the property that 95% of the on sample data, has the property that 95% of the intervals will contain the true dataintervals will contain the true data
The interval has 95% probability of containing The interval has 95% probability of containing the true value of the parameterthe true value of the parameter
A number of different approaches have been A number of different approaches have been described to estimate the confidence interval described to estimate the confidence interval around the ICERaround the ICER
Many methods only described for the two Many methods only described for the two product caseproduct case
Confidence IntervalExpected Value
∆C
∆Q
Estimating the confidence interval around ICER
Utilizes the distribution of Utilizes the distribution of ∆∆C and C and ∆∆QQMethodsMethods
The box methodThe box methodTaylorTaylor’’s seriess seriesFiellerFieller’’s methods methodNonNon--parametric bootstrapparametric bootstrapParametric bootstrapParametric bootstrapJackknifeJackknife
A Comparison of the first four methods found A Comparison of the first four methods found FeillerFeiller’’ss method and nonmethod and non--parametric bootstrap to parametric bootstrap to perform the best perform the best [Polsky D. et al. Health Econ. 6:243]
Choice of method to estimate CI
Are the required assumptions satisfied?Are the required assumptions satisfied?Is the required data available?Is the required data available?
Cost-Effectiveness Acceptability Curves (CEAC)
Estimate probability that a given intervention will Estimate probability that a given intervention will meet the CEA decision criterion, and hence will meet the CEA decision criterion, and hence will be the optimal interventionbe the optimal interventionCEACsCEACs commonly derived using probabilistic commonly derived using probabilistic sensitivity analysis (PSA)sensitivity analysis (PSA)
Specify probability distributions for all Specify probability distributions for all uncertain input parametersuncertain input parametersPropagate uncertainty through the model using Propagate uncertainty through the model using simulationsimulationDerive probability that each intervention is Derive probability that each intervention is optimaloptimal
CEAC
Drug AProb(ICER>WTP)
WTP per QALY
CEAC – Example for two products
Drug A
Drug B
Prob
WTP per QALY
CEAC – Example for multiple products
0%10%20%30%40%50%60%70%80%90%
100%
$0 $20,000 $40,000 $60,000 $80,000 $100,000
WTP per QALY
Pro
babi
lity Drug ADrug B
Drug C
CEAC – Recognized Issues
Does not contain a decision ruleDoes not contain a decision ruleThe product with the highest probability of The product with the highest probability of being optimal may not be the one with the being optimal may not be the one with the largest expected net benefitlargest expected net benefit
Modification proposed to examine the Modification proposed to examine the probability that the product with the probability that the product with the largest expected net benefit is the optimal largest expected net benefit is the optimal productproduct
Claxton et al. Health Economics 14:339
Fenwick et al. Health Economics 10:779
Stochastic League Tables
Presents the probability that a specific Presents the probability that a specific intervention would be included in the intervention would be included in the optimal mix of interventions for various optimal mix of interventions for various levels of resource availabilitylevels of resource availability11
Analysis presented for multiple sets of Analysis presented for multiple sets of mutually exclusive alternativesmutually exclusive alternatives
1Hutubessy et al. Health Economics 10:473
Application of Uncertainty Analysis in Real World Decision
Making
Model
Drug Efficacy
Drug Cost
Other Model Parameters
ICER
Model
Drug Efficacy
Drug Cost
Other Model Parameters
ICER
Parameter Uncertainty
CI / Acceptance Curve
Communicating Results to Decision Makers
A lack of understanding of the calculations A lack of understanding of the calculations of the ICER can lead toof the ICER can lead to
Angry rejectionAngry rejectionUnquestioned acceptanceUnquestioned acceptance
It is critical to communicate the model It is critical to communicate the model structure, key inputs and assumptions firststructure, key inputs and assumptions firstParameter uncertainty and model Parameter uncertainty and model uncertainty can then be addresseduncertainty can then be addressed
Influence Diagram Corresponding to Model
Response to Drug
Disease progression rate from moderate to Severe
Time spent in Moderate and Severe disease states
Total QALYs
Total Cost
Need to communicate
What are the key inputs that drive the model What are the key inputs that drive the model results?results?What are the data sources for these key inputs?What are the data sources for these key inputs?
What is the uncertainty around these input What is the uncertainty around these input parameters?parameters?
How do these key inputs drive the final results?How do these key inputs drive the final results?E.g. How does the time spent in each health E.g. How does the time spent in each health state change with efficacy assumptionsstate change with efficacy assumptions
Tornado Diagram
Base Case
Efficacy (± 10%)
Drug Cost (± 10%)
Utility (± 10%)
Discount rate (0% - 10%)
ICER
Present Best vs Worst vs most-likely case scenarios
Based on the range of values for the key Based on the range of values for the key input parameters present, Best, Worst, and input parameters present, Best, Worst, and Most likely scenario analysisMost likely scenario analysis
Could be included in the tornado diagramCould be included in the tornado diagramNeed to provide justification for the choice Need to provide justification for the choice of the parameter values for these scenariosof the parameter values for these scenarios
Questions that the Decision Maker may ask
Which treatment should I chooseWhich treatment should I chooseCould use the model results for the most likely Could use the model results for the most likely scenario analysisscenario analysis
If I choose a particular treatmentIf I choose a particular treatmentWhat is the probability that I chose a treatment What is the probability that I chose a treatment that is not optimalthat is not optimal
Provided by the CEACProvided by the CEACWhat is the magnitude of the loss in welfare in What is the magnitude of the loss in welfare in scenarios where my choice is not the optimal scenarios where my choice is not the optimal one (Downside risk)one (Downside risk)
Issues with the CEAC
CEAC as currently used provides only the CEAC as currently used provides only the probability that a given product is the optimal probability that a given product is the optimal choice for a given value of Vchoice for a given value of VCEAC does not provide information for cases CEAC does not provide information for cases where it is not the optimal product what the loss of where it is not the optimal product what the loss of net benefit is compared to the optimal productnet benefit is compared to the optimal product
This information may be important to the This information may be important to the decision makerdecision maker
Example
Drug ADrug A Drug BDrug B
∆∆CostCost11 ∆∆QALYQALY11 ∆∆CostCost11 ∆∆QALYQALY11
ExpectedExpected $12,000$12,000 1.21.2 $15,000$15,000 11
11Compared to current standard of careCompared to current standard of care
• There is a 80% probability that drug A is optimal
• Which Drug should the decision maker choose
Example - ContinuedDrug ADrug A Drug BDrug B
∆∆CostCost11 ∆∆QALYQALY11 ∆∆CostCost11 ∆∆QALYQALY11
Scenario Scenario 1 (20%)1 (20%)
$20,000$20,000 --22 $15,000$15,000 11
Scenario Scenario 2 (80%)2 (80%)
$10,000$10,000 22 $15,000$15,000 11
ExpectedExpected $12,000$12,000 1.21.2 $15,000$15,000 11
11Compared to current standard of careCompared to current standard of care
Should the decision maker be provided this information
Proposed modification to CEAC
Plot the cumulative distribution of the difference Plot the cumulative distribution of the difference in NB between a given intervention X and the next in NB between a given intervention X and the next best interventionbest intervention
In cases where X is not the optimal intervention In cases where X is not the optimal intervention this will be the difference in NB between X and this will be the difference in NB between X and the optimal intervention, and will be negativethe optimal intervention, and will be negativeIn cases where X is the optimal intervention In cases where X is the optimal intervention this difference will be positivethis difference will be positive
So, the point where the curve intersects the So, the point where the curve intersects the YY--axis provides the probability that X is the axis provides the probability that X is the optimal intervention (the point on the optimal intervention (the point on the CEAC)CEAC)
Modified CEAC
P
Drug A
p1
$1,000
p2
-$1,000 0
1
∆NB vs next best option
Modified CEAC
P
Drug A
0
1
Drug B
∆NB vs next best option
How should the information be used to make a formulary decision
Should the model results be used to always Should the model results be used to always identify a single optimal intervention?identify a single optimal intervention?
If so, should this be based solely on expected If so, should this be based solely on expected valuevalue
Alternatively, should it be used to identify a set of Alternatively, should it be used to identify a set of interventions in cases where interventions cannot interventions in cases where interventions cannot be clearly differentiated from a costbe clearly differentiated from a cost--effectiveness effectiveness perspectiveperspective
Do these models capture all relevant Do these models capture all relevant information?information?How good is the benefit measure?How good is the benefit measure?
Use of CI-s in decision making: CE panel report
Test hypothesis regarding the sign and Test hypothesis regarding the sign and magnitude of the ICERmagnitude of the ICERAllow the decision maker to determine how Allow the decision maker to determine how much confidence they should place in the much confidence they should place in the ICERICERGuide decision about future researchGuide decision about future research
An alternative to CEAC?
A
NHB B
VChoose A
Need Better Information
Choose B
Conclusions
It is widely accepted that uncertainty around costIt is widely accepted that uncertainty around cost--effectiveness estimates should presented to effectiveness estimates should presented to decision makersdecision makersHowever, whether decision makers can understand However, whether decision makers can understand and incorporate this information into decision and incorporate this information into decision making is not well understoodmaking is not well understoodFurther examination on how we can most Further examination on how we can most effectively present this information is neededeffectively present this information is needed