PE_PSA_Presenting Uncertainty Around Cost-effectivenessestimates to Decision Makers How Sure Are We

8/2/2019 PE_PSA_Presenting Uncertainty Around Cost-effectivenessestimates to Decision Makers How Sure Are We

1/40

PRESENTING UNCERTAINTY

AROUND COST-EFFECTIVENESS ESTIMATES

TO DECISION MAKERS: HOWSURE ARE WE?

MohanMohan BalaBala, PhD., PhD.

CentocorCentocor research & development, Incresearch & development, Inc..

JosephineJosephine MauskopfMauskopfRTI Health SolutionsRTI Health Solutions

GaryGary ZarkinZarkinRTI InternationalRTI International


2/40

Overview of workshop

Present a brief review of the differentPresent a brief review of the differentmethods used to incorporate uncertaintymethods used to incorporate uncertainty

into economic analysisinto economic analysis

Examine issues with current approachesExamine issues with current approaches

Explore alternate approaches to presentExplore alternate approaches to present

uncertainty to decision makersuncertainty to decision makers


3/40

Economic Analysis in Health

Care

CostCost

--effectiveness analysis often used by decisioneffectiveness analysis often used by decision

makers to understand whether an interventionmakers to understand whether an intervention

provides good value for moneyprovides good value for money

Incremental CostIncremental Cost--effectiveness Ratio (ICER) iseffectiveness Ratio (ICER) isthe metric commonly used to make decisionsthe metric commonly used to make decisions

regarding costregarding cost--effectivenesseffectiveness

Net Health Benefit (NHB) and Net benefit (NB)Net Health Benefit (NHB) and Net benefit (NB)

are alternate metrics that have been proposedare alternate metrics that have been proposed


4/40

Definitions

ICERICER

NHBNHB

NB

CC CostCostQQ QALYSQALYS

VV SocietalSocietal

WTPWTPperper

QALYQALY

CCICER =ICER = QQ

NHB =NHB = QQCCVV

NB

NB = V XNB = V X QQ CC


5/40

CEA Decision Criterion

We will pay at most $V (often $50,000) forWe will pay at most $V (often $50,000) foreach additional QALYeach additional QALY

So we will choose A compared to B (ASo we will choose A compared to B (A

being more costly and more effective) ifbeing more costly and more effective) if

ICERICERABAB < V< V

NHBNHBABAB > 0> 0

NBNBABAB > 0> 0


6/40

Estimating ICER

Effectiveness (efficacy) estimates obtainedEffectiveness (efficacy) estimates obtained

from clinical trialsfrom clinical trials These data commonly used as input toThese data commonly used as input to

models that provide QALY estimatesmodels that provide QALY estimates

May require extrapolation beyond trialMay require extrapolation beyond trialperiodperiod

Costs obtained from either clinical trials orCosts obtained from either clinical trials or

secondary data sourcessecondary data sources LongLong--term costs and cost offsets areterm costs and cost offsets are

modeledmodeled

Uncertainty around the cost and effectivenessUncertainty around the cost and effectivenessestimates result in uncertainty around the ICERestimates result in uncertainty around the ICER


7/40

Sources of Uncertainty

Uncertainty around model inputsUncertainty around model inputs(Parameter Uncertainty)(Parameter Uncertainty)

Utility associated with health statesUtility associated with health states

Cost associated with health statesCost associated with health statesTransition probabilitiesTransition probabilities

Uncertainty around model structure (ModelUncertainty around model structure (ModelUncertainty)Uncertainty)

Not commonly accounted forNot commonly accounted for


8/40

Options for Incorporating

Uncertainty

(Deterministic) Sensitivity Analysis(Deterministic) Sensitivity AnalysisVary 1 or 2 inputs at a timeVary 1 or 2 inputs at a time

Scenario analysisScenario analysis Confidence IntervalsConfidence Intervals

CostCost--effectiveness Acceptability Curveseffectiveness Acceptability Curves

Stochastic league tablesStochastic league tables


9/40

Sensitivity Analysis - Tornado

DiagramBase Case

Efficacy ( 10%)

Drug Cost ( 10%)

Utility ( 10%)

Discount rate (0% - 10%)

ICER


10/40

Issues with One/Two-Way

Sensitivity Analysis

No standards to determine whichNo standards to determine whichparameter should be varied and how muchparameter should be varied and how much

Sensitivity analysis involving more thanSensitivity analysis involving more thantwo parameters difficult to presenttwo parameters difficult to present

Still, these analyses are easy for DecisionStill, these analyses are easy for DecisionMakers to comprehend and can beMakers to comprehend and can bevaluable in some casesvaluable in some cases


11/40

Scenario Analysis

Best or Worst case scenariosBest or Worst case scenarios Alternate realistic scenariosAlternate realistic scenarios

Accounts for correlations between inputAccounts for correlations between inputparameter valuesparameter values

May be easier for decision makers toMay be easier for decision makers to

understandunderstand


12/40

Confidence Interval

A 95% confidence interval for a parameter, basedA 95% confidence interval for a parameter, basedon sample data, has the property that 95% of theon sample data, has the property that 95% of theintervals will contain the true dataintervals will contain the true data

The interval has 95% probability of containingThe interval has 95% probability of containingthe true value of the parameterthe true value of the parameter

A number of different approaches have beenA number of different approaches have been

described to estimate the confidence intervaldescribed to estimate the confidence intervalaround the ICERaround the ICER

Many methods only described for the twoMany methods only described for the two

product caseproduct case


13/40

Confidence IntervalExpected

Value

C

Q


14/40

Estimating the confidence

interval around ICER Utilizes the distribution ofUtilizes the distribution ofC andC and QQ

MethodsMethods The box methodThe box method

TaylorTaylors seriess series

FiellerFiellers methods method

NonNon--parametric bootstrapparametric bootstrap

Parametric bootstrapParametric bootstrap JackknifeJackknife

A Comparison of the first four methods foundA Comparison of the first four methods found

FeillerFeillerss method and nonmethod and non--parametric bootstrap toparametric bootstrap toperform the bestperform the best [Polsky D. et al. Health Econ. 6:243]


15/40

Choice of method to estimate CI

Are the required assumptions satisfied?Are the required assumptions satisfied? Is the required data available?Is the required data available?


16/40

Cost-Effectiveness Acceptability

Curves (CEAC) Estimate probability that a given intervention willEstimate probability that a given intervention will

meet the CEA decision criterion, and hence willmeet the CEA decision criterion, and hence willbe the optimal interventionbe the optimal intervention

CEACsCEACs commonly derived using probabilisticcommonly derived using probabilisticsensitivity analysis (PSA)sensitivity analysis (PSA)

Specify probability distributions for allSpecify probability distributions for alluncertain input parametersuncertain input parameters

Propagate uncertainty through the model usingPropagate uncertainty through the model usingsimulationsimulation

Derive probability that each intervention isDerive probability that each intervention isoptimaloptimal


17/40

CEAC

Drug AProb

(ICER>

WTP)

WTP per QALY


18/40

CEAC Example for two

products

Drug A

Drug B

Prob

WTP per QALY


19/40

CEAC Example for multiple

products

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

$0 $20,000 $40,000 $60,000 $80,000 $100,000

WTP per QALY

Probab

ility

Drug ADrug B

Drug C


20/40

CEAC Recognized Issues

Does not contain a decision ruleDoes not contain a decision rule

The product with the highest probability ofThe product with the highest probability ofbeing optimal may not be the one with thebeing optimal may not be the one with the

largest expected net benefitlargest expected net benefitModification proposed to examine theModification proposed to examine the

probability that the product with theprobability that the product with thelargest expected net benefit is the optimallargest expected net benefit is the optimalproductproduct

Claxton et al. Health Economics 14:339

Fenwick et al. Health Economics 10:779


21/40

Stochastic League Tables

Presents the probability that a specificPresents the probability that a specificintervention would be included in theintervention would be included in the

optimal mix of interventions for variousoptimal mix of interventions for various

levels of resource availabilitylevels of resource availability11

Analysis presented for multiple sets ofAnalysis presented for multiple sets of

mutually exclusive alternativesmutually exclusive alternatives

1Hutubessy et al. Health Economics 10:473


22/40

Application of UncertaintyAnalysis in Real World Decision

Making


23/40

Model

Drug

Efficacy

Drug Cost

Other Model

Parameters

ICER


24/40

Model

Drug

Efficacy

Drug Cost

Other Model

Parameters

ICER

Parameter

Uncertainty

CI /

Acceptance

Curve


25/40

Communicating Results to

Decision Makers A lack of understanding of the calculationsA lack of understanding of the calculations

of the ICER can lead toof the ICER can lead to

Angry rejectionAngry rejection

Unquestioned acceptanceUnquestioned acceptance It is critical to communicate the modelIt is critical to communicate the model

structure, key inputs and assumptions firststructure, key inputs and assumptions first Parameter uncertainty and modelParameter uncertainty and model

uncertainty can then be addresseduncertainty can then be addressed


26/40

Influence Diagram

Corresponding to Model

Response

to Drug

Disease

progression

rate from

moderate toSevere

Time

spent in

Moderate

andSevere

disease

states

Total

QALYs

Total

Cost


27/40

Need to communicate What are the key inputs that drive the modelWhat are the key inputs that drive the model

results?results?

What are the data sources for these key inputs?What are the data sources for these key inputs?

What is the uncertainty around these inputWhat is the uncertainty around these inputparameters?parameters?

How do these key inputs drive the final results?How do these key inputs drive the final results?

E.g. How does the time spent in each healthE.g. How does the time spent in each health

state change with efficacy assumptionsstate change with efficacy assumptions


28/40

Tornado DiagramBase Case

Efficacy ( 10%)

Drug Cost ( 10%)

Utility ( 10%)Discount rate (0% - 10%)

ICER


29/40

Present Best vs Worst vs most-

likely case scenarios

Based on the range of values for the keyBased on the range of values for the keyinput parameters present, Best, Worst, andinput parameters present, Best, Worst, and

Most likely scenario analysisMost likely scenario analysis

Could be included in the tornado diagramCould be included in the tornado diagram

Need to provide justification for the choiceNeed to provide justification for the choice

of the parameter values for these scenariosof the parameter values for these scenarios


30/40

Questions that the Decision

Maker may ask Which treatment should I chooseWhich treatment should I choose

Could use the model results for the most likelyCould use the model results for the most likelyscenario analysisscenario analysis

If I choose a particular treatmentIf I choose a particular treatment What is the probability that I chose a treatmentWhat is the probability that I chose a treatment

that is not optimalthat is not optimal

Provided by the CEACProvided by the CEAC What is the magnitude of the loss in welfare inWhat is the magnitude of the loss in welfare in

scenarios where my choice is not the optimalscenarios where my choice is not the optimal

one (Downside risk)one (Downside risk)


31/40

Issues with the CEAC CEAC as currently used provides only theCEAC as currently used provides only the

probability that a given product is the optimalprobability that a given product is the optimal

choice for a given value of Vchoice for a given value of V

CEAC does not provide information for casesCEAC does not provide information for caseswhere it is not the optimal product what the loss ofwhere it is not the optimal product what the loss of

net benefit is compared to the optimal productnet benefit is compared to the optimal product

This information may be important to theThis information may be important to thedecision makerdecision maker


32/40

Example

Drug ADrug A Drug BDrug B

CostCost11 QALYQALY11 CostCost11 QALYQALY11

ExpectedExpected $12,000$12,000 1.21.2 $15,000$15,000 11

11Compared to current standard of careCompared to current standard of care

There is a 80% probability that drug A is

optimal

Which Drug should the decision maker

choose

Example - Continued


33/40

Example Continued

Drug ADrug A Drug BDrug B

CostCost11 QALYQALY11 CostCost11 QALYQALY11

ScenarioScenario

1 (20%)1 (20%)$20,000$20,000 --22 $15,000$15,000 11

ScenarioScenario2 (80%)2 (80%)

$10,000$10,000 22 $15,000$15,000 11

ExpectedExpected $12,000$12,000 1.21.2 $15,000$15,000 11

11Compared to current standard of careCompared to current standard of care

Should the decision maker be provided this

information

P d difi i CEAC


34/40

Proposed modification to CEAC

Plot the cumulative distribution of the differencePlot the cumulative distribution of the differencein NB between a given intervention X and the nextin NB between a given intervention X and the next

best interventionbest intervention In cases where X is not the optimal interventionIn cases where X is not the optimal intervention

this will be the difference in NB between X andthis will be the difference in NB between X and

the optimal intervention, and will be negativethe optimal intervention, and will be negative In cases where X is the optimal interventionIn cases where X is the optimal intervention

this difference will be positivethis difference will be positive

So, the point where the curve intersects theSo, the point where the curve intersects theYY--axis provides the probability that X is theaxis provides the probability that X is theoptimal intervention (the point on theoptimal intervention (the point on the

CEAC)CEAC)

Modified CEAC


35/40

Modified CEAC

P

Drug A

p1

$1,000

p2

-$1,0000

1

NB vs next best option

Modified CEAC


36/40

Modified CEAC

P

Drug A

0

1

Drug B

NB vs next best option

H h ld th i f ti b


37/40

How should the information be

used to make a formulary decision Should the model results be used to alwaysShould the model results be used to always

identify a single optimal intervention?identify a single optimal intervention? If so, should this be based solely on expectedIf so, should this be based solely on expected

valuevalue

Alternatively, should it be used to identify a set ofAlternatively, should it be used to identify a set ofinterventions in cases where interventions cannotinterventions in cases where interventions cannotbe clearly differentiated from a costbe clearly differentiated from a cost--effectivenesseffectivenessperspectiveperspective

Do these models capture all relevantDo these models capture all relevantinformation?information?

How good is the benefit measure?How good is the benefit measure?


38/40

Use of CI-s in decision making:

CE panel report

Test hypothesis regarding the sign andTest hypothesis regarding the sign and

magnitude of the ICERmagnitude of the ICER

Allow the decision maker to determine howAllow the decision maker to determine how

much confidence they should place in themuch confidence they should place in theICERICER

Guide decision about future researchGuide decision about future research


39/40

An alternative to CEAC?

A

NHBB

VChoose A

Need Better Information

Choose B


40/40

Conclusions It is widely accepted that uncertainty around costIt is widely accepted that uncertainty around cost--

effectiveness estimates should presented toeffectiveness estimates should presented todecision makersdecision makers

However, whether decision makers can understandHowever, whether decision makers can understandand incorporate this information into decisionand incorporate this information into decision

making is not well understoodmaking is not well understood

Further examination on how we can mostFurther examination on how we can mosteffectively present this information is neededeffectively present this information is needed

Documents

PE_PSA_Presenting Uncertainty Around Cost-effectivenessestimates to Decision Makers How Sure Are We