INCREASING THE TRANSPARENCY INCREASING THE TRANSPARENCY OF CEA MODELING ASSUMPTIONS: OF CEA MODELING ASSUMPTIONS: A SENSITIVITY ANALYSIS BASED ON A SENSITIVITY ANALYSIS BASED ON

STRENGTH OF EVIDENCESTRENGTH OF EVIDENCE

RS Braithwaite RS Braithwaite MS Roberts MS Roberts AC JusticeAC Justice

IntroductionIntroduction

Tragicomic anecdoteTragicomic anecdote

IntroductionIntroduction Policy makers/clinicians reluctant to use Policy makers/clinicians reluctant to use

CEA because assumptions difficult to CEA because assumptions difficult to understandunderstand Using Cost-Effectiveness Analysis to Improve Using Cost-Effectiveness Analysis to Improve

Health Care: Opportunities and BarriersHealth Care: Opportunities and Barriers. . Neumann PJ 2005Neumann PJ 2005

CMS (26th National meeting of SMDM, 2004)CMS (26th National meeting of SMDM, 2004) CEA modelers may base parameter CEA modelers may base parameter

estimates on studies that have limited estimates on studies that have limited evidence.evidence.

Modelers may not consider all studies with Modelers may not consider all studies with comparable evidence and applicabilitycomparable evidence and applicability

ObjectiveObjective

To develop a method to clarify the To develop a method to clarify the tradeoff between strength of tradeoff between strength of evidence and precision of CEA evidence and precision of CEA results. results.

MethodsMethods

Proof of concept based on Proof of concept based on hypothetical data and simplified hypothetical data and simplified model of HIV natural history.model of HIV natural history.

Question: Question: What is the cost-What is the cost-effectiveness of Directly Observed effectiveness of Directly Observed Therapy (DOT) for HIV patients?Therapy (DOT) for HIV patients?

MethodsMethods Basic idea Basic idea

When data sources have insufficient When data sources have insufficient strength of evidence, we should no strength of evidence, we should no longer use them to estimate model longer use them to estimate model parameters. parameters.

Instead, we should assume that little is Instead, we should assume that little is known and specify them using wide known and specify them using wide probability distributions with the fewest probability distributions with the fewest embedded assumptions embedded assumptions Uniform distributionUniform distribution

MethodsMethods Assess strength of evidence based on USPTF Assess strength of evidence based on USPTF

guidelines which specify three valuation domains guidelines which specify three valuation domains Study designStudy design

Extent to which design differs from controlled experiment Extent to which design differs from controlled experiment Level 1 = best (RCT) Level 1 = best (RCT) Level 3=worst (expert opinion, anecdotal evidence)Level 3=worst (expert opinion, anecdotal evidence)

Internal validityInternal validity Extent to which results represent truth in study populationExtent to which results represent truth in study population Good = best (little LTFU, objective assessment)Good = best (little LTFU, objective assessment) Poor = worst (large or diverging LTFU, subjective Poor = worst (large or diverging LTFU, subjective

assessment) assessment) External validityExternal validity

Extent to which results represent truth in target populationExtent to which results represent truth in target population High = best (similar pt characteristics, care settings)High = best (similar pt characteristics, care settings) Low = worst (dissimilar pt characteristics, care settings)Low = worst (dissimilar pt characteristics, care settings)

MethodsMethods

Vary evidence criteria in 3 domains from Vary evidence criteria in 3 domains from most to least inclusivemost to least inclusive Individually and in aggregateIndividually and in aggregate

If evidence meets or exceeds criteria, use If evidence meets or exceeds criteria, use it to estimate parameter input distributionit to estimate parameter input distribution

If evidence does not meet criteria, do not If evidence does not meet criteria, do not use ituse it Use uniform distribution over plausible range Use uniform distribution over plausible range

sufficiently wide to be acceptable to all sufficiently wide to be acceptable to all CEA usersCEA users

MethodsMethods

For natural history parameters that can only be For natural history parameters that can only be observed rather than determined experimentally observed rather than determined experimentally observational studies eligible for Level 1 designobservational studies eligible for Level 1 design Overall mortality rate due to age-, sex-, and race-Overall mortality rate due to age-, sex-, and race-

related causesrelated causes When more than one source of evidence met When more than one source of evidence met

criteria, we used that source with greatest criteria, we used that source with greatest statistical precisionstatistical precision Alternative: pool weighting by inverse of varianceAlternative: pool weighting by inverse of variance

When substituting uniform distribution make When substituting uniform distribution make sure that direction of aggregate effect is neutralsure that direction of aggregate effect is neutral Maximizes conservatism of approachMaximizes conservatism of approach

MethodsMethods

Model: extremely simple 10-Model: extremely simple 10-parameter probabilistic simulation of parameter probabilistic simulation of DOT in HIVDOT in HIV

17 data sources considered 17 data sources considered

ResultsResults

Base Case: No evidence criteria Base Case: No evidence criteria All 17 data sources eligible for parameter All 17 data sources eligible for parameter

estimationestimation Study Design = High Study Design = High

13 out of 17 sources were eligible13 out of 17 sources were eligible Internal Validity = GoodInternal Validity = Good

9 out of 17 sources were eligible9 out of 17 sources were eligible External Validity = HighExternal Validity = High

5 out of 17 sources were eligible5 out of 17 sources were eligible All three criteriaAll three criteria

Only 3 out of 17 sources were eligible Only 3 out of 17 sources were eligible

Results: All EvidenceResults: All Evidence

Results: Design = 1Results: Design = 1

Results: Internal Validity Results: Internal Validity = Good= Good

Results: External Validity Results: External Validity = High= High

Results: All EvidenceResults: All Evidence

Results: Design = 1Results: Design = 1

Results: Internal Validity Results: Internal Validity = Good= Good

Results: External Validity Results: External Validity = High= High

Results – OverallResults – Overall

No evidence criteria No evidence criteria $78,000/QALY $78,000/QALY

Study Design = 1 Study Design = 1 $227,000/QALY$227,000/QALY

Internal Validity = Good Internal Validity = Good $158,000/QALY$158,000/QALY

External Validity = High External Validity = High >$6,000,000/QALY >$6,000,000/QALY

All three criteria > All three criteria > $6,000,000/QALY $6,000,000/QALY

LimitationsLimitations Incorporates a simple model of HIV that Incorporates a simple model of HIV that

was constructed solely for the purpose of was constructed solely for the purpose of illustrating proof of concept. illustrating proof of concept.

Method is likely to need further Method is likely to need further refinement before it could be used on refinement before it could be used on more complex and realistic simulations. more complex and realistic simulations.

Method only addresses parameter Method only addresses parameter uncertainty, leaving other determinates uncertainty, leaving other determinates of modeling uncertainty unexplored. of modeling uncertainty unexplored.

ConclusionsConclusions

Strength of evidence may have Strength of evidence may have profound impact on the precision and profound impact on the precision and estimates of CEAsestimates of CEAs

With all evidence was permitted results With all evidence was permitted results similar to previously published DOT similar to previously published DOT CEA (Goldie03)CEA (Goldie03) $40,000 to $75,000/QALY$40,000 to $75,000/QALY Little uncertaintyLittle uncertainty

With stricter evidence criteria our With stricter evidence criteria our results differed markedlyresults differed markedly > $ 150,000/QALY > $ 150,000/QALY Great uncertaintyGreat uncertainty

ImplicationsImplications

Sensitivity analysis by strength of Sensitivity analysis by strength of evidence concept can be linked to any evidence concept can be linked to any desired ranking method for strength of desired ranking method for strength of evidence, and therefore can be evidence, and therefore can be customized to facilitate its use by customized to facilitate its use by expert panels and organizations. expert panels and organizations. Advance of this work does not lie in its Advance of this work does not lie in its

specification of particular hierarchy of specification of particular hierarchy of strength of evidencestrength of evidence

Advance lies in showing how any hierarchy Advance lies in showing how any hierarchy can be implemented within CEA model. can be implemented within CEA model.

ImplicationsImplications

Users who think “Users who think “any data is better than any data is better than no datano data” will likely base inferences on ” will likely base inferences on model results that incorporate all data model results that incorporate all data sources, regardless of strength of evidencesources, regardless of strength of evidence

Users who think “Users who think “my judgment my judgment supersedes all but the best datasupersedes all but the best data” would ” would likely only base inferences on model results likely only base inferences on model results that reflect only highest grades of evidence. that reflect only highest grades of evidence.

Many models may fail to provide conclusive Many models may fail to provide conclusive results when validity criteria are stringent. results when validity criteria are stringent. Nonetheless, in the long run this may help CEA Nonetheless, in the long run this may help CEA

to become a more essential decision making tool.to become a more essential decision making tool.

Strength of Evidence Meets Evidence Criteria?

Variable Data Source

Design Internal Validity

External Validity

Study Design 1

Internal Validity Good

External Validity High

All 3 Cri-teria

Distribution if data meets

evidence criteria (mean, standard

error)

Distribution if data does not meet evidence criteria (lower bound, upper

bound) Mortality rate in absence of HIV

Observational: Life tables (15)

1 Good Low Yes Yes No No Point estimates, variable

Uniform (0.5X, 1.5X estimates)

Mortality rate attributable to HIV

Observational; 1 study in similar population (16)

1 Good Low Yes Yes No No Normal (0.19; 0.06)

Uniform (0, 0.38)

Impact of HIV treatment on mortality

Observational; 13 studies pooled from similar populations (17, 18)

1 Good High Yes Yes Yes Yes Normal (0.15, 0.02)

NA

Observational (19) 1 Fair Low Yes No No No Normal (0.92, 0.02)

Observational (20) 1 Good High Yes Yes Yes Yes Normal (0.75, 0.02)

Observational (21)

1 Good Low

Probability of taking HIV medications

Observational (22)

1 Good Low

Yes

Yes

No

No

Normal (0.53, 0.05)

NA

Effectiveness of DOT

Randomized controlled trial; 1 study in dissimilar population (19)

1 Good Low Yes Yes No No Normal (0.46, 0.01)

Uniform (0, 2)

Utility with HIV Observational; 1 study in similar population (20)

1 Poor Low Yes No No No Normal (0.87; 0.04)

Uniform (0.5, 1.0)

Decrement in utility with HIV treatment

Observational; 1 study in similar population (unpublished)

1 Poor Low Yes No No No Normal (0.05, 0.01)

Uniform (0, 0.5)

Expert Opinion

3 Poor Low No No No No Point estimate $4700

Observational: 1 study in dissimilar population (Burman)

2-2 Good Low No Yes No No Point estimate $4600

Observational: 1 study in dissimilar population (Moore)

2-2 Poor Low

Annual Cost of DOT

Observational: 1 study in dissimilar population (Palmer)

2-2 Poor Low

No

No

No

No

Normal ($6100; $2200)

Uniform ($200, $36,500)

Observational; 1 study in similar population (Bozette)

1 Poor High Annual Cost of non-drug HIV care

Observational; 1 study in similar population (Goldie)

1 Poor High

Yes

No

Yes

No

Normal ($9000, $300)

Uniform ($200, $20,000)

Cost of HIV drugs

Observational; 1 study in similar population (Goldie)

1 Good High Yes Yes Yes Yes Normal ($10300, $700)

NA