How to Build a Model to Predict Major Reimbursement Decisions by Major HTA / CER Agencies

Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED

HOW TO BUILD A MODEL TO PREDICT REIMBURSEMENT DECISIONS BY MAJOR HTA/CER AGENCIES Kermit Daniel, PhD – Chief Analytics Officer, Context Matters, Inc.December 2013

2Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED

• Why build a model?

• Can we build a model?

• How do we build a model?

• How will we know if we built a good model?

BUILDING A PREDICTIVE MODEL OF HTA BEHAVIOR


A GOOD MODEL WILL

• Improve predictions

• Disentangle multiple influences

• Provide actionable guidance for how to increase the likelihood of a positive reimbursement decision

WHY BUILD A MODEL?


A MODEL IS

• Useful simplification

• Mathematical: Y = Xβ + ε

Y: Probability of a positive recommendation

X: Observable influences of recommendations

β: Parameters we will estimate

ε: Effect of influences we don’t observe

WHY BUILD A MODEL?


WHAT HAS TO BE TRUE?

• Agencies behave in ways that can be described by a simple model that we can estimateo Consistency - non-random, consistent behavior

• Through time• Across therapeutic areas• Across agencies

o Transparency – we can observe decision factors (or good proxies) Y = Xβ + ε

CAN WE BUILD A MODEL?

THIS EXPLAINS A LOT THIS EXPLAINS A LITTLE


AGENCIES FREQUENTLY AGREE WHEN PRESENTED WITH THE SAME DRUG/INDICATION TO REVIEW

CONSISTENCY ACROSS AGENCIES

FREQUENCY OF AGREEMENT BETWEEN AGENCY PAIRS

CADTH HAS NICE PBAC

SMC 68% 78% 85% 76%

PBAC 63% 72% 85%

NICE 61% 65%

HAS 58%

Note: Reviews of 94 drugs reviewed by at least two agencies between January 2005 – February 2013.


AGENCY PAIRSame Clinical

Outcome

Different Clinical

Assessment P value

Same Economic Outcome

Different Economic

Assessment P value

PBAC:SMC 0.05* 0.00*

Agree on decision 27 4 20 7

Disagree on decision 5 4 1 8

CADTH:PBAC 0.00* 0.14

Agree on decision 16 4 11 8

Disagree on decision 2 11 3 9

SMC:HAS 0.01*

Agree on decision 32 9

Disagree on decision 4 9

THE CLINICAL AND/OR ECONOMIC ASSESSMENT AND THE REIMBURSEMENT DECISION ARE USUALLY CONSISTENT

CONSISTENCY BETWEEN DECISION AND ASSESSMENTS

Note: Reviews of 94 drugs reviewed by at least two agencies between January 2005 – February 2013. Asterisk and shading indicates significance at 5% or better.


WE OBSERVE A LOT ABOUT REVIEWS, INCLUDING . . . TRANSPARENCY

DRUGS• Chemical type• Indications• Regulatory history• Orphan status

AGENCIES• Identity• Region• Agency-specific factors, e.g., additional benefit

SUBMISSIONS• Number• Professional, patient, other group support

CLINICAL STUDIES• Number, dates• Design, e.g., goal, comparators, size, subpopulations, length

• HTA assessment

CONCLUSIONS• Recommendation• Restrictions• Factors discussed, e.g., study limitations, PRO use

ECONOMIC MODELS• Manufacturer comparators & assumptions

• Agency model• HTA assessment


HOW DO WE DECIDE WHAT DATA TO INCLUDE?

• Predict outcomes that matter

• Capture implications of agency objectives and constraints

• Incorporate what we observe about agency behavior – but this is dangerous!

HOW DO WE BUILD A MODEL?


Positive Decision Not More Restrictive

Positive Decision More Restrictive

Oncology

46%5

4%

Non-Oncology

45%5

5%

PREDICT OUTCOMES THAT MATTERPOSITIVE DECISIONS OFTEN ADD RESTRICTIONS

Note: Based on 150 NICE reviews of 72 drugs for 34 diseases between January 2007 – August 2013.

NICE


PRIMARY OUTCOMES THAT ARE PROs ARE MORE LIKELY TO BE MENTIONED IN THE DECISION RATIONALE

CAPTURE IMPLICATIONS OF WHAT WE OBSERVE ABOUT AGENCY BEHAVIOR

Note: Based on a Chi-squared test, the difference between the observed frequencies and the expected frequencies were statistically significant at the .01 level. Data span 2005 – April 2013.

Neurology and Respiratory Indications


RANDOM VARIABLESTHE DANGER OF A “DATA MINING” APPROACH

Correlation with Y

PossibleDrivers

X25X29X40X42X51X68X72X94

Shading indicates correlation with Y is significant at 5%.


WE APPEAR TO HAVE A POWERFUL PREDICTIVE MODELTHE DANGER OF A “DATA MINING” APPROACH

• Very unlikely to have occurred by chance (p=1/10,000)

• Explains 2/3 of the variation in Y

• All estimates significant at better than 5%

• Two are significant at 1% or better


THE DANGER OF A “DATA MINING” APPROACH

10 15 20 25 30 35 40 45 50 550

10

20

30

40

50

60

Actual vs. Predicted

Actual

Predicted

45° (predicted = actual)

R2 = .65

WE APPEAR TO HAVE A POWERFUL PREDICTIVE MODEL

Source: Y and all Xs are independent random variables.


THE MODEL IS USELESS: IT HAS NO PREDICTIVE VALUETHE DANGER OF A “DATA MINING” APPROACH

10 15 20 25 30 35 40 45 50 550

10

20

30

40

50

60

Out-of-Sample Prediction

Actual

Predicted

Best Prediction

Source: Y and all Xs are independent random variables.


HOW DO WE JUDGE OUR MODEL?

• Predicts well

• Predicts better than alternatives

HOW WILL WE KNOW IF WE BUILT A GOOD MODEL?


POSITIVE DECISIONS BY AGENCY

Note: Based on 213 reviews of about 40 therapeutic areas between 2005 – May 2013 reviewed by NICE and at least one other major agency: SMC, PBAC, HAS, and CADTH.

POSITIVE DECISION RATES (2007-2013)

Positive Decision Rate

HAS 95%

NICE 72%

CADTH 67%

SMC 67%

PBAC 62%


HOW DO WE JUDGE OUR MODEL?

• Predicts well

• Predicts better than alternatives

• Provides actionable guidance for how to increase the likelihood of a positive reimbursement decision

HOW WILL WE KNOW IF WE BUILT A GOOD MODEL?

Health & Medicine

How to Build a Model to Predict Major Reimbursement Decisions by Major HTA / CER Agencies