The Value of Bayesian Approaches in the Regulatory Setting ... · Achieve optimal sample size •...
60
The Value of Bayesian Approaches in the Regulatory Setting: Lessons from the Past and Perspectives for the Future Telba Irony, Ph.D. Deputy Director, Office of Biostatistics and Epidemiology Center for Biologics Evaluation and Research FDA
The Value of Bayesian Approaches in the Regulatory Setting ... · Achieve optimal sample size • Advantageous when there is no prior information • Crucial when using prior information
(industry)Discussions by G Campbell T Irony G Pennello (government)Over 200 in attendance
Why 1998bull Computational feasibility (simulations MCMC)bull Philosophical debate shift subjectivityobjectivitybull Least burdensome provision in the regs (1997)bull Expected payoff smaller (shorter) trials
77
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Adjunct to mammography for women with BIRADS 3 or 4Approved April 16 1999Strength borrowed from two other studiesBayesian multinomial logistic hierarchical modelldquoModel predicts a substantial reduction in total number of
false-negative biopsies while increasing the net number of cancers detectedrdquowwwfdagovcdrhpdfp970033bpdf
First Bayesian Approvals
8
INTERFIX Intervertebral Body Fusion Device(for Degenerative Disk Disease )
Bayesian interim analysis
Non-informative prior
Predictive distribution to stop the trial early
Exchangeability assumption
1st Bayesian Labelwwwfdagovcdrhpdfp970015bpdf
9
bull It can increase the efficiency of clinical trials
bull It can reduce the size and duration of trial same decision reached faster
bull Sources clinical trials conducted overseas sponsorrsquos own previous studies legally available data on same or similar products data registries pilot (feasibility) studies prior information on control groups adult prior information extrapolated for pediatric population
I 2 The use of prior information
10
Controls borrowing from Historical Controls New Device borrowing from Similar Devices Multi-center trials borrowing strength across different
centers (a major issue in medical device trials due to variation among centers - physicians)
Possibilities
Bayesian Hierarchical Models Study 1Mean 1
n1
Study 2Mean 2
n2
Study 3Mean 3
n3
New StudyNew Mean
n
Borrowing as Variability among studies
New study sample size as Borrowing
Patients are not exchangeable across studiesStudies are exchangeable
11
Priors might be too informative Remediesbull discount the prior distribution in some waybull increase the stringency of the success criterionbull increase the sample size of the pivotal trialbull Bayesian hierarchical models
bull Agreement to be reached in advance between sponsor and FDA (exchangeability suitability of the prior)
bull Desire to control type I error (significance level α = 5)bull No subjective prior (expert opinion)bull Caution with advisory panel meetings could disagreebull Clinical reviewers need to agree with exchangeability assumption
Regulatory Perspective
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
2
Contributors to these ideas includeGreg Campbell (ex-CDRH)
Martin Ho (CDRH)
Megan Moncur (CBER)
Dione Price (CDER)
John Scott (CDER)
DisclaimerThis talk reflects my views and represent my own best
judgement These comments do not bind or obligate FDA
3
OutlineI Past experience Bayes for regulation of medical devices
(industry)Discussions by G Campbell T Irony G Pennello (government)Over 200 in attendance
Why 1998bull Computational feasibility (simulations MCMC)bull Philosophical debate shift subjectivityobjectivitybull Least burdensome provision in the regs (1997)bull Expected payoff smaller (shorter) trials
77
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Adjunct to mammography for women with BIRADS 3 or 4Approved April 16 1999Strength borrowed from two other studiesBayesian multinomial logistic hierarchical modelldquoModel predicts a substantial reduction in total number of
false-negative biopsies while increasing the net number of cancers detectedrdquowwwfdagovcdrhpdfp970033bpdf
First Bayesian Approvals
8
INTERFIX Intervertebral Body Fusion Device(for Degenerative Disk Disease )
Bayesian interim analysis
Non-informative prior
Predictive distribution to stop the trial early
Exchangeability assumption
1st Bayesian Labelwwwfdagovcdrhpdfp970015bpdf
9
bull It can increase the efficiency of clinical trials
bull It can reduce the size and duration of trial same decision reached faster
bull Sources clinical trials conducted overseas sponsorrsquos own previous studies legally available data on same or similar products data registries pilot (feasibility) studies prior information on control groups adult prior information extrapolated for pediatric population
I 2 The use of prior information
10
Controls borrowing from Historical Controls New Device borrowing from Similar Devices Multi-center trials borrowing strength across different
centers (a major issue in medical device trials due to variation among centers - physicians)
Possibilities
Bayesian Hierarchical Models Study 1Mean 1
n1
Study 2Mean 2
n2
Study 3Mean 3
n3
New StudyNew Mean
n
Borrowing as Variability among studies
New study sample size as Borrowing
Patients are not exchangeable across studiesStudies are exchangeable
11
Priors might be too informative Remediesbull discount the prior distribution in some waybull increase the stringency of the success criterionbull increase the sample size of the pivotal trialbull Bayesian hierarchical models
bull Agreement to be reached in advance between sponsor and FDA (exchangeability suitability of the prior)
bull Desire to control type I error (significance level α = 5)bull No subjective prior (expert opinion)bull Caution with advisory panel meetings could disagreebull Clinical reviewers need to agree with exchangeability assumption
Regulatory Perspective
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
3
OutlineI Past experience Bayes for regulation of medical devices
(industry)Discussions by G Campbell T Irony G Pennello (government)Over 200 in attendance
Why 1998bull Computational feasibility (simulations MCMC)bull Philosophical debate shift subjectivityobjectivitybull Least burdensome provision in the regs (1997)bull Expected payoff smaller (shorter) trials
77
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Adjunct to mammography for women with BIRADS 3 or 4Approved April 16 1999Strength borrowed from two other studiesBayesian multinomial logistic hierarchical modelldquoModel predicts a substantial reduction in total number of
false-negative biopsies while increasing the net number of cancers detectedrdquowwwfdagovcdrhpdfp970033bpdf
First Bayesian Approvals
8
INTERFIX Intervertebral Body Fusion Device(for Degenerative Disk Disease )
Bayesian interim analysis
Non-informative prior
Predictive distribution to stop the trial early
Exchangeability assumption
1st Bayesian Labelwwwfdagovcdrhpdfp970015bpdf
9
bull It can increase the efficiency of clinical trials
bull It can reduce the size and duration of trial same decision reached faster
bull Sources clinical trials conducted overseas sponsorrsquos own previous studies legally available data on same or similar products data registries pilot (feasibility) studies prior information on control groups adult prior information extrapolated for pediatric population
I 2 The use of prior information
10
Controls borrowing from Historical Controls New Device borrowing from Similar Devices Multi-center trials borrowing strength across different
centers (a major issue in medical device trials due to variation among centers - physicians)
Possibilities
Bayesian Hierarchical Models Study 1Mean 1
n1
Study 2Mean 2
n2
Study 3Mean 3
n3
New StudyNew Mean
n
Borrowing as Variability among studies
New study sample size as Borrowing
Patients are not exchangeable across studiesStudies are exchangeable
11
Priors might be too informative Remediesbull discount the prior distribution in some waybull increase the stringency of the success criterionbull increase the sample size of the pivotal trialbull Bayesian hierarchical models
bull Agreement to be reached in advance between sponsor and FDA (exchangeability suitability of the prior)
bull Desire to control type I error (significance level α = 5)bull No subjective prior (expert opinion)bull Caution with advisory panel meetings could disagreebull Clinical reviewers need to agree with exchangeability assumption
Regulatory Perspective
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
4
I Past ExperienceBayesian clinical trials for the regulation of medical devices
5
I1 Initial Idea - 1998Why Bayesian methods to regulate medical devices
bull Mechanism of action is often physical not pharmacokinetic localized effects rather than systemic
bull Medical device companies conduct pre-clinical animal studies and bench testing
bull There is often information available on trials overseasbull Available prior information on similar devices (previous
generations)bull If you modify a device slightly the behavior may be very
similar (but you still need to notify the FDA)bull In contrast if you modify a drug formulation slightly the safety
and efficacy may not be well understoodbull Availability of historical controls for borrowing strength
6
HIMAFDA Workshop ndash November 1998ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
Health Industry Manufacturers Association (todayrsquos AdvaMed)Presentations by Don Berry Bill Strawderman Mike Escobar
(industry)Discussions by G Campbell T Irony G Pennello (government)Over 200 in attendance
Why 1998bull Computational feasibility (simulations MCMC)bull Philosophical debate shift subjectivityobjectivitybull Least burdensome provision in the regs (1997)bull Expected payoff smaller (shorter) trials
77
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Adjunct to mammography for women with BIRADS 3 or 4Approved April 16 1999Strength borrowed from two other studiesBayesian multinomial logistic hierarchical modelldquoModel predicts a substantial reduction in total number of
false-negative biopsies while increasing the net number of cancers detectedrdquowwwfdagovcdrhpdfp970033bpdf
First Bayesian Approvals
8
INTERFIX Intervertebral Body Fusion Device(for Degenerative Disk Disease )
Bayesian interim analysis
Non-informative prior
Predictive distribution to stop the trial early
Exchangeability assumption
1st Bayesian Labelwwwfdagovcdrhpdfp970015bpdf
9
bull It can increase the efficiency of clinical trials
bull It can reduce the size and duration of trial same decision reached faster
bull Sources clinical trials conducted overseas sponsorrsquos own previous studies legally available data on same or similar products data registries pilot (feasibility) studies prior information on control groups adult prior information extrapolated for pediatric population
I 2 The use of prior information
10
Controls borrowing from Historical Controls New Device borrowing from Similar Devices Multi-center trials borrowing strength across different
centers (a major issue in medical device trials due to variation among centers - physicians)
Possibilities
Bayesian Hierarchical Models Study 1Mean 1
n1
Study 2Mean 2
n2
Study 3Mean 3
n3
New StudyNew Mean
n
Borrowing as Variability among studies
New study sample size as Borrowing
Patients are not exchangeable across studiesStudies are exchangeable
11
Priors might be too informative Remediesbull discount the prior distribution in some waybull increase the stringency of the success criterionbull increase the sample size of the pivotal trialbull Bayesian hierarchical models
bull Agreement to be reached in advance between sponsor and FDA (exchangeability suitability of the prior)
bull Desire to control type I error (significance level α = 5)bull No subjective prior (expert opinion)bull Caution with advisory panel meetings could disagreebull Clinical reviewers need to agree with exchangeability assumption
Regulatory Perspective
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
5
I1 Initial Idea - 1998Why Bayesian methods to regulate medical devices
bull Mechanism of action is often physical not pharmacokinetic localized effects rather than systemic
bull Medical device companies conduct pre-clinical animal studies and bench testing
bull There is often information available on trials overseasbull Available prior information on similar devices (previous
generations)bull If you modify a device slightly the behavior may be very
similar (but you still need to notify the FDA)bull In contrast if you modify a drug formulation slightly the safety
and efficacy may not be well understoodbull Availability of historical controls for borrowing strength
6
HIMAFDA Workshop ndash November 1998ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
Health Industry Manufacturers Association (todayrsquos AdvaMed)Presentations by Don Berry Bill Strawderman Mike Escobar
(industry)Discussions by G Campbell T Irony G Pennello (government)Over 200 in attendance
Why 1998bull Computational feasibility (simulations MCMC)bull Philosophical debate shift subjectivityobjectivitybull Least burdensome provision in the regs (1997)bull Expected payoff smaller (shorter) trials
77
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Adjunct to mammography for women with BIRADS 3 or 4Approved April 16 1999Strength borrowed from two other studiesBayesian multinomial logistic hierarchical modelldquoModel predicts a substantial reduction in total number of
false-negative biopsies while increasing the net number of cancers detectedrdquowwwfdagovcdrhpdfp970033bpdf
First Bayesian Approvals
8
INTERFIX Intervertebral Body Fusion Device(for Degenerative Disk Disease )
Bayesian interim analysis
Non-informative prior
Predictive distribution to stop the trial early
Exchangeability assumption
1st Bayesian Labelwwwfdagovcdrhpdfp970015bpdf
9
bull It can increase the efficiency of clinical trials
bull It can reduce the size and duration of trial same decision reached faster
bull Sources clinical trials conducted overseas sponsorrsquos own previous studies legally available data on same or similar products data registries pilot (feasibility) studies prior information on control groups adult prior information extrapolated for pediatric population
I 2 The use of prior information
10
Controls borrowing from Historical Controls New Device borrowing from Similar Devices Multi-center trials borrowing strength across different
centers (a major issue in medical device trials due to variation among centers - physicians)
Possibilities
Bayesian Hierarchical Models Study 1Mean 1
n1
Study 2Mean 2
n2
Study 3Mean 3
n3
New StudyNew Mean
n
Borrowing as Variability among studies
New study sample size as Borrowing
Patients are not exchangeable across studiesStudies are exchangeable
11
Priors might be too informative Remediesbull discount the prior distribution in some waybull increase the stringency of the success criterionbull increase the sample size of the pivotal trialbull Bayesian hierarchical models
bull Agreement to be reached in advance between sponsor and FDA (exchangeability suitability of the prior)
bull Desire to control type I error (significance level α = 5)bull No subjective prior (expert opinion)bull Caution with advisory panel meetings could disagreebull Clinical reviewers need to agree with exchangeability assumption
Regulatory Perspective
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
6
HIMAFDA Workshop ndash November 1998ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
Health Industry Manufacturers Association (todayrsquos AdvaMed)Presentations by Don Berry Bill Strawderman Mike Escobar
(industry)Discussions by G Campbell T Irony G Pennello (government)Over 200 in attendance
Why 1998bull Computational feasibility (simulations MCMC)bull Philosophical debate shift subjectivityobjectivitybull Least burdensome provision in the regs (1997)bull Expected payoff smaller (shorter) trials
77
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Adjunct to mammography for women with BIRADS 3 or 4Approved April 16 1999Strength borrowed from two other studiesBayesian multinomial logistic hierarchical modelldquoModel predicts a substantial reduction in total number of
false-negative biopsies while increasing the net number of cancers detectedrdquowwwfdagovcdrhpdfp970033bpdf
First Bayesian Approvals
8
INTERFIX Intervertebral Body Fusion Device(for Degenerative Disk Disease )
Bayesian interim analysis
Non-informative prior
Predictive distribution to stop the trial early
Exchangeability assumption
1st Bayesian Labelwwwfdagovcdrhpdfp970015bpdf
9
bull It can increase the efficiency of clinical trials
bull It can reduce the size and duration of trial same decision reached faster
bull Sources clinical trials conducted overseas sponsorrsquos own previous studies legally available data on same or similar products data registries pilot (feasibility) studies prior information on control groups adult prior information extrapolated for pediatric population
I 2 The use of prior information
10
Controls borrowing from Historical Controls New Device borrowing from Similar Devices Multi-center trials borrowing strength across different
centers (a major issue in medical device trials due to variation among centers - physicians)
Possibilities
Bayesian Hierarchical Models Study 1Mean 1
n1
Study 2Mean 2
n2
Study 3Mean 3
n3
New StudyNew Mean
n
Borrowing as Variability among studies
New study sample size as Borrowing
Patients are not exchangeable across studiesStudies are exchangeable
11
Priors might be too informative Remediesbull discount the prior distribution in some waybull increase the stringency of the success criterionbull increase the sample size of the pivotal trialbull Bayesian hierarchical models
bull Agreement to be reached in advance between sponsor and FDA (exchangeability suitability of the prior)
bull Desire to control type I error (significance level α = 5)bull No subjective prior (expert opinion)bull Caution with advisory panel meetings could disagreebull Clinical reviewers need to agree with exchangeability assumption
Regulatory Perspective
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
77
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Adjunct to mammography for women with BIRADS 3 or 4Approved April 16 1999Strength borrowed from two other studiesBayesian multinomial logistic hierarchical modelldquoModel predicts a substantial reduction in total number of
false-negative biopsies while increasing the net number of cancers detectedrdquowwwfdagovcdrhpdfp970033bpdf
First Bayesian Approvals
8
INTERFIX Intervertebral Body Fusion Device(for Degenerative Disk Disease )
Bayesian interim analysis
Non-informative prior
Predictive distribution to stop the trial early
Exchangeability assumption
1st Bayesian Labelwwwfdagovcdrhpdfp970015bpdf
9
bull It can increase the efficiency of clinical trials
bull It can reduce the size and duration of trial same decision reached faster
bull Sources clinical trials conducted overseas sponsorrsquos own previous studies legally available data on same or similar products data registries pilot (feasibility) studies prior information on control groups adult prior information extrapolated for pediatric population
I 2 The use of prior information
10
Controls borrowing from Historical Controls New Device borrowing from Similar Devices Multi-center trials borrowing strength across different
centers (a major issue in medical device trials due to variation among centers - physicians)
Possibilities
Bayesian Hierarchical Models Study 1Mean 1
n1
Study 2Mean 2
n2
Study 3Mean 3
n3
New StudyNew Mean
n
Borrowing as Variability among studies
New study sample size as Borrowing
Patients are not exchangeable across studiesStudies are exchangeable
11
Priors might be too informative Remediesbull discount the prior distribution in some waybull increase the stringency of the success criterionbull increase the sample size of the pivotal trialbull Bayesian hierarchical models
bull Agreement to be reached in advance between sponsor and FDA (exchangeability suitability of the prior)
bull Desire to control type I error (significance level α = 5)bull No subjective prior (expert opinion)bull Caution with advisory panel meetings could disagreebull Clinical reviewers need to agree with exchangeability assumption
Regulatory Perspective
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
8
INTERFIX Intervertebral Body Fusion Device(for Degenerative Disk Disease )
Bayesian interim analysis
Non-informative prior
Predictive distribution to stop the trial early
Exchangeability assumption
1st Bayesian Labelwwwfdagovcdrhpdfp970015bpdf
9
bull It can increase the efficiency of clinical trials
bull It can reduce the size and duration of trial same decision reached faster
bull Sources clinical trials conducted overseas sponsorrsquos own previous studies legally available data on same or similar products data registries pilot (feasibility) studies prior information on control groups adult prior information extrapolated for pediatric population
I 2 The use of prior information
10
Controls borrowing from Historical Controls New Device borrowing from Similar Devices Multi-center trials borrowing strength across different
centers (a major issue in medical device trials due to variation among centers - physicians)
Possibilities
Bayesian Hierarchical Models Study 1Mean 1
n1
Study 2Mean 2
n2
Study 3Mean 3
n3
New StudyNew Mean
n
Borrowing as Variability among studies
New study sample size as Borrowing
Patients are not exchangeable across studiesStudies are exchangeable
11
Priors might be too informative Remediesbull discount the prior distribution in some waybull increase the stringency of the success criterionbull increase the sample size of the pivotal trialbull Bayesian hierarchical models
bull Agreement to be reached in advance between sponsor and FDA (exchangeability suitability of the prior)
bull Desire to control type I error (significance level α = 5)bull No subjective prior (expert opinion)bull Caution with advisory panel meetings could disagreebull Clinical reviewers need to agree with exchangeability assumption
Regulatory Perspective
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
9
bull It can increase the efficiency of clinical trials
bull It can reduce the size and duration of trial same decision reached faster
bull Sources clinical trials conducted overseas sponsorrsquos own previous studies legally available data on same or similar products data registries pilot (feasibility) studies prior information on control groups adult prior information extrapolated for pediatric population
I 2 The use of prior information
10
Controls borrowing from Historical Controls New Device borrowing from Similar Devices Multi-center trials borrowing strength across different
centers (a major issue in medical device trials due to variation among centers - physicians)
Possibilities
Bayesian Hierarchical Models Study 1Mean 1
n1
Study 2Mean 2
n2
Study 3Mean 3
n3
New StudyNew Mean
n
Borrowing as Variability among studies
New study sample size as Borrowing
Patients are not exchangeable across studiesStudies are exchangeable
11
Priors might be too informative Remediesbull discount the prior distribution in some waybull increase the stringency of the success criterionbull increase the sample size of the pivotal trialbull Bayesian hierarchical models
bull Agreement to be reached in advance between sponsor and FDA (exchangeability suitability of the prior)
bull Desire to control type I error (significance level α = 5)bull No subjective prior (expert opinion)bull Caution with advisory panel meetings could disagreebull Clinical reviewers need to agree with exchangeability assumption
Regulatory Perspective
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
10
Controls borrowing from Historical Controls New Device borrowing from Similar Devices Multi-center trials borrowing strength across different
centers (a major issue in medical device trials due to variation among centers - physicians)
Possibilities
Bayesian Hierarchical Models Study 1Mean 1
n1
Study 2Mean 2
n2
Study 3Mean 3
n3
New StudyNew Mean
n
Borrowing as Variability among studies
New study sample size as Borrowing
Patients are not exchangeable across studiesStudies are exchangeable
11
Priors might be too informative Remediesbull discount the prior distribution in some waybull increase the stringency of the success criterionbull increase the sample size of the pivotal trialbull Bayesian hierarchical models
bull Agreement to be reached in advance between sponsor and FDA (exchangeability suitability of the prior)
bull Desire to control type I error (significance level α = 5)bull No subjective prior (expert opinion)bull Caution with advisory panel meetings could disagreebull Clinical reviewers need to agree with exchangeability assumption
Regulatory Perspective
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
11
Priors might be too informative Remediesbull discount the prior distribution in some waybull increase the stringency of the success criterionbull increase the sample size of the pivotal trialbull Bayesian hierarchical models
bull Agreement to be reached in advance between sponsor and FDA (exchangeability suitability of the prior)
bull Desire to control type I error (significance level α = 5)bull No subjective prior (expert opinion)bull Caution with advisory panel meetings could disagreebull Clinical reviewers need to agree with exchangeability assumption
Regulatory Perspective
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
12
bull Problematic to control type I error rates at traditional α values 5 or 25 Tradition needs to be relaxed (increase α) otherwise all prior
information is discounted no gains
bull Arbitrarily redefine new levels of α Discount prior Power priors with arbitrary discount parameters Arbitrary effective sample size (try elicitation from clinicians) Hierarchical models with arbitrary hyper parameters
bull Priors may have more patients than current trial arbitrarily discount prior distribution (50 )
Regulatory Perspective
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
13
bull Problematic to use hierarchical models with only one prior study (canrsquot estimate the variability among studies)
bull Clinicians often unsure about exchangeability assumption
bull Subjectivity How to choose the prior Whose prior Selection bias unfavorable prior information may have been
omitted or selected (control group) Will future regulators or advisory panel members agree with
current regulators
bull Legal prior information may not be legally available
Regulatory Perspective
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
14
bull Inherent to the Bayesian approach (Likelihood Principle)
bull Can reduce the size (length) of a trial faster decision
bull Can increase the size (length) of a trial If it happens it is neededbull Interim analyses for decisions on stopping or continuing recruiting
based on predictive distributions sample size decided and optimized during the trial ldquoGoldilocksrdquo trials
bull Modeling results at early follow-up times predict results at the final follow-up Model refined at interim looks when all follow-up results from patients recruited early are available
bull Adaptive randomization Probability of assignment to a treatment depends on data obtained thus far May be ethically appealing if allocate more patients to the best treatments
I 3 Bayesian Adaptive Designs
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
15
bull Treatment vs Control Success or Failure at 24 monthsbull Follow-up times 3 6 12 and 24 monthsbull Interim looks For sample size adaptation For effectiveness For futility
bull Constant or varying accrual ratebull Model earlier visits are used to predict 24-month results of
patients that have not yet reached the 24-month follow-upbull Assumes exchangeability among patients recruited early and later
in the trial
Example Bayesian adaptive design
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
16
Interim Looks for Sample Size Adaptation
100 complete pts Calculate PP for
futility
Accrue 300 pts
Accrue 50 more pts if Nlt800
No
Yes
Calculate PP for adaptation
Stop and claim FutilityNext Slide
Stop recruiting
Stop Accrual and Label N
Yes
No
PP = Posterior Probability
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
17
Interim looks for Effectiveness
N is now fixedFour interim looks at 6 12 and 18 months
Wait 6 months Calculate PP All pts complete
Test PP for success
No Yes
No
Stop for Success Claim Futility
NoYesYes
100 complete ptsTest success
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
18
bull Increase the probability of trial success (insurance)
bull Achieve optimal sample size
bull Advantageous when there is no prior information
bull Crucial when using prior information (hierarchical model) amount of strength to be borrowed is uncertain avoid failure for lack of power
bull Very advantageous when Bayesian modeling is used to predict an endpoint from earlier follow up visits ndash savings in sample size
Regulatory Perspective
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
19
bull Stopping early occurs when surprises arisebull Treatment is better (success) or worse (futility) than predictedbull Sample variability is smaller than predictedbull Bayesian model makes good predictions (correlation among
follow up times is high)
bull Simulations were used to assess operating characteristics of the trial design - control type I error rates and power (no mathematical formulas for Bayesian adaptive designs)
Regulatory Perspective
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
20
bull Calculate error rates for Bayesian trial designsbull Increase trial predictability and help sponsors prepare and budget
for different scenarios and surprisesbull Readily understood by clinicians who can observe what will
happen under various scenariosbull Provide ability to ldquolook into the futurerdquo to avoid ldquoanticipated
regretrdquo if the trial were to fail what would we do differently in retrospect
I 4 Simulations
ldquoWhen you do the real trial it is not the first time you are doing it it is 1000001thrdquo
Simulate the trial thousands of times making assumptions about the true value of the endpoints and look at the average performance
How often does it get the right answer
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
21
bull Simulations are conducted at the design stage
bull Devise a comprehensive number of scenarios to generate data (need clinicians input)
bull Make assumptions to generate data
bull Could assess and control error rates (type I and type II) under ldquoall plausible scenarios and assumptionsrdquo
bull It may be more difficult for the FDA to review
bull It may take more effort to reach agreement with the FDA at the design stage
bull Sponsorrsquos documentation including the simulation code is useful to facilitate the review
Regulatory Perspective
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
22
Simulations are essential to strategize trial design
bull Independently of whether the design is Bayesian or not
bull Choose design type Adaptive or not Bayesian or frequentist Will prediction be used Sophisticated adaptation or just sample size re-estimation
bull Calculate probabilities of success under different scenarios
bull Calculate expected trial duration and expected trial cost
bull Optimize clinical trial design features
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
23
Design features to be optimized
bull Stopping rules for success and futilitybull Number and timing of interim analysesbull Prior probabilities hierarchical model parameters discount
factorsbull Predictive modelbull Minimum sample size (should also consider safety)bull Maximum sample sizebull Randomization ratiobull Accrual rate (not too fast and not too slow)bull Dosetreatment selectionbull Number of centersbull Use of covariates (subgroup analysis)
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
24
I5 Predictive Probabilities
bull Probability of future events given observed data
bull Probability of results for a future patient in the trial
bull Probability of results for missing patients
bull Help to decide when to stop a trial
bull Help to decide whether to stop or to continue recruiting
bull Help physicians and patients make decisions about the use of a treatment (if used in labeling)
bull Predict a clinical outcome from a valid surrogate (modeling)
bull Adjust trial results for missing data
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
25
II Decision Analysis
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
26
Foundations of the Bayesian Approach Approval of medical products
Learn from evidence about benefits and harms of medical products in the presence of uncertainty to make better decisions
Expected Utility of Decision 119941119941119946119946
Whose Utilitybull Regulators FDA Reviewersbull Societybull Patientsbull Physicians and caregiversbull Payers
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
27
II1 Benefit-Risk determinations for medical products
Medical product submitted for approval
Does the benefit outweigh the risk
Possible decisions bull Approve bull Donrsquot approve bull Request more information (value of information)
General Idea
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
28
Points for consideration and discussion
bull How to assess utilities to the benefits and risks
bull Who should assess utilities (reviewers physicians patients public)
bull Should a tradeoff between safety and effectiveness be pre-specified
bull How should we assess the value (or cost) of obtaining additional safety and effectiveness information
bull How should we assess the optimal amount of information (time) that warrants action
Utility assessment Benefit-Risk tradeoff
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
29
Device treatments
1 Life threatening disease no alternative is available2 Life threatening disease alternative is available3 Other devices exist but nothing works well (osteoarthritis)4 Several good alternatives exist (stricter)
IdeaUtility Assessments and Thresholds for Approval
0
01
02
03
04
05
06
07
08
09
1
Pr(
effe
ctiv
e|da
ta)
Pr(safe|data)
Different Utility Assessments
Example 1
Example 2
Example 3
Example 4
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
Chart1
Example 1
Example 2
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
06480331263
1275862069
19108910891
5397260274
06296296296
12
18235294118
4511627907
06114519427
11290322581
17378640777
38585858586
0593495935
10625
16538461538
33571428571
05757575758
1
15714285714
296
05582329317
09411764706
14905660377
26376811594
05409181637
08857142857
14112149533
23708609272
05238095238
08333333333
13333333333
21463414634
05069033531
07837837838
12568807339
19548022599
04901960784
07368421053
11818181818
17894736842
04736842105
06923076923
11081081081
1645320197
04573643411
065
10357142857
15185185185
04412331407
06097560976
09646017699
14061135371
04252873563
05714285714
08947368421
1305785124
04095238095
05348837209
08260869565
12156862745
03939393939
05
07586206897
11343283582
03785310734
04666666667
06923076923
10604982206
03632958801
04347826087
06271186441
09931972789
03482309125
04042553191
05630252101
09315960912
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
005
005
005
005
01
01
01
01
015
015
015
015
02
02
02
02
025
025
025
025
03
03
03
03
035
035
035
035
04
04
04
04
045
045
045
045
05
05
05
05
055
055
055
055
06
06
06
06
065
065
065
065
07
07
07
07
075
075
075
075
08
08
08
08
085
085
085
085
09
09
09
09
095
095
095
095
Sheet1
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
P(S)
Example 1
Example 2
Example 6
Example 3
Example 4
005
06480331263
1275862069
141044776119
19108910891
5397260274
01
06296296296
12
12027027027
18235294118
4511627907
015
06114519427
11290322581
103086419753
17378640777
38585858586
02
0593495935
10625
88636363636
16538461538
33571428571
025
05757575758
1
76315789474
15714285714
296
03
05582329317
09411764706
6568627451
14905660377
26376811594
035
05409181637
08857142857
56422018349
14112149533
23708609272
04
05238095238
08333333333
48275862069
13333333333
21463414634
045
05069033531
07837837838
41056910569
12568807339
19548022599
05
04901960784
07368421053
34615384615
11818181818
17894736842
055
04736842105
06923076923
28832116788
11081081081
1645320197
06
04573643411
065
23611111111
10357142857
15185185185
065
04412331407
06097560976
18874172185
09646017699
14061135371
07
04252873563
05714285714
14556962025
08947368421
1305785124
075
04095238095
05348837209
10606060606
08260869565
12156862745
08
03939393939
05
06976744186
07586206897
11343283582
085
03785310734
04666666667
03631284916
06923076923
10604982206
09
03632958801
04347826087
00537634409
06271186441
09931972789
095
03482309125
04042553191
-02331606218
05630252101
09315960912
Sheet1
Pr(Safe|data)
Pr(Effective|data)
Different Utility Assessments
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
Sheet2
Example 1
Example 2
Example 6
Example 3
Example 4
Pr(safe|data)
Pr(effective|data)
Different Utility Assessments
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
Sheet3
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
30
Real example carotid stents
Analysis of Safety and Effectiveness of previously approved carotid stents to retrospectively determine the approval threshold
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
Chart1
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
CB_DATA_
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
Sheet1
David ChanControl arm with CEA
David Chan454 patients enrolled 11 patients without stent implanted
David ChanAlso provided in SSED
David ChanAlso provided in SSED
David ChanFolow-up table noted 4 (not 6) deaths
David ChanFollow-up table notes 21 (not 24) deaths
David ChanOPC 16 - used conservative 11 although 98 from SAPPHIRE then added 5 delta
David ChanOPC 161
David ChanOPC 136 without delta margin specified
David ChanOPC 145
David ChanOPC 145
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Trial
P060001 Proteacutegeacute
P050025 NexStent
P040038 Xact
P040012 AccuLink
P040012 AccuLink
P040012 AccuLink
P030047 Cordis Precise
P030047 Cordis Precise
P030047 Cordis Precise
Trial notes
Date of approval
12407
102706
9605
83004
83004
83004
42104
42104
42104
Trial arm
ARM 1
ARM 1
ARM 1
ARM 1
ARM 2
ARM 3
ARM 1
ARM 2
ARM 3
Embolic protection
Follow-up
Starting sample size
419
419
1000
443
443
1000
305
305
1000
158
158
1000
278
278
1000
145
145
1000
167
167
1000
167
167
1000
406
406
1000
30 days
413
419
986
438
443
989
295
305
967
156
158
987
278
278
1000
143
145
986
160
167
958
149
167
892
398
406
980
1 year
374
419
893
417
443
941
269
305
882
143
158
905
263
278
946
157
167
940
146
167
874
383
406
943
Included lost to follow-up
Data category
Weight (ref)
LCL
UCL
Weight
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
n
N
nN
LCL
UCL
Weighted
Combined endpoints
Death MI CVA (30 days) + ipsilateral CVA (1 year)
06
052
068
060
29
370
78
55
110
78
553
19
404
47
30
72
47
572
26
305
85
59
122
85
549
13
158
82
49
136
82
551
27
278
97
68
138
97
542
20
167
120
79
178
120
528
32
167
192
139
258
192
485
64
406
158
125
196
158
505
Ipsilateral CVA
03
023
038
030
19
395
48
31
74
48
286
14
421
33
20
55
33
290
24
305
79
53
114
79
276
7
158
44
22
89
44
287
17
278
61
39
96
61
282
63
17
109
63
281
66
13
120
66
280
71
24
117
71
279
Major
02
014
027
020
16
395
41
25
65
41
192
5
421
12
05
27
12
198
11
305
36
20
63
36
193
2
158
13
03
45
13
197
3
278
11
04
31
11
198
1
167
06
01
33
06
199
5
167
30
13
68
30
194
13
406
32
19
54
32
194
Minor
01
006
016
010
4
395
10
04
26
10
99
9
421
21
11
40
21
98
13
305
43
25
72
43
96
5
158
32
14
72
32
97
14
278
50
30
83
50
95
6
167
36
17
76
36
96
3
167
18
06
51
18
98
16
406
39
24
63
39
96
Death (all-cause)
05
042
058
050
35
395
89
64
121
89
456
19
421
45
29
69
45
477
26
305
85
59
122
85
457
14
158
89
54
143
89
456
24
278
86
59
125
86
457
12
167
72
42
121
72
464
21
167
126
84
185
126
437
41
406
101
75
134
101
450
Stroke-related
10
097
100
100
9
395
23
12
43
23
977
0
421
00
00
09
00
1000
3
305
10
03
29
10
990
1
158
06
01
35
06
994
3
278
11
04
31
11
989
61
00
121
61
939
92
00
185
92
908
67
00
134
67
933
Weighted average
944
959
952
949
967
978
973
976
936
953
945
949
936
960
949
956
935
954
945
949
905
947
926
929
857
917
887
890
645
920
827
910
Safety endpoints
08
07
06
05
09
08
13
11
10
09
21
21
30
30
182
93
Death MI CVA (30 days)
03
023
038
030
26
414
63
43
90
63
281
17
438
39
24
61
39
288
23
305
75
51
111
75
277
12
158
76
44
128
76
277
24
278
86
59
125
86
274
11
145
76
43
131
277
8
167
48
24
92
48
286
16
167
96
60
150
96
271
28
406
69
48
98
69
279
Procedural complication
02
014
027
020
81
417
194
159
235
194
161
80
454
176
144
214
176
165
109
305
357
306
413
357
129
11
158
70
39
120
70
186
27
278
97
68
138
97
181
8
145
55
28
105
189
67
167
401
330
477
401
120
34
167
204
150
271
204
159
139
406
342
298
390
342
132
Death (30 days)
10
097
100
100
8
417
19
10
37
19
981
2
438
05
01
16
05
995
3
305
10
03
29
10
990
4
158
25
10
63
25
975
6
278
22
10
46
22
978
2
145
14
04
49
986
2
167
12
03
43
12
988
4
167
24
09
60
24
976
9
406
22
12
42
22
978
Myocardial infarction (30 days)
03
023
038
030
4
417
10
04
24
10
297
1
438
02
00
13
02
299
2
305
07
02
24
07
298
4
158
25
10
63
25
292
8
278
29
15
56
29
291
2
145
14
04
49
296
4
167
24
09
60
24
293
10
167
60
33
107
60
282
7
406
17
08
35
17
295
CVA (30 days)
05
042
058
050
20
417
48
31
73
48
476
15
438
34
21
56
34
483
21
305
69
45
103
69
466
7
158
44
22
89
44
478
15
278
54
33
87
54
473
8
145
55
28
105
472
6
167
36
17
76
36
482
5
167
30
13
68
30
485
20
406
49
32
75
49
475
Weighted average
937
954
946
955
961
973
967
970
924
945
935
939
940
964
953
960
941
960
951
955
966
921
948
935
943
924
950
938
945
925
945
935
939
Effectiveness endpoints
09
08
06
06
11
10
13
11
10
09
14
13
14
12
10
10
Ipsilateral CVA (31 days to 1 year)
10
097
100
100
3
370
08
03
24
08
992
3
421
07
02
21
07
993
3
305
10
03
29
10
990
1
154
06
01
36
06
994
3
272
11
04
32
11
989
2
167
12
03
43
12
988
5
167
30
13
68
30
970
12
406
30
17
51
30
970
Non-stroke neurological
03
023
038
030
8
395
20
10
39
20
294
20
01
39
20
294
25
305
82
56
118
82
275
1
154
06
01
36
06
298
3
272
11
04
32
11
297
20
01
39
20
294
20
01
39
20
294
20
01
39
20
294
Target lesion revascularization
03
023
038
030
1
370
03
00
15
03
299
50
11
89
50
285
2
305
07
02
24
07
298
7
158
44
22
89
44
287
6
278
22
10
46
22
294
1
167
06
01
33
06
298
7
167
42
20
84
42
287
3
406
07
03
21
07
298
Target vessel revascularization
02
014
027
020
1
370
03
00
15
03
199
9
443
20
11
38
20
196
12
00
24
12
198
45
00
89
45
191
23
00
46
23
195
0
167
00
00
22
00
200
0
167
00
00
22
00
200
0
406
00
00
09
00
200
Technical success
02
014
027
020
408
419
974
954
985
974
195
422
454
930
902
950
930
186
295
305
967
941
982
967
193
928
863
993
928
186
264
277
953
921
972
953
191
139
145
959
913
981
145
159
912
858
947
912
182
363
405
896
863
922
896
179
Angiographic success
02
014
027
020
397
397
1000
990
1000
1000
200
423
433
977
958
987
977
195
295
305
967
941
982
967
193
153
156
981
945
993
981
196
268
271
989
968
996
989
198
141
142
993
961
999
145
158
918
864
951
918
184
368
407
904
872
929
904
181
Procedure success
03
023
038
030
419
444
944
918
962
944
283
395
454
870
836
898
870
261
269
305
882
841
914
882
265
143
156
917
863
951
917
275
249
272
915
876
943
915
275
133
142
937
884
966
140
159
881
821
922
881
264
355
404
879
843
907
879
264
Restenosis (ge 50 by ultrasound)
03
023
038
030
27
395
68
47
98
68
279
65
357
182
145
225
182
245
9
221
41
22
76
41
288
216
22
411
216
235
216
22
411
216
235
24
122
197
136
276
197
241
30
96
313
229
411
313
206
78
282
277
228
332
277
217
Weighted average
971
980
976
979
939
953
946
948
954
967
961
964
925
963
945
950
932
968
951
955
929
951
940
947
906
941
925
932
917
936
927
930
Certainty
5
05
04
07
07
07
07
19
18
19
17
12
10
18
16
10
09
Sheet2
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
AccuLink ARM 1 ARCHeR 1 without embolization protection non-randomized
AccuLink ARM 2 ARCHeR 2 with embolization protection non-randomized
Cordis Precise ARM 1 Randomized carotid artery stent
Cordis Precise ARM 2 Randomized carotid endarterectomy
Cordis Precise ARM 3 Non-randomized carotid artery stent (patients too risky for endarterectomy)
Sheet2
2
2
3
1
1
Protege 2007
NexStent 2006
Xact 2005
AccuLink 2004
Cordis Precise 2004
Safety
Effectiveness
09761388286
0946101789
09606862564
09446378133
09404244843
09509008749
092456781
09269073086
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
09462513799
09669656257
09346756747
09529808641
09352963977
00078857266
00088636439
00043597153
00050441631
00058940941
00063841721
00071168419
00074178545
00104580257
00110525382
00065113492
00069983967
00113342688
00134776798
00182509309
00194395404
00127279543
00144851082
00104269665
00116868057
0951177776
09381998925
00090788896
00099047458
00172421343
00186280418
0012108611
00137948978
00160242392
00184459307
09352005998
00095853012
00101101167
00089843841
00095207539
Sheet3
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
31
Factors for Benefit Risk Determinationguidance and article (1)
bull Benefits type magnitude probability durationbull Risks severities types probabilities duration risk of false
positives and false negatives for diagnostic devices
Additional Factors Contextbull Uncertainty bull Severity and chronicity of the diseasebull Patient tolerance for risk and perspective on benefitbull Availability of alternative treatmentsbull Risk mitigationbull Post-market informationbull Novel technology for unmet medical need
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
32
bull Design conduct quality of clinical studiesbull Potential biasesbull Missing data data qualitybull Quality of analyses sensitivity analysesbull Subgroup analysesbull Generalization of results ndash subgroups small sample sizebull What is the probability that a patient in the intended
population will receive the benefits or incur the risksbull Significance level ()
II 2 Uncertainty
Assess the strength of statistical evidence in light of other factors Would accept more uncertainty if the potential benefit is substantial for an unmet need if the safety profile is better than existent if patients would accept more uncertainty
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
33
ldquoRisk tolerance will vary among patients and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit hellip FDA would consider evidence relating to patientsrsquo perspective of what constitutes a meaningful benefitrdquo (guidance (1))
The CDRH - CBER Benefit-Risk guidance document for medical devices did not say how to submit Patient Preference Information to the Center
Patient tolerance for risk and perspective on benefit
II 3 Patient Input
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
34
Proof of Concept Obesity Study
bull Explore how to elicit and incorporate patient preferences into regulatory decision making
bull Device treatments for obesity involve difficult benefit-risk tradeoffs
bull Broad array of devices in the pipeline with diverse benefit-risk profiles
bull Assess feasibility of eliciting patient preferences
bull Assess the use of quantitative patients preferences
bull Patient input had been anecdotal during public hearings
bull Explore the use of results in regulatory decision making process
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
35
Which is a favorable Benefit-Risk tradeoff
Risks
Weight Lossdarr Benefit
darrR
isk
New Device
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
36
Obesity Studybull Sample ~650 subjects with BMI ge 30 willing to lose weight
bull Administered via the Internet
Discrete-Choice Experiment (DCE)
bull Respondents evaluate choices between pairs of hypothetical weight-loss device-treatments
bull Each treatment is defined by its attributes and levels (including surgical procedure)
bull Only devices subjects assumed insurance covers all costs
bull The pattern of choices reveals the patientsrsquo preferences
bull Ex Patients would tolerate 2 more months of mild AEs to lose 25 more pounds
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
37
Attributes and Levels Obesity StudyAttribute Levels
Type of Operation Endoscopic Laparoscopic Open Surgery
Diet restrictions Eat frac14 cup at a timeWait 4 hours between eating Canrsquot eat hard-to-digest foods
Average weight-loss 5 of body weight10 of body weight20 of body weight30 of body weight
How long weight-loss lasts 6 months1 year 5 years
Comorbidity improvement NoneReduce risk (or current dosage) by half Eliminate risk (or current dosage)
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
38
Attribute LevelsHow long side effect lasts None
1 month1 year5 years
Chance of serious Side Effects requiring hospitalization
None5 chance hospitalization no surgery20 chance hospitalization no surgery5 hospitalization for surgery
Chance of dying from getting weight-loss device
None1 3 5 10
Attributes and Levels Obesity Study
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
39
Choice Question Example Feature Device A Device B
Type of operation Endoscopic surgery
Recommended diet restriction Wait 4 hours between meals
On average how much weight is lost
30 lbs 60 lbs
On average how long the weight loss lasts Weight loss lasts 5 years Weight loss lasts 1 year
Average reduction in dose of prescription drugs for diabetes at the lower weight
Eliminates the need for prescription drug
On average how long side effects last (Remember that side effects will limit your ability to do daily activities several times a month)
Last 1 month Last 1 year
Chance of a side effect requiring hospitalization None
Chance of dying from getting the weight loss device
10
(10 out of 100)
1
(1 out of 100)
Which weight-loss device do you think is better for people like you
Device A
Device B
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
40
Results Preference Weights
Better outcomes have significantly higher weights
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
4141
Mortality Risk Weight Loss and Weight-Loss Duration are the most important
Results Preference Weights
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
42
Decision Aid Tool
bull Calculates the minimum benefit patients would require for a treatment with a given mortality risk and other attributes
bull Calculates the maximum risk patients would accept for a treatment with given weight-loss benefit and other attributes
bull Results reported for various levels from risk averse to risk tolerant
bull Calculates the proportion of patients who would choose to get the device instead of status quo
bull The estimated values inform clinicians in the determination of the ldquominimum clinically significant benefitrdquo that will be used in the clinical trial design and analysis
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
Decision Aid Tool MAR
MAR
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
Decision Aid ToolProportion of Patients who prefer receiving the device
over their status quo
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
45
Regulatory Impacts of the Obesity Study
bull The study published in 2015 (see Surgical Endoscopy (2)) quantifies patientsrsquo values to help define minimum clinically meaningful benefit
bull Method adaptable for other medical products
bull DCE Only one of existing preference elicitation methods
bull Maestro System a vagus nerve stimulator indicated for weight-loss was approved on January 14 2015 estimated 10 patients accepting the device was instrumental to its approval
bull Motivated development of a project by MDIC amp CDRH catalog of methods for eliciting patient preference (see report (3))
bull Helped develop the Patient Preference Info guidance document (2016) by CDRH amp CBER (see guidance (4))
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
46
Patient Preference Initiative
See references (2) (3) (4)
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
47
III Lessons Learned
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
48
bull Strict control of type I error at a traditional α is problematic If fixed at the same α level all prior distribution is discounted Discount factors
Direct discountPower priorsHyper-parameters of hierarchical modelsEffective sample size
Discount factors are arbitrary too abstract and difficult for clinicians to provide input Originate endless discussions
bull If control is required α should be determined based on the amount of uncertainty tolerated (by patients clinicians or reviewers) through a decision analysis method see (5) Patient Centered Clinical Trials (Drug Discovery Today)
III Lessons Learned
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
bull Strategy use the full Bayesian approach with a success threshold for the posterior probability See Ebola Trial in Clinical Trials (6)
bull Threshold could be determined via full decision analysis
bull Hierarchical model hyperparameters arbitrary and difficult to assess due to scarcity of clinical input (clinicians donrsquot get it)
bull Hierarchical models are problematic when used with only 2 studies (variability between 2 studies cannot be estimated)
bull Prior information may be available but not legally available
III Lessons Learned
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
50
III Lessons Learned
bull Adaptive Bayesian design is a must when prior distributions are used avoid near misses
bull Simulations extremely helpful at the design stage to strategize and optimize trial designs
bull The use of predictive distributions in labels is very useful but require the statisticians involvement
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
bull Safety (ex Hepatitis B Vaccine Heplisav) (see review memo (8))
bull Expansion of indication
bull Rare diseases or Small populations
bull Unmet medical need for life threatening or irreversible debilitating diseases
bull Expedited Access Program (EAP) (CDRH and CBER) (See EAP guidance (9)) If control of type I error is required the significance level needs to be higher
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
53
Value of Bayes to regulate medical products
bull Account for the totality of evidence by using prior distributions
bull Introduce flexibility into the design of clinical trials ndash adaptive designs and likelihood principle
bull Use decision analysis to make ldquohardrdquo regulatory decisions
bull Define thresholds for approval (or amount of evidence or uncertainty tolerated or significance level) by taking into account additional factors reflecting context
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
54
bull The strength of evidence may be scientifically determined in light of other factors
Medical need (unmet) Severity and chronicity of the disease Does the treatment precludes future treatments (eg
gene therapy) Patient value for benefit and tolerance for risk and
bull More uncertainty could be accepted if The benefit is substantial for an unmet medical need The safety profile is better than existent Patients and physicians will accept it
Value of Bayes to regulate medical products
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
55
bull Patient InputPatient preference information is an important complement to
clinical and statistical evidence and can enhance regulatory decision making
Evidence on patient preference can be scientifically obtained Patient preference information can provide insights to
reviewers who may have very limited experience with rare disease patients
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
(2) Obesity StudyM P Ho J M Gonzalez H P Lerner C Y Neuland J M Whang M McMurry Heath A B Hauber T Irony (2015) ldquoIncorporating patient-preference evidence into regulatory decision makingrdquo Surgical Endoscopy doi 101007s00464-014-4044-2
Irony T and Ho Martin (2016) ldquoBenefit-Risk Determinations at the Center for Devices and Radiological Health at the FDArdquo in Q Jiang and E He (Eds) Benefit Risk Assessment Methods in Medicinal Product Development Bridging Qualitative and Quantitative Assessments Boca Raton FL CRC Press
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proschan MA Dodd L Price D (2016) ldquoStatistical Considerations for a Trial of Ebola Virus Disease Therapeuticsrdquo Clinical Trials doi1011771740774515620145
Chaudhuri SE Ho M P Irony T Sheldon M Lo A W(2017) ldquoPatient-centered clinical trialsrdquo Drug Discovery Today httpsdoiorg101016jdrudis201709016
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
telbaironyfdahhscom
59
Extra slides Choice Model
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
59
Extra slides Choice Model
For any pair of Device Profiles A and B with k attributes the expected probability of choosing device A (C=A) is given by
Here the mean odds ratio for any 2 treatment profiles is
)exp(11)Prob(
sumsum minus+==
kB
kA
ACββ
)exp( sumsum minus BA ββ
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
Proof of Concept Obesity Study
Which is a favorable Benefit-Risk tradeoff
Obesity Study
Attributes and Levels Obesity Study
Slide Number 38
Choice Question Example
Results Preference Weights
Slide Number 41
Slide Number 42
Slide Number 43
Slide Number 44
Regulatory Impacts of the Obesity Study
Patient Preference Initiative
III Lessons Learned
III Lessons Learned
Slide Number 49
III Lessons Learned
Slide Number 51
Promising areas for use of prior information
Slide Number 53
Slide Number 54
Slide Number 55
References
References
Slide Number 58
Extra slides Choice Model
Choice Model
60
Choice Model
The statistical model uses observed choice patterns to quantify preferencesAssumed the logit-choice probability function a
[ ] [ ][ ]
prodsum=
=
=T
tJ
j
ijt
ijti
jTij
U
UCC
1
1
1
exp
expProb
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
a The use of a logit form for the choice probability function was based on McFaddenrsquos seminal work on the analysis of choice behavior (1974)
The model controls for differences across respondentsrsquo preferences (heterogeneity)Many choice practitioners employ the model as accepted good practice
The Value of Bayesian Approaches in the Regulatory SettingLessons from the Past and Perspectives for the Future
Slide Number 2
Outline
I Past ExperienceBayesian clinical trials for the regulation of medical devices
Slide Number 5
HIMAFDA Workshop ndash November 1998 ldquoBayesian Methods in Medical Devices Clinical Trialsrdquo
TransScan T-2000 Multi-frequency Impedance Breast Scanner
Slide Number 8
I 2 The use of prior information
Slide Number 10
Slide Number 11
Slide Number 12
Regulatory Perspective
I 3 Bayesian Adaptive Designs
Example Bayesian adaptive design
Interim Looks for Sample Size Adaptation
Interim looks for Effectiveness
Regulatory Perspective
Regulatory Perspective
I 4 Simulations
Regulatory Perspective
Simulations are essential to strategize trial design
Design features to be optimized
I5 Predictive Probabilities
II Decision Analysis
Slide Number 26
Slide Number 27
Slide Number 28
Slide Number 29
Slide Number 30
Factors for Benefit Risk Determinationguidance and article (1)
Slide Number 32
Patient tolerance for risk and perspective on benefit
