-
1
February 15, 2018
MDICx webinar seriesFrom Stories to Evidence: Quantitative
patient-preference information to inform product-development and
regulatory reviews
Shelby ReedProfessor, Duke School of Medicine
F. Reed JohnsonProfessor, Duke School of Medicine
Juan Marcos GonzalezAssistant Professor, Duke School of
Medicine
-
From Stories to Evidence: Quantitative patient-preference
information to inform product-development and regulatory
reviewsMDIC Webinar February 15, 2018
Shelby ReedProfessor, Duke School of Medicine
F. Reed JohnsonProfessor, Duke School of Medicine
Juan Marcos GonzalezAssistant Professor, Duke School of
Medicine
This presentation reflects the views of the authors and should
not be construed to represent the policies of the U.S. FDA.
Preference Evaluation Research Group (PrefER)
-
Patient-centered healthcare movement
3
-
Who is qualified to make relative-importance value
judgments?
Clinicians
Politicians
Patients Informed consent
Individualistic ethic:Individuals are the best judge of their
own welfare
Patient preferences are critical in determining when aproducts
benefits outweigh its risks .
-- Robert M. Califf (JAMA 2017) 4
-
Increasing Support from FDA
4
FDAs guidance on benefit-risk determinations for device
approvals describes patient tolerance for risk and perspective on
benefit as an explicit factor the agency may consider in approval
decisions.
-
Dr. Rob Califf, FDA Deputy, Former FDA Commissioner
6
You dont know peoples preferences unless you ask them. How do
people look at these differences? And I fell in love with the
discrete-choice experiments, which I had heard about from the
Business School, but had not seen in action and I think that
provides major advantages. So this is a great day for me. Its you
know been a long time. Who would have thought it would come from
the device world? I think its a major triumph for the device world
that were here today, not just talking about it, but with the FDA
very involved. To the extent that FDA takes preferences seriously,
I think its a great day.
Release Event for the MDIC Framework for Integrating Patient
Perspective into Medical Device Benefit-Risk Assessments and the
FDA Center for Devices and Radiological Health Draft Guidance, May
13, 2015
-
History of CDRH interest in PPI
7
Based on Levitan, NIH HCS Collaboratory and PCORnet Grand
Rounds, 2016
-
Preference-Sensitive Regulatory Decisions
Risks
WeightLoss
Patient Engagement Advisory Board (2015)Bennett LevitanAvailable
at:
https://www.noexperiencenecessarybook.com/wwzgN/powerpoint-presentation.html
Diet & Exercise
?
8
Less weight loss
Lower risks
-
Regulatory Impact of the Study EnteroMedicss Maestro
Rechargeable
System for weight loss
Device failed to meet its original trial endpoints
Device was approved in January 2015 First new obesity device
approved by FDA since
2007
First approval to result from CDRH's patient preference
initiative
http://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm430223.htm
9
-
FDA understands that patients and care-partners who live with a
disease or condition on a daily basis and utilize devices in their
care may have developed their own insights into and perspectives on
the benefits and risks of devices reviewed...
--August 2016 Guidance
Center for Devices and Radiological Health
10
-
What are preferences?
Qualitative or quantitative statements of the relative
desirability or acceptability of attributes that differ among
alternative interventions.
Medical Device Innovation Consortium (PCBR Framework Report
2015)
Stories from individuals
Evidence representative
of a groupOften obtained from surveys
Defined by what people are willing to
give up
or health states care processes convenience features other
Characteristics or features
-
Approaches Qualitative methods (focus groups, public
meetings)
Identify areas of concern
Provide context for product-development and regulatory
decisions
Simple quantitative methods (ranking, threshold)
Prioritization
Tradeoffs involving only two outcomes
More advanced quantitative methods (choice experiments,
best-worst scaling) Tradeoffs involving more than two outcomes
Statistical preference measures (risk tolerance, minimum
acceptable benefit, time equivalents)
Publishable regulatory-quality evidence
Todays focus: discrete-choice experiments12
-
Discrete-Choice Experiments to Quantify Patient Preferences
Developed, tested, and validated over past 40 years in market
research
transportation planning
environmental economics
health
Daniel McFadden received the Nobel Prize in Economics in 2000
for conceptual and statistical foundations
Increased interest and regulatory support because of commitment
to patient-centered healthcare
10
-
Example Choice Task: Cystic Fibrosis
14Mohamed, Johnson, Balp, Calado, The Patient 2016
-
Choice-Experiment Features
Also known as choice-based conjoint analysis
Alternatives consist of combinations of features
Preferences among alternatives depend on the relative importance
of features
Respondents indicate choices among hypothetical alternatives
Statistical analysis of pattern of choices indicates relative
importance of features
12
-
Survey Development
EndpointsChoice Tasks
Pretest Interviews
ExperimentalDesign
Prefer X
Prefer Y
Prefer X
Prefer YPrefer X
Level B1Level B2Level R1Level R2
Prefer Y
Level B1Level B2Level R1Level R2
Data Collection& Analysis
Preference Weights
Definitions
Steps in a Choice-Experiment Study
16
-
Applications over product life cycles
CLINICAL DEVELOPMENT
REGULATORY REVIEW
ACCESS
Weighted endpoints
Benefit-risk
Value frameworks
Personalized medicine
USE
Evidence reviews
GUIDANCE
Study
1 2 3
Quality + - ++
N 100 50 300
EndptA - + +
EndptB + + -
RTx A
Tx B
Outcome 1Outcome 2Outcome 3
SAE 1SAE 2SAE 3
Side effects
Cost
Convenience
Alternatives
Health outcomes
Disease severity
17
-
Applications over product life cycles
CLINICAL DEVELOPMENT
REGULATORY REVIEW
ACCESS
Weighted endpoints
Benefit-risk
Value frameworks
Personalized medicine
USE
Evidence reviews
GUIDANCE
Weighted PRO
endpoints
EXAMPLE
18
Study
1 2 3
Quality + - ++
N 100 50 300
EndptA - + +
EndptB + + -
RTx A
Tx B
Outcome 1Outcome 2Outcome 3
SAE 1SAE 2SAE 3
Side effects
Cost
Convenience
Alternatives
Health outcomes
Disease severity
-
European Organisation for Research and Treatment of Cancer
Mohamed AF, Hauber AB, Johnson FR, Coon CD. Patient preferences
and linear scoring rules for patient reported outcomes. Patient.
2010;3(4):217-27.
Patient-Reported Outcome Quality of Life QuestionnairePreference
Weights
16
-
Applications over product life cycles
CLINICAL DEVELOPMENT
REGULATORY REVIEW
ACCESS
Weighted endpoints
Benefit-risk
Value frameworks
Personalized medicine
USE
Evidence reviews
GUIDANCE
Weight-Loss
Devices
EXAMPLE
21
Study
1 2 3
Quality + - ++
N 100 50 300
EndptA - + +
EndptB + + -
RTx A
Tx B
Outcome 1Outcome 2Outcome 3
SAE 1SAE 2SAE 3
Side effects
Cost
Convenience
Alternatives
Health outcomes
Disease severity
-
Preferences and Regulatory Decisions
Regulators Benefit-
Risk Threshold
Benefit
Risk o Disapprove
+ Approve
oo
o
o
oo
++
+
+
++
Patients Benefit-Risk Threshold
+
++
o
o
oo
o
+
22
-
Preference-Sensitive Regulatory Decisions
Risks
WeightLoss
Patient Engagement Advisory Board (2015)Bennett LevitanAvailable
at:
https://www.noexperiencenecessarybook.com/wwzgN/powerpoint-presentation.html
Diet & Exercise
?
23
Less weight loss
Lower risks
-
FDA Obesity Study
24Ho, Gonzalez, Lerner, et al., Surgical Endoscopy. 2015
-
FDA Obesity Study
25Ho, Gonzalez, Lerner, et al., Surgical Endoscopy. 2015
-
FDA Obesity Study
Ho et al. Surgical Endoscopy (2015)
30% weight loss 1.2% mortality risk
26Ho, Gonzalez, Lerner, et al., Surgical Endoscopy. 2015
-
Weight-Loss Device Decision Tool
27
-
Applications over product life cycles
CLINICAL DEVELOPMENT
REGULATORY REVIEW
ACCESS
Weighted endpoints
Benefit-risk
Value Personalized medicine
USE
Evidence reviews
GUIDANCE
Value Frameworks
EXAMPLE
28
Study
1 2 3
Quality + - ++
N 100 50 300
EndptA - + +
EndptB + + -
RTx A
Tx B
Outcome 1Outcome 2Outcome 3
SAE 1SAE 2SAE 3
Side effects
Cost
Convenience
Alternatives
Health outcomes
Disease severity
-
Value Frameworks
2014 2015 2016 2017
29
-
StdCareExpTrx
StdCareExpTrx
QALYsQALYsCC
ICER
=
To compare cost-effectiveness for treatments across conditions,
outcomes must be measured using the same units.
Traditional Value Assessment
-
Other Elements of Value
Other Elements of Value
Option value
Value of knowing
Equity
Value of hope
Impact on
others
Dosing regimen
Clinicalbenefits
Cost
Value
Adapted from Garrison L, 2016 and Neumann PJ, 2016.
31
-
ASCO Value Framework Scoring Rubric
0 to 100 (OS>PFS>RR)
Clinical benefit-20 to 20
Toxicity0 to 10
Palliation*0 to 20
Survival curve* 0 to 10
Trx-free interval*0 to 20
Quality of life*
Clinical benefit Toxicity Bonus points*Net health Benefits Cost
per month
DAC:____Patient payment:___
-
Drug Abacus from MSKCC
$12,000 to $300,000
10% to 30%
1.0 to 3.0
1.0 to 3.0
1.0 to 3.0
1.0 to 3.0 Monthly drug price
http://www.drugabacus.org/drug-abacus/tool/
-
Applications over product life cycles
CLINICAL DEVELOPMENT
REGULATORY REVIEW
ACCESS
Weighted endpoints
Benefit-risk
Value frameworks
Personalized medicine
USE
Evidence reviews
GUIDANCE
Crohns Disease
EXAMPLE
34
Study
1 2 3
Quality + - ++
N 100 50 300
EndptA - + +
EndptB + + -
RTx A
Tx B
Outcome 1Outcome 2Outcome 3
SAE 1SAE 2SAE 3
Side effects
Cost
Convenience
Alternatives
Health outcomes
Disease severity
-
More time in remission
Higher risk of cancer
Lower risk of infections
Less use of steroids
TNF- inhibitors vs. Corticosteroids for Crohns disease
Evidence Reviews: Benefits and harms of Crohns disease
therapies
More time in remission
Higher risk of cancerLower risk of infections
Less use of steroids
35
-
Choice Question Example: Crohns Disease
PCORI, NCT02316678; PI: James Lewis 36
-
Patient-Centered Comparative Effectiveness Research
Patient Preferences
Real-World DataDCE
Simulation Model
Time equivalents Treatment 1 Treatment 25.5 3.7
Attributes
Add-on therapy
Hospital-izations Survival
Adverse event 1
Adverse event 2
Time equivalents Treatment 1 Treatment 2Group 1 Group 2 Group
3
Outcomes
Add-on therapy
Hospital-izations Survival
Adverse event 1
Adverse event 2
Time Equivalents
Event rates: Treatment 1Treatment 2
Time equivalents Treatment 1 Treatment 2Group 1 Group 2 Group
3
-
Comparative Effectiveness
Anti-TNFs Prolonged CS DifferenceRemission-time equivalents,
mean (SD)
5.3 (4.0) 4.5 (3.7) 0.8 (0.5 1.1)
Medicaid and Medicare claims analysis
Markov modelDCE
PCORI, NCT02316678; PI: James Lewis38
-
Potential Patient-Centered Applications
Effects Treatment 1 Treatment 2Add-on therapy - +
Hospitalizations + -
Survival - +
Adverse event 1 + -
Adverse event 2 - +
Traditional Comparative Effectiveness Research
Effects Treatment 1 Treatment 2Add-on therapy - +
Hospitalizations + -
Survival - +
Adverse event 1 + -
Adverse event 2 - +
Time equivalents
Preference-based Comparative Effectiveness Research
-
Latent-Class Choice-Model EstimatesCrohns Disease
-10-9-8-7-6-5-4-3-2-10
0 4 8 12 0 4 8 12 0 4 8 12 0 2 8 12 0 5 1530 0 2 5 8 0 2 5 8
SevereDuration
ModerateDuration
MildDuration
SteroidDuration
InfectionRisk
CancerRisk
SurgeryRisk
Efficacy Class Steroid Class Risk Class
PCORI, NCT02316678; PI: James Lewis 40
-
Comparative Effectiveness
Remission-time equivalents
Anti-TNFs Prolonged CSDifference (95% CI)
Efficacy Class 1.3 (6.7) 0.1 (6.4) 1.3 (0.8, 1.7)
Steroid Class 6.9 (2.9) 6.4 (2.7) 0.6 (0.4, 0.8)
Risk Class 7.8 (2.6) 7.3 (2.5) 0.5 (0.3, 0.7)
PCORI, NCT02316678; PI: James Lewis
Medicaid and Medicare claims analysis
Markov modelDCE
41
-
Applications over product life cycles
CLINICAL DEVELOPMENT
REGULATORY REVIEW
ACCESS
Weighted endpoints
Benefit-risk
Value frameworks
Personalized medicine
USE
Evidence reviews
GUIDANCE
Shoulder dislocation
EXAMPLE
42
Study
1 2 3
Quality + - ++
N 100 50 300
EndptA - + +
EndptB + + -
RTx A
Tx B
Outcome 1Outcome 2Outcome 3
SAE 1SAE 2SAE 3
Side effects
Cost
Convenience
Alternatives
Health outcomes
Disease severity
-
Preferences in Practice Guidelines
Clinicians must communicate evidence-based options for
treatment, inclusive of their benefits and risks, and patients must
be allowed to express their goals and preferences.
Gynecol Oncol. 2016;143(1):3-15.
-
Potential Patient-Centered Applications
Patient characteristicsdemographic and contextual variables
Choice questionsgenerate patient-level preference weights, or
classify patients into preference groups
Decision model predict expected outcomes with alternative
treatments
Compute net benefitscombine expected outcomes with preference
weights
Decision aids
-
First-time shoulder dislocationPersonalized Medicine
45
-
0 5 10 15 20 25 30 35
Limits on Shoulder Motion
Avoid High-Risk Activities
Duration of PT
Chance of Recurrence
Out-of-Pocket Cost
Relative Importance
Importance of Attributes in Shoulder Dislocation
N=374
Personalized Medicine
Streufert BD, Reed SD, Johnson FR, Huber JC, Orlando LA, Taylor
DC, Mather III RC. Orthop J Sports Med. PMID 28377932
46
-
Decision-analytic model
Personalized Medicine
Adaptive choice questions to generate patient-level
preference weights
Demographics and contextual variables
Patient Output:
47
Clinical literature
-
Benefits of Personalized Medicine
Patients More informed patients Improved patient-provider
communication Greater patient satisfaction Improved adherence
and
patient outcomes
Health System Improved efficiency of health
care delivery Documentation of patient-
centered care Provide justification for
changes in practice patterns
-
Opportunities and Challenges
Documenting patient concerns about new devices is useful for
regulatory reviews and at other points in product life cycles.
Stated-preference methods are relatively unfamiliar and there is
limited experience with health applications.
Methods are highly flexible and can be adapted for evaluating
almost any preference-sensitive decision.
FDA has provided guidance on developing patient-preference
evidence for devices.
49
-
50
DISCUSSIONPlease submit your question
via the chat box
-
Next session
How do you define study objectives and scope?
What study-team skills are required?
How do you work effectively with the technical team?
What steps are required for conducting a regulatory-quality
study?
How long will it take and what will it cost?
51
Planning and implementing a choice-experiment study
-
Contact Information
Shelby [email protected]
919 668 8991
Reed [email protected]
919 668 1075
Juan Marcos [email protected]
919 668 5157
52
mailto:[email protected]:[email protected]:[email protected]
-
53
Join us for the next 2 sessions
March 15 - Session 2: Example applications and lessons
learnedinstrument development
April 19 - Session 3: Example applications and lessons
learnedanalysis and reporting
Recordings will be available on http://mdic.org/mdicx
February 15, 2018From Stories to Evidence: Quantitative
patient-preference information to inform product-development and
regulatory reviewsMDIC Webinar February 15, 2018Patient-centered
healthcare movementWho is qualified to make relative-importance
value judgments?Increasing Support from FDADr. Rob Califf, FDA
Deputy, Former FDA CommissionerHistory of CDRH interest in
PPIPreference-Sensitive Regulatory DecisionsRegulatory Impact of
the StudyCenter for Devices and Radiological HealthWhat are
preferences? ApproachesDiscrete-Choice Experiments to Quantify
Patient PreferencesExample Choice Task: Cystic
FibrosisChoice-Experiment FeaturesSteps in a Choice-Experiment
StudyApplications over product life cyclesApplications over product
life cyclesEuropean Organisation for Research and Treatment of
CancerApplications over product life cyclesPreferences and
Regulatory DecisionsPreference-Sensitive Regulatory DecisionsFDA
Obesity StudyFDA Obesity StudyFDA Obesity StudyWeight-Loss Device
Decision ToolApplications over product life cyclesValue
FrameworksTraditional Value AssessmentOther Elements of ValueASCO
Value Framework Scoring RubricDrug Abacus from MSKCC Applications
over product life cyclesSlide Number 35Choice Question Example:
Crohns DiseasePatient-Centered Comparative Effectiveness
ResearchComparative EffectivenessPotential Patient-Centered
ApplicationsLatent-Class Choice-Model EstimatesCrohns
DiseaseComparative EffectivenessApplications over product life
cyclesPreferences in Practice GuidelinesPotential Patient-Centered
ApplicationsPersonalized MedicinePersonalized MedicinePersonalized
MedicineBenefits of Personalized MedicineOpportunities and
ChallengesSlide Number 50Next sessionContact InformationJoin us for
the next 2 sessions
