Stated Preference Methods

Research in Health Care

Decision Making

A Critical Review of Its Use in the

European Regulatory Environment.

Kevin Marsh, Evidera

Axel Mühlbacher, Hochschule Neubrandenburg

Janine van Til, University of Twente

7 November 2017

Objectives

▪ To map which stated preference methods are being used

in the European regulatory environment

• “A more systematic consideration of preferences as part of

regulatory decisions”

• Identification, weighting and aggregation of decision criteria

▪Outline, conclusion:• Identification of gaps in the use of preference methods

• Implications of existing experience for the use of stated preference

methods

• A research agenda for the development of stated preference

methods

• Limitations with the approach adopted in this research

Overview of the Review

Do

ne

alr

ead

y

Cu

rren

t

tas

k

Problem: Literature and website reviews revealed little information on stated preference methods

used

Purpose of survey: Identification of stated preference methods used by the institutions involved in

HTA

INTRODUCTION

TBD

▪ SIG WG objectives

– Definition of preference methods

▪ Process / findings to date

▪ Next steps

▪ Introduce the workshop

CASE STUDY 1

CEA VS MCDA

Case study 1: Comparison

NICE GYMESZI

Location England and Wales Hungary

Decision type Reimbursement Reimbursement

Technology Medicines New hospital

medical

technologies

(devices)

Stakeholder

preference

General population Decision makers

Method TTO Point allocation

Status In use In use

Case study 1: NICE

8

𝐼𝐶𝐸𝑅 =𝐶1 − 𝐶0𝐸1 − 𝐸0

QALY

Years of lifeUtility

Patient experienceGeneral population preference

Abbreviations: ICER = incremental cost-effectiveness ratio; QALY = quality-adjusted life year

Case study 1: NICE

9

Abbreviations: ICER = incremental cost-effectiveness ratio; QALY = quality-adjusted life year

𝐼𝐶𝐸𝑅 =𝐶1 − 𝐶0𝐸1 − 𝐸0

QALY

Years of lifeUtility

Patient experienceGeneral population preference

NICE (2014) Consultation Paper. Value Based Assessment of Health Technologies

Abbreviations: HRQoL = health-related quality of life; ICER = incremental cost-effectiveness ratio

Case study 1: GYMESZI

▪ Implemented in 2010

▪ Evaluation of 14

applications for medical

technology

▪ Criteria and weights

established by a committee

comprising of:

– The Health Financing Agency

– The Ministry of Health

– Clinical experts

– Health economists

10

Case study 1: Assessment

11

Criteria Description NICE GYMESZI

Completeness Are all criteria considered

Consistency/

transparency

Are criteria treated consistently across

evaluations?

Opportunity cost Are costs appropriated considered

Trade-offs Are criteria weights valid

Criteria

properties

Does the analytical framework

appropriately reflect the nature of the

criteria

/ ?

Case study 1: Poll

▪ Which of the two examples do you think is the appropriate way

to incorporate preferences into reimbursement decisions

– NICE

– GYMESZI

– NEITHER

12

CASE STUDY 2

IQWIG’S PILOTS: DCE VS AHP

German pilot – DCE Hepatitis C

Discrete Choice Experiment

Mühlbacher, A. & Johnson, F.R. Appl Health Econ Health Policy (2016) 14: 253. doi:10.1007/s40258-

016-0232-7

German pilot – DCE Hepatitis C

▪ Objectives:• The pilot’s aim was to investigate to what extent a DCE can be

used as a method for the identification, weighting and prioritization in the case of multiple endpoints

▪ Method:• DCE including 7 attributes of antiviral treatment of hepatitis C:

• Treatment efficacy (Probability of sustained viral response 6 months after end of therapy)

• Adverse events (Duration of flu-like symptoms after injection, Probability of gastro-intestinal symptoms, Probability of psychiatric symptoms, Probability of skin symptoms and/or alopecia)

• Treatment burden (Duration of antiviral therapy, Frequency of interferon injections)

• 2 Subsamples: Patient and expert respondents

German pilot – DCE Hepatitis C

▪ Conclusion:

• For patients it was shown that it was feasible to weight patient-

relevant outcomes via a DCE

• In the comparison of patient preferences and opinions of

healthcare professionals, the sequence was the same for 4 of the 7

attributes; however, the magnitude of the weighting deviated for

further attributes.

• DCE method can be applied to other questions

German pilot – AHP Depression

German pilot – AHP Depression

▪ Objectives:

• This pilot project’s goal was to examine to what extent the AHP

method can be applied in health economic evaluations in Germany

in the identification, weighting, and prioritization of multiple patient-

relevant outcomes.

▪ Method:

• AHP evaluating 11 endpoints of antidepressant treatment

• 2 Subsamples: Patient and expert respondents

• Response

• Remission

• Cognitive function

• Reduction of anxiety

• Social function

• Avoidance of relapse

• Reduction of pain

• Other serious adverse events

• (Attempted) Suicide

• Other adverse events

• Sexual dysfunction

German pilot – AHP Depression

▪ Results: ▪ Conclusion:

• AHP method can be applied

both in patients and

healthcare professionals

• AHP enables elicitation of

preferences of individuals

for certain treatment goals

and outcomes in a step-by-

step approach and

calculation of the weights for

each of these outcomes by

means of a matrix algebra

Danner, Marion, et al. "Integrating patients' views into health technology

assessment: Analytic hierarchy process (AHP) as a method to elicit patient

preferences." International journal of technology assessment in health care 27.4

(2011): 369-375.

German pilot –

DCE PERIODONTAL DISEASE

VALUE IN HEALTH 19 ( 2 0 1 6 ) A347– A766

[Available in German

only]

German pilot –

DCE PERIODONTAL DISEASE

▪Objective:

• The pilot’s aim was to explore whether a preference

elicitation using a DCE can be conducted within 3

months

▪ Method:

• DCE including 4 attributes of treatment and disease characteristics

of periodontal treatment alternative

• Tooth loss

• Symptoms & complaints

• Frequency of periodontist visits

• Cost

• Patient respondents

German pilot –

DCE PERIODONTAL DISEASE

▪Results:• Relative importance of attributes

• Tooth loss (0.73 relative weight)

• Symptoms & complaints (0.22)

• Frequency of periodontist visits (0.03)

• Costs (0.02)

▪ Conclusion:

• DCE is feasible within 3 months

DCE versus AHP

Criterion DCE AHPMethodological approach Decompositional Compositional

Basis assumption

Characteristics independently of one another,

but interactions can be tested; any

combinations of attributes are possible

Characteristics independently of one another

Evaluation process Holistic evaluation of stimuliPair-wise comparisons of alternatives and

decision criteria

Reality levelHigh, but sometimes assessment task is

complexLess realistic, but easy assessment task

Benefit/ value model Additive part worth model Weighted additive model

Flexibility according benefit/ value

functionHigh. Different utility functions possible Minor. Only additive value function possible

Target object/ respondentsMarket segment on the basis of individual

customerSingle decision-maker or group

Scale level of the input Ordinal or interval scaled Interval scaled

Scale level of the output Interval scaled Ratio scaled

Estimation technique e.g. OLS, RPL, MXL, LC (…) e.g. Eigenvalue

Interpretation of importance weights Part worth value of attribute (utility scale)Relative importance of one criteria for target

achievement (no utility scale)

Feedback during evaluation process Validity testing (live feedback not easy)Consistency test and sensitivity analysis

possible

Cognitive stress for respondents High. Grows with the increasing of attributes Minor

Survey rangeLess, but complex assessment of complete

stimuliVarious, but easy pair-wise comparisons

Restrictions on use Up to six attributes with 2-4 levels Various attributes possible

Mühlbacher/Kaczynski (2015), Mühlbacher/Johnson (2016), Neidhardt et al. (2012), Mulye (1998), Helm et al. (2003)

Backup – DCE PERIODONTAL DISEASE

VALUE IN HEALTH 20 ( 2 0 1 7 ) A 3 9 9 – A 8 1 1

Backup – DCE PERIODONTAL DISEASE

▪ Objective:

• The study’s aim is to test validity with changing attribute and

multidimensionality

▪ Method:

• Two DCE decision-making models (model 1 and model 2)Model 1• Tooth loosening/tooth loss

(compound attribute with varying severity and the number of teeth concerned)

• Gum bleeding,

• Pain in everyday life (considered both the severity and the duration of pain)

• Pain during therapy

• Therapy administration

• Application of antibiotics

Model 2• Tooth loosening/tooth loss

(compound attribute with varying severity and the number of teeth concerned)

• Gum bleeding,

• Pain in everyday life (considered both the severity and the duration of pain)

• Pain during therapy

• Side effect: infection

• Side effect: antibiotic resistance

Backup – DCE PERIODONTAL DISEASE

▪ Results – Model 1Random parameter logit model (95% confidence interval) (model 1,

N=300)

Backup – DCE PERIODONTAL DISEASE

▪ Results – Model 2Random parameter logit model (95% confidence interval) (model 2,

N=310)

CASE STUDY 3

EMA’S PILOTS: SWING

WEIGHTING WORKSHOPS VS

PATIENT SURVEYS

Articles:

▪ Incorporating Patient Preferences Into Drug Development and

Regulatory Decision Making: Results From a Quantitative Pilot

Study With Cancer Patients, Carers, and Regulators –

Postmus et al., 2017

▪ Is quantitative benefit–risk modelling of drugs desirable or

possible? – Phillips et al., 2011

Is quantitative benefit–risk modelling of drugs

desirable or possible? - Phillips et al., 2011

▪ Objective: To determine whether quantitative benefit-risk

modelling is possible

▪ Method: MCDA with the expected utility rule

▪ Case study: Weight Loss Drug

▪ Value Tree: one favourable effect and five unfavourable effects

▪ Value Functions: concave value function, provided by an

expert roleplaying an assessor

▪ Weight Elicitation: Swing Weighting

Quantitative Pilot Study With Cancer Patients,

Carers, and Regulators - Postmus et al., 2017

▪ Objectives: To explore the feasibility of a lean method for

eliciting individual patient preferences

▪ Method: MCDA

▪ Case study: Treatment of Melanoma

▪ Value Tree: one favourable effect and two unfavourable effects

▪ Value Functions: linear value function, assumption by team

▪ Weight Elicitation: Swing Weighting – Ordinal Judgements

MCDA - Swing Weighting

Weighting equates the units of preference value across all scales.

Philips et al.,

2011

Best Worst

Weight Loss 15% 0%

Anxiety No Yes

Sleep Disorders No Yes

Mood Alterations No Yes

Depressive

Disorders

No Yes

Irritability No Yes

Postmus et al.,

2016

Best Worst

Overall Survival 65% 45%

Moderate Toxicity 5% 20%

Severe Toxicity 15% 35%

MCDA - Swing Weighting – Example Postmus

Full RankingFull Ranking

MCDA - Swing Weighting – Bisection Method

Worst

Survival (1st) X < or > 55%? 45%

Severe Toxicity (2nd) 15% 35%

45% ----------------------55%---------------------> 65%

15% ----------------> 35%

?

MCDA - Swing Weighting – Bisection Method

Worst

Survival (1st) X < or > 50%? 45%

Severe Toxicity (2nd) 15% 35%

45% ----------------------55%---------------------> 65%

15% -> 35%

?

50%

MCDA - Swing Weighting – Bisection Method

Worst

Survival (1st) X < or > 60%? 45%

Severe Toxicity (2nd) 15% 35%

45% ----------------------55%---------------------> 65%

15% -------------------------------> 35%

?

60%

Weight of least importnat criterion is determined relative to 2nd most important criterion

Postmus et al., 2016

Is quantitative benefit–risk modelling of drugs

desirable or possible? - Phillips et al., 2011

▪ Method: MCDA with the expected utility rule

▪ Case study: Weight Loss Drug

▪ Value Tree: one favourable effect and five unfavourable effects

▪ Value Functions: concave value function, provided by an

expert roleplaying an assessor

▪ Weight Elicitation: Swing Weighting – Relative Judgements

SWING Weighting – Philips et al., 2011

– Data collection:

• Role-playing by experts -> Verbal elicitation

process

– Unfavourable Effects:

• Compare all unfavourable effects. Determine

most important effect (W = 100)

• Determine relative weight all other effects

compared to most important (½; ¼).

– Favourable vs. Unfavourable Effects:

• 0 -> 15% Weightloss OR All Unfavourable

Effects -> No Unfavourable Effects

Benefits of SWING weighting - Philips

▪ Swing weighting results in explicit assessment of assessors

judgements, which could increase transparancy and

consistency

▪ Swing weighing can accommodate any kind of data

▪ Using pre-defined scales (and global estimates of weight

ranges) allows weights to be assigned to all scales even

before data about the options are considered

Benefits of SWING weighting - Postmus

▪ Bisection swing weighting was considered intuitive by patients

▪ Regulators felt that outcomes where useful to identify

subgroups of patients

▪ Recommendation to combine with face-to-face meetings with

patients to understand preference constructions and context

▪ Regulators felt that outcomes where easy to interpret, time

frame of elicitation process was feasible

49

Case study 3: Assessment

Philips et al., 2011 Postmus et al., 2016

Elicitation Process Ratio Statements Ordinal Statements

Value tree Lower level of

Aggregration

High level of

Aggregration

Outcomes (weights) Potentially high(er) level

of discrimination

Lower level of

discrimination

Outcomes (value tree) Elicited from experts Assumed linear by

decision analysts

Feasibility Questions

based on studies

Can patients/regulators

participate in verbal

elicitation process?

What is the feasibility

with a higher number of

criteria?

CONCLUSION

TBD

▪ Pull together the themes emerging from the case studies