A framework for translating

evidence to

recommendations within a

hospital-based HTA unit

Nisha Almeida, PhD

McGill University Health centre

Disclosure

I have no actual or potential conflict of interest in relation to this topic or

presentation.

2

Background

Proliferation of HTA units

Hospital-based HTA units to support decision-making

Useful tools to adopt cost-effective and clinically pertinent technologies

Yet, while there are numerous guidelines to evaluate and rate the clinical

evidence (e.g. GRADE, Cochrane), there are few comparable tools to

guide decision-makers in framing recommendations

what factors must be considered in this decision-making process?

3

Factors important to decision-making

Everyone agrees that efficacy, safety and cost of the technology are important considerations

Reviewed 24 past TAU reports

Correlated the overall strength of 3 factors (efficacy, cost, safety) with the final recommendation

Approved: overall assessment supports a recommendation for routine use with support through the hospital operating budget

Approved for evaluation: overall assessment does not support permanent approval but is promising enough to warrant an evaluation,

Not approved: The technology is not approved for financial support through the hospital operating budget, but may be re-evaluated based on new evidence

But many more factors/dimensions to consider

4

0

1

2

3

4

5

6

7

8

APPROVED APPROVED FOR

EVALUATION

NOT APPROVED

Fre

qu

en

cy

Recommendation type

Weak Moderate Strong

Previous frameworks

GRADE1 OHTAC2

Domain Comment

Balance between

desirable and

undesirable

effects

The larger the difference between the desirable

and undesirable effects, the higher the

likelihood that a strong recommendation is

warranted. The narrower the gradient, the

higher the likelihood that a weak

recommendation is warranted

Quality of

evidence

The higher the quality of evidence, the higher

the likelihood that a strong recommendation is

warranted

Values and

preferences

The more values and preferences vary, or the

greater the uncertainty in values and

preferences, the higher the likelihood that a

weak recommendation is warranted

Costs (resource

allocation)

The higher the costs of an intervention—that is,

the greater the resources consumed—the lower

the likelihood that a strong recommendation is

warranted

Domain Sub-domain

Overall clinical

benefit

Effectiveness

Safety

Burden of illness

Need

Feasibility of

adoption

Economic feasibility

Organizational feasibility

Value for money Economic evaluations

Consistency with

expected societal

and ethical values

Expected societal values

Expected ethical values

1 Grading of Recommendations Assessment, Development and Evaluation, McMaster University

2 Ontario Health Technology Appraisal Committee

5

Goal

Exhaustively enumerate relevant domains important to the decision-making

process within a hospital setting

To create a checklist that is structured and transparent

to ensure that all relevant domains are given equal consideration

to adequately document the factors and reasons shaping the final

recommendation

That can be applied in a systematic fashion to each technology to be evaluated

6

Objectives

Identify relevant domains that stakeholders in the decision-making process

recognize as important to shape a recommendation.

Propose specific criteria to rate the strength of the evidence, information,

or experience, for each domain.

Estimate the extent to which members of the TAU Policy Committee

endorse the dimensions and criteria, and identify barriers and facilitators to

adoption of the checklist for the decision making process.

7

Methods

Used OHTAC framework as a starting point to define global domains:

Clinical benefit;

Cost;

Feasibility;

Ethics

Performed a review of our past reports, to identify domains that played a role

in the decision-making process

Two members of TAU independently reviewed a random sample of past reports

Considered domains from a survey of hospitals in France, that included 47

determinants (grouped under 8 domains) based on an extensive literature

review

8

Methods

Developed a preliminary list of important domains

Created a survey in order to solicit the opinions of experts within and

outside the MUHC community

To determine the importance of the suggested domains

Gather other relevant domains

Collect information on MUHC values and preferences

9

Survey domains Question: Which of the following domains do you think are most important for consideration by stakeholders when framing recommendations for use of a health technology in a hospital setting?

Domain Sub-domain

Clinical benefit • Magnitude of effectiveness

• Quality of evidence for effectiveness

• Safety

• Burden of illness to patient

• Absence of alternative treatment options

Value for money • Costs related to the technology

• Increased hospital efficiency

• External financial support

Feasibility • Ease of implementation

• Prior hospital experience with the technology

• Need for evidence of effectiveness in the local setting

Ethics and values • Disease is a public health priority

• Disease is rare

• Benefit of technology to society

• Impact on delivery of equitable care

• Impact on patient-important outcomes

Strategic

considerations

• Impact of technology on attracting new patients and/or health professionals

• Impact on creating research opportunities and external collaborations

• Ability to offer a cutting-edge technology or new alternative treatment

• Availability of the technology in other local centres

10

Comments 11

Results

We sent the survey out to 61

panelists, and received 52

completed responses (85%)

12

Type N (Total= 52)

MUHC member 38

MUHC non-member 14

HTA member 16

Administrator 12

0

10

20

30

40

50

60

70

80

90

100

MAGNITUDE OF EFFECTIVENESS

QUALITY OF THE EVIDENCE FOR EFFECTIVENESS

SAFETY OF THE TECHNOLOGY

BURDEN OF ILLNESS TO PATIENT

ABSENCE OF ALTERNATIVES

COST RELATED TO THE TECHNOLOGY

INCREASED HOSPITAL EFFICIENCY

EXTERNAL FINANCIAL SUPPORT FOR THE TECHNOLOGY

EASE OF IMPLEMENTATION

PRIOR HOSPITAL EXPERIENCE WITH THE TECHNOLOGY

NEED FOR EVIDENCE OF EFFECTIVENESS IN THE LOCAL SETTING

DISEASE IS A PUBLIC HEALTH PRIORITY

DISEASE IS RARE

BENEFIT OF TECHNOLOGY TO SOCIETY

IMPACT ON DELIVERY OF EQUITABLE CARE

IMPACT ON PATIENT-IMPORTANT OUTCOMES

IMPACT OF TECHNOLOGY ON ATTRACTING NEW PATIENTS AND/OR HEALTH PROFESSIONALS

IMPACT ON CREATING RESEARCH OPPORTUNITIES AND EXTERNAL COLLABORATIONS

ABILITY TO OFFER A CUTTING-EDGE TECHNOLOGY OR NEW ALTERNATIVE TREATMENT

AVAILABILITY OF THE TECHNOLOGY IN OTHER LOCAL CENTRES

Percent of respondents rating the domain 4 or 5

13

Percent of respondents rating the domain 4 or 5

0

10

20

30

40

50

60

70

80

90

100

MAGNITUDE OF EFFECTIVENESS QUALITY OF THE EVIDENCE FOR

EFFECTIVENESS

SAFETY OF THE TECHNOLOGY

BURDEN OF ILLNESS TO PATIENT

ABSENCE OF ALTERNATIVES

COST RELATED TO THE TECHNOLOGY

INCREASED HOSPITAL EFFICIENCY

EXTERNAL FINANCIAL SUPPORT FOR THE TECHNOLOGY

EASE OF IMPLEMENTATION

PRIOR HOSPITAL EXPERIENCE WITH THE TECHNOLOGY

NEED FOR EVIDENCE OF EFFECTIVENESS IN THE LOCAL SETTING

DISEASE IS A PUBLIC HEALTH PRIORITY

DISEASE IS RARE

BENEFIT OF TECHNOLOGY TO SOCIETY

IMPACT ON DELIVERY OF EQUITABLE CARE

IMPACT ON PATIENT-IMPORTANT OUTCOMES

IMPACT OF TECHNOLOGY ON ATTRACTING NEW PATIENTS AND/OR HEALTH

PROFESSIONALS

IMPACT ON CREATING RESEARCH OPPORTUNITIES AND EXTERNAL

COLLABORATIONS

ABILITY TO OFFER A CUTTING-EDGE TECHNOLOGY OR NEW ALTERNATIVE

TREATMENT

AVAILABILITY OF THE TECHNOLOGY IN OTHER LOCAL CENTRES

MUHC members

MUHC non-members

14

Consensus on domains

Domain Sub-domain Consensus

Clinical benefit

• Magnitude of effectiveness 100

• Quality of evidence for effectiveness 94.2

• Safety 98.0

• Burden of illness to patient 76.9

• Absence of alternative treatment options 57.7

Value for money

• Costs related to the technology 92.3

• Increased hospital efficiency 92.3

• External financial support 21.2

Feasibility • Ease of implementation 78.8

• Prior hospital experience with the technology 34.6

• Need for evidence of effectiveness in the local setting 61.5

Ethics and values

• Disease is a public health priority 69.2

• Disease is rare 46.2

• Benefit of technology to society 73.1

• Impact on delivery of equitable care 78.8

• Impact on patient-important outcomes 90.4

Strategic

considerations

• Impact of technology on attracting new patients and/or health professionals

36.5

• Impact on creating research opportunities and external collaborations 51.9

• Ability to offer a cutting-edge technology or new alternative treatment 69.2

• Availability of the technology in other local centres 63.5

15

Absence of alternatives

Rating Frequency

0 (Don't know) 1.9

1 (Not at all

important) 1.9

2 (Not very

important) 1.9

3 (Somewhat

important) 36.5

4 (Important) 38.4

5 (Indispensable) 19.2

Comments

Depends on context of the patient; would

be important to consider in rare diseases with

serious negative outcomes

Clearly important given financial constraints

The weight to give at this criterion is unclear.

Absence of alternative treatment should not

be used to justify the decision of introduce a

medical technology when there is weak

evidence;

alternative treatment options arguably

belongs in the ethics domain

There are always alternative options. It

should be explicit and understood that

"magnitude of effectiveness" means

magnitude compared to the next best

options

16

Prior hospital experience with

technology

Rating Frequency

0 (Don't know) 0

1 (Not at all

important) 7.7

2 (Not very

important) 23.0

3 (Somewhat

important) 34.6

4 (Important) 26.9

5 (Indispensable) 7.7

Comments

if you are doing a reassessment,

then this item would be important.

Not intrinsically important

although it may help to get going

Not unless this prior experience

generated data on clinical

benefit, cost or other aspects of

interest. Again, something to be

flagged on a "Do not consider" list.

17

Suggestions for new domains

Clarify: comparative effectiveness and safety

Level of innovation

Impact on patient outcomes

No. of patients affected

Cost-effectiveness

Quantify cost-avoidance

Ability to increase cross-institution collaboration

Impact on budget of other units

Impact on services of other units (disruptiveness)

Need for the technology

Impact on attractiveness of the centre

Clinical pertinence

18

Reorganization of domains Old version

Domain Sub-domain

Clinical benefit

1. Magnitude of effectiveness

2. Quality of evidence for effectiveness

3. Safety

4. Burden of illness to patient

5. Absence of alternative treatment options

Value for money

6. Costs related to the technology

7. Increased hospital efficiency

8. External financial support

Feasibility 9. Ease of implementation

10. Prior hospital experience with the technology

11. Need for evidence of effectiveness in the local setting

Ethics and values

12. Disease is a public health priority

13. Disease is rare

14. Benefit of technology to society

15. Impact on delivery of equitable care

16. Impact on patient-important outcomes

Strategic

considerations

17. Impact of technology on attracting new patients

18. Impact on creating research opportunities

19. Ability to offer a cutting-edge technology

20. Availability in other local centres

Domain Sub-domain

Clinical benefit

1. Magnitude of effectiveness

2. Quality of evidence for effectiveness

3. Safety

Impact on

Patient

4. Impact on patient convenience

5. Personal utility: patient values and preference

6. Patient-reported outcomes (QoL)

Value for

money (local setting)

7. Budget impact (Net cost)

8. Costs avoided /increased hospital efficiency

9. No. of patients affected by technology

10. Budget impact on other services

11. Cost-effectiveness

Feasibility

(impact on local setting)

12. Availability of local expertise

13. Disruptiveness

14. Need to generate local evidence

15. Ability to increase cross-institution collaboration

16. Personnel satisfaction

17. Impact of innovativeness of the technology

Impact on

healthcare system

18. Benefit to society (reduces health care costs)

19. Burden on other health care centres

20. Need: unnecessary duplication

Ethical considerations: Disruption of access to care

New version 19

Rating the strength of each criterion 20

DOMAINS Sub-domains RATING, COMPARED TO ALTERNATIVES

Clin

ica

l

be

ne

fit

Magnitude of effectiveness Strong effect Moderate effect Non-inferior

Quality of the evidence Strong Moderate Weak

Safety of the technology Strong effect Moderate effect Non-inferior /Concern for

harm

Imp

ac

t o

n

Pa

tie

nt

Patient perception and

preferences/personal utility

Superior Equivalent Inferior

Impact on patient convenience (pain, side

effects, time, invasiveness)

Superior Equivalent Inferior

Pt-reported outcomes( satisfaction, impact

on QoL/reduction in period of disability)

Superior Equivalent Inferior

Va

lue

fo

r

mo

ne

y

Budget impact (net cost) Low Equivalent High

Costs avoided High Equivalent Low

Number of patients affected Low Equivalent High

Impact on budget of other departments Low Equivalent High

Cost-effectiveness, if available High Equivalent Low

Fe

asi

bili

ty

Availability of local expertise Yes No

Disruptiveness of other services Low Equivalent High

Need to generate local evidence Yes No

Cross-institution collaboration Yes No

Satisfaction of personnel High Equivalent Low

Innovativeness of the technology High Equivalent

Imp

ac

t

on

he

alth

syst

em

Benefit of the technology to society Yes No

Burden on other healthcare services Low Equivalent High

Need for the technology Yes No

Ethical

considerations

Disruption of access to care Low High

An example: IORT for early breast cancer 21 Decision domain Judgment Reason for judgment

Clin

ica

l

be

ne

fit Magnitude of effectiveness Inconclusive Recurrence rate: 3.3% vs. 1.3%. No CIs provided to judge non-inferiority criterion.

Quality of the evidence Low single RCT with serious concerns in the analysis and presentation of results

Safety profile of intervention Equivalent to alternative Complication rate: 17% in IORT vs 15.5% in EBRT;

Pa

tie

nt

imp

ac

t Patient personal utility Superior

Impact on patient convenience Superior Would reduce inconvenience associated with weekly external beam radiation

Pt-reported outcomes Superior

Va

lue

fo

r m

on

ey

Net cost (Budget impact) Moderate If 15 IORT procedures are performed annually instead of EBRT, it would result in

an increase of $11500 in the budget

Costs avoided Balanced Balanced because IORT increases OR time (takes an additional 40-60 minutes

after surgery), but may result in decrease in wait times for EBRT

Number of patients affected Moderate 16 women have been treated with Intrabeam® since November 2013

Budget impact on other services

Cost-effectiveness

Fe

asi

bility

Availability of local expertise Yes

Disruptiveness of other services

Need to generate local evidence Yes single RCT with several concerns and inconclusive results; a field evaluation may

help answer concerns about recurrence rates, and risk

Cross-institution collaboration

Satisfaction of personnel High MUHC radiation oncologists report good experience with using Intrabeam

Innovativeness of the technology High

Imp

ac

t

on

he

alth

syst

em

Benefit of the technology to society Yes

Burden on other health services

Need for the technology Yes

Ethical concerns None

Overall assessment Moderate: Approval for

evaluation

Intrabeam has the potential of reducing unnecessary exposure to radiation, and

the burden on women of repeated EBRT visits.

However, the procedure remains experimental, and without longer term results

on local breast cancer recurrences and safety, this technology can only be

approved within the context of a research study at the MUHC

Future directions

Pilot test the checklist and ratings among members of the MUHC TAU policy

committee: hospital administrators, patient representatives, and quality

and risk management representatives

solicit their input on the practicality and usability of the checklist

Based on feedback, the checklist will be updated and re-tested at a

second meeting, with the eventual goal of pilot testing the checklist at

other hospital-based technology assessment units.

22

Conclusions

Explicit frameworks directed at hospital-based HTAs do not exist

Our framework would

help in creating a systematic and standardized evaluation process by ensuring

all relevant domains are given due consideration

document the decision-making process in an explicit fashion

enable an understanding of the value judgments and reasoning behind

reaching a recommendation

23

Final list of domains DOMAINS

Clinical benefit Magnitude of effectiveness, all clinically important outcomes

Quality of the evidence (including consistency of magnitude and direction of effect between studies)

Safety of the technology

Impact on Patient Impact on convenience (invasiveness, pain, time)

Patient personal utility: Patient perception and preferences

Patient-reported outcomes (QoL)

Value for money

(Impact on the

local setting)

Net cost (includes acquisition, maintenance, procedure and training costs, and accounts for external support)

Costs avoided, measured as increased hospital efficiency

Impact on budget of other departments

Number of patients affected by the technology (budget impact)

Cost-effectiveness, if available

Feasibility

(Impact on the

local setting)

Availability of local expertise (clinical, technical) in the technology

Impact on resources of affected department and/or of services provided by other departments (disruptiveness)

Need to generate evidence of effectiveness in the local setting (due to lack of strong evidence from RCTs)

Ability to increase cross-institution collaboration (through case-sharing or research activities)

Satisfaction of personnel involved with or affected by the technology

Impact of innovativeness of the technology (increase attractiveness of hospital to patients /professionals)

Impact on

healthcare

system /society

Benefit of the technology to society (e.g. technology enables patient to return to work faster, or reduces use of

antibiotics, resulting in cost-savings to healthcare system)

Burden on other healthcare centres: transfers to other centres/ increase in home monitoring

Need for the technology: does it support local innovation and economic growth, or is it unnecessary duplication

of services that increases healthcare costs?

Ethical

considerations

Disruption of access to care by introduction of technology; or by refusal to introduce technology because of cost

alone, when no alternatives available

25

Burden of illness

Rating Frequency

0 (Don't know) 0

1 (Not at all

important) 0

2 (Not very

important) 3.8

3 (Somewhat

important) 19.2

4 (Important) 67.3

5 (Indispensable) 9.6

% rating 4 or 5 76.9

Comments

Mostly important on a national

level

Important but can be very difficult

to capture/compare

In theory this information should

be captured by a cost-

effectiveness metric. A disease

with a low burden of illness should

result in a smaller gain in quality-

adjusted life years and a large

ICER.

burden of illness is

controversial...not sure it belongs

in clinical benefit

important factor but would have

like to had the prevalence of the

problem for patient

26

Survey domains 27 Question: Which of the following domains do you think are most important for consideration by

stakeholders when framing recommendations for use of a health technology in a hospital setting?

Domain Sub - domain

Clinical benefit • Magnitude of effectiveness

• Quality of evidence for effectiveness

• Safety

• Burden of illness to patient

• Absence of alternative treatment options

Value for money • Costs related to the technology

• Increased hospital efficiency

• External financial support

Feasibility • Ease of implementation

• Prior hospital experience with the technology

• Need for evidence of effectiveness in the local setting

Ethics and values • Disease is a public health priority

• Disease is rare

• Benefit of technology to society

• Impact on delivery of equitable care

• Impact on patient-important outcomes

Strategic

considerations

• Impact of technology on attracting new patients and/or health professionals

• Impact on creating research opportunities and external collaborations

• Ability to offer a cutting edge technology or new alternative treatment

• Availability of the technology in other local centres

28

DOMAIN SUB-DOMAIN % OF PARTICIPANTS RATING THE DOMAINS AS IMPORTANT OR INDISPENSABLE (A

RATING OF 4 OR 5)

MUHC MEMBER

(N=38)

MUHC NON-

MEMBER (N=14)

HTA (N=16) ADMINISTRATOR

(N=12)

TOTAL

(N=52)

CLI

NIC

AL

BEN

EFIT

Magnitude of effectiveness 100 100 100 100 100

Quality of the evidence for effectiveness 92 100 100 92 94

Safety of the technology 97 100 100 92 98

Burden of illness to patient 79 71 75 83 77

Absence of alternatives 50 79 81 50 58

VA

LUE

FO

R

MO

NEY

Cost related to the technology 89 100 100 92 92

Increased hospital efficiency 97 79 88 92 92

External financial support for the technology 21 21 25 17 21

FEA

SIB

ILIT

Y

Ease of implementation 79 79 81 75 79

Prior hospital experience with the technology 37 29 25 42 35

Need for evidence of effectiveness in the

local setting

61 64 69 75 62

ETH

ICS A

ND

VA

LUES

Disease is a public health priority 74 57 56 75 69

Disease is rare 42 57 44 33 46

Benefit of technology to society 82 50 56 75 73

Impact on delivery of equitable care 84 64 69 92 79

Impact on patient-important outcomes 92 86 88 100 90

STR

ATE

GIC

CO

NSID

ER

ATI

ON

Impact of technology on attracting new

patients and/or health professionals

39 29 31 33 37

Impact on creating research opportunities 53 50 44 50 52

Ability to offer a cutting-edge technology 74 57 44 83 69

Availability of the technology in other local

centres

63 64 69 58 63