Generalizability of economic evaluations: Using results from other geographic areas or from multinational trials to help inform health care decision making in Canada

Ron GoereeProgram for Assessment of Technology in Health (PATH)Clinical Epidemiology & Biostatistics (CE&B), McMaster UniversityCenter for Evaluation of Medicines (CEM), St. Joseph’s Hospital Hamilton, Ontario

April 26, 2005.

Core Funding for this project was provided by a CCOHTA HTA Capacity Building Grant(2004/05 - Grant #67)

Acknowledgements

PATH (Ron Goeree, Natasha Burke, Gord Blackhouse, Daria O’Reily, Jean-Eric Tarride) is core funded by the Ontario Ministry of Health & Long-term Care (MOHLTC).

New program of research on the evaluation of new and existing health care technologies.

Other collaborators:Andrea Manca, Mark Sculpher (York, UK)

Andrew Willan (University of Toronto)

Background

Pressure on health care decision makers to make more efficient use of scarce health care resources

Increasing demand for ‘high quality’ effectiveness data (RCT or meta-analyses RCTs)

Also increasing demands to determine ‘value for money’ through high quality economic appraisals

Includes stochastic trials (concurrent clinical & economic) and decision analytic models

Background (2)

Clinical and economic evaluations are expensive and time consuming (sometimes taking years)

Ever growing list of technologies that need to be assessed or re-assessed over time

Often funding decisions need to be made today (or at least earlier than years needed to complete trials)

Fortunately, published evaluations from another jurisdiction or from multinational trials are available

Q. How can information from these evaluations be used for local reimbursement/funding decisions?

Generalizability

Many alternative meanings and definitions

Usually refers to extent data collected on a particular patient population, under unique study circumstances, at a particular point in time, treated in a particular way, can be applied to another group of patients, under different study circumstances, at another point in time or under different treatment conditions

Applies to both cost and clinical effectiveness data

Focus of report is not on clinical generalizability

Generalizability (2)

For purposes of this report, we refer to 2 separate but related applications of using results from an economic evaluation or HTA report

First, is where an evaluation is done in another geographic area (e.g. UK) - transferability

Portability, extrapolation, external generalizability

Second, is where the target country (e.g. Canada) participated in a multinational trial, and where country-specific cost-effectiveness is desired

Objectives

To gain a better understanding of the factors affecting, and approaches used for, transferability and analyses of multinational trials

10 inter-related systematic literature reviews

Factors affecting transferability or analyses of multinational trialsApproaches which have been proposed or usedSystems for determining transferability potential

Conceptual papers and empirical studies

Generalizability

Countries Studied Do Not Include Target Country

Countries Studied Include Target Country

Transferability Multinational Economic Evaluations

Factors Affecting Multinational Analyses

Traditional Approaches for Analyzing Multinational Trials

Conceptual Factors Affecting Transferability

Transferability Factors from Empirical Studies

Statistical Applications in Multinational Analyses

Criteria/Guidelines for Determining Transferability

Application of Multilevel Modeling Approaches

Literature Review

Approaches for Transferring Economic Evaluations

Generalizability

Initial Search of Electronic Databases n=4140

Title and Abstract Rejected n=3415 (82%)

Title and Abstract Accepted n=725 (18%)

Bibliography and Hand Search n=81

Full-Text Review n=808

Included in Systematic Review

n=248 (31%)

Not Included in Systematic Review

n=560 (69%)

Generalizability Papers

248248Total Papers62005**145Other Journals/Books (109 with < 5 papers/journal)

112004**5Respiratory Medicine

3320035New England Journal of Medicine

2220025Lancet

2520015Clinical Therapeutics

2220006European Journal of Cancer

3219996Circulation

2519987Health Policy

2119978European Heart Journal

12199610Health Economics

11199512International Journal of Technology Assessment in Health Care

281981-199434Pharmacoeconomics

# PapersYear of Publication

# PapersSource of ArticleNumber of Papers by Source and Year of Publication

Papers by Year of Publication

0

5

10

15

20

25

30

35

1981-1991

1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Year of Publication

# of

Pap

ers

Incl

uded

in R

evie

ws

(N=2

48)

Variability Factors (70+)

Methodological CharacteristicsHealth Care System CharacteristicsProvider CharacteristicsDisease CharacteristicsPatient Characteristics

Proposed Classification System

Variability Factors (1)

24767

101082123

Demographics (age, gender, race), education, socio-economic statusRisk factors, medical history, genetic factorsLifestyle, environmental factorsMortality rates, life expectancyAttitudes toward treatment, culture, religion, hygiene, nutritionCompliance and adherence rates, ethical standardsPopulation values (utilities)Population density, immigration, emigration, travelling patternsIncome, employment rates, productivity, work loss time, friction timeType of insurance coverage, user fees, co-payments, deductiblesIncentives for patients

# CitationsPatient Characteristics

Variability Factors (2)

1767

Epidemiology (incidence/prevalence, disease progression/spread)Disease severity, case mixDisease interaction, co-morbidity, concurrent medications

# CitationsDisease Characteristics

Variability Factors (3)

39172212

11

Clinical practice, conventions, guidelines, norms Experience, education, training, skills, learning curve positionQuality of care providedMethod of remuneration (supplier-induced demand)Patient identificationCultural attitudesIncentives for providers, liability

# CitationsProvider Characteristics

Variability Factors (4)

352120512962133110

Absolute or relative pricesAvailable resources (staff, facilities, equipment), programs, servicesOrganization of delivery system, structure, level of competitionLevel of technological advancement, innovation and availabilityAvailable treatment options (comparators)Capacity utilization, economies of scale, technical efficiencyInput mix (personnel, equip.), specialization of labor, joint productionAccess to programs and services, gatekeepers, historical differencesWaiting lists, referral patternsRegulatory and organizational infrastructure, licensing of productsAvailability of generics or substitutesMarket form of suppliers, payment of suppliers, supplier incentivesIncentives for institutions

# CitationsHealth Care System Characteristics

Variability Factors (5)

89

18436562

Costing methodology, estimation procedures (e.g. productivity cost)Study perspectiveStudy factors (artificial trial conditions, industry-related bias)Timing of the economic evaluationClinical endpoints/outcome measuresDiscount ratesExchange rates, purchasing power paritiesOpportunity cost (foregone benefits)Affordability (CE thresholds)

# CitationsMethodological Characteristics

Systems for Transferability

Review identified 3 criteria, guidelines, decision rule systems for transferability

One focused on a head-to-head comparison of 10 key variability factors – yes/no (Heyland, 1996)

One focused on determining transferability potential – sufficient quality and detail (Spath, 1999)

One focused on determining transferability potential, followed by a sequence of steps for transferability act itself - Decision Chart (Welte, 2004)

Source: Welte, 2004

Transferability Approaches

Target Country Only

Target Country Only

Target Country Only

5

Target Country Only

Target Country Only

Studied Country Only

4

Target Country Only

Mixture of Studied and

Target Country

Studied Country Only

3

Target Country Only

Studied Country Only

Studied Country Only

2

Mixture of Studied and

Target Country

Studied Country Only

Studied Country Only

1

Least to Most Country-Specific Analysis

Unit Cost Data

Resource Utilization Data

Relative Clinical Efficacy Data

Source of Data by Transferability FactorModeling Approach

Transferability Approaches(2)

220#5 - Target country relative clinical efficacy, resource utilization and unit costs

15114#4 - Studied country relative clinical efficacy, target country resource utilization and unit costs

945#3 - Studied country relative clinical efficacy, mixture of studied and target country resource utilization, target country unit costs

11011#2 - Studied country relative clinical efficacy and resource utilization, target country unit costs

101#1 - Studied country relative clinical efficacy and resource utilization, mixture of studied and target country unit costs

Modeling Approaches000PPPs

101Exchange Rates

Non-modeling Approaches

Total

Decision Analytic ModelsTrials

Number of Empirical Studies

Transferability Approach

Multinational Economic Evaluations (MEE)

Factors affecting analyses of multinational trial data similar as with transferability

• clinical practice• unit prices• health care system

Two general approaches for MEE:Analyses that do not recognize the multilevel (hierarchical) nature in the data (patients nested within countries) - traditional approachesAnalyses that do recognize multilevel nature in the data - more recent statistical applications

Classifying Traditional Approaches Analysis of MEE

We developed a classification system based on the source for 2 data inputs (relative clinical efficacy and resource utilization)Three sources of data for each data input

Pooled trial-wideCountry-specific from within the trial (sub-group analysis)Country-specific from data collected outside the trial (separate sub-study)

Early/Traditional Approaches

2Target country-specific from data collected outside trial

Target country-specific from trial data (sub-group)

5

9Target country-specific from trial data (sub-group)

Target country-specific from trial data (sub-group)

4

16Target country-specific from data collected outside trial

Trial-wide pooled data3

15Target country-specific from trial data (sub-group)

Trial-wide pooled data2

80Trial-wide pooled dataTrial-wide pooled data1

# of Studies

Source for Resource Utilization Data

Source for Relative Clinical Efficacy Data

Approach

* With all approaches, country-specific unit costs are used

Problems with Standard Cost-Effectiveness Approach

The standard CE model assumes no between jurisdiction variability by ignoring natural clustering in the data (i.e. by country)This will produce inaccurate cost-effectiveness estimates and underestimate the measure of sampling variation (standard errors)Error will depend on degree of clustering in the data which in a nested dataset (patient within countries) is measured by the intraclass correlation coefficient

Analyses That Recognize Hierarchical Structures

Instead of separating MEE data for purposes of conducting country-specific analyses, an alternative is to analyze the pooled data, but recognize that there are potential country interactions (cost, effects, cost-effectiveness)In other words, the data can be recognized as having ‘levels’ or being hierarchical in nature

Patient levelCountry levelCan even include centre or provider levels

Standard Cost-Effectiveness Analysis (one level)

is cost for patient i

is treatment arm (0, 1)

is mean cost in control arm

is mean cost in treatment arm

random error term

i i iY tα β ε= + +

iY

it

αα β+

iε

Simple Hierarchical Model Multilevel Modeling (MLM)

ij ij j j ij ijY t v u tα β ε= + + + +

fixed effect random effectsIn addition to patients being seen as a random effect, country is also considered a random effect2 sources of error, one for each level in the dataThis model can be extended to include additional levels (e.g. centers) and other ‘cluster-level’ (country) explanatory variables (covariates)Successfully applied, work has just begun with MLM

Conclusions (Transferability)

Our variability review identified over 70 factors which are of potential concern when conducting transferability studies or analyzing multinational economic evaluationsThe most frequently cited factors are: absolute or relative prices, clinical practice, treatment efficacy and demographicsAlmost all (95%) of the empirical transferability studies have used studied country efficacy dataMost (75%) have substituted in target country resource utilization data into the analysis

Conclusions (MEE)

Factors affecting MEE are similar as those for transferability (prices, clinical practice, HC system)Most common approach (66%) for analyzing MEE has been to use trial-wide pooled efficacy and trial-wide pooled resource utilization dataIncreasing concerns that approaches that do not recognize the multilevel nature of MEE will result in biased cost-effectiveness estimates and underestimated standard errorsMLM is a promising new approach to address these concerns

Recommendations

Continue to update/develop guidelines and impose high quality standards of good practiceEncourage detailed and transparent reporting of methods (use of technical reports, web pages)Further development of criteria, guidelines or decision rules for transferability (potential, conduct)Development of guidelines for analyses of MEEFurther development of multilevel approaches (MLM) – model structure, which covariates, number of centers/patients per country,…