Methods for Systematic Reviews of Economic Evaluations for ...€¦ · Methods: The following steps were taken for systematic reviews of economic evaluations: (1) systematic searches

Methods for SystematicReviews of Economic Evaluationsfor the Guide to Community Preventive ServicesVilma G. Carande-Kulis, MS, PhD, Michael V. Maciosek, PhD, Peter A. Briss, MD, Steven M. Teutsch, MD, MPH,Stephanie Zaza, MD, MPH, Benedict I. Truman, MD, MPH, Mark L. Messonnier, MS, PhD,Marguerite Pappaioanou, DVM, PhD, Jeffrey R. Harris, MD, MPH, Jonathan Fielding MD, MPH, MBA,Task Force on Community Preventive Services

Objectives: This paper describes the methods used in the Guide to Community Preventive Services:Systematic Reviews and Evidence-Based Recommendations (the Guide) for conducting systematicreviews of economic evaluations across community health-promotion and disease-preven-tion interventions. The lack of standardized methods to improve the comparability ofresults from economic evaluations has hampered the use of data on costs and financialbenefits in evidence-based reviews of effectiveness. The methods and instruments devel-oped for the Guide provide an explicit and systematic approach for abstracting economicevaluation data and increase the usefulness of economic information for policy making inhealth care and public health.

Methods: The following steps were taken for systematic reviews of economic evaluations: (1) systematicsearches were conducted; (2) studies using economic analytic methods, such as costanalysis or cost-effectiveness, cost-benefit or cost-utility analysis, were selected according toexplicit inclusion criteria; (3) economic data were abstracted and adjusted using astandardized abstraction form; and (4) adjusted summary measures were listed in summarytables.

Results: These methods were used in a review of 10 interventions designed to improve vaccinationcoverage in children, adolescents and adults. Ten average costs and 14 cost-effectivenessratios were abstracted or calculated from data reported in 24 studies and expressed in 1997USD. The types of costs included in the analysis and intervention definitions variedextensively. Gaps in data were found for many interventions.

Medical Subject Headings (MeSH): cost-effectiveness, costs, economic evaluation, system-atic reviews (Am J Prev Med 2000;18(1S):75–91) © 2000 American Journal of PreventiveMedicine

Introduction

This paper describes the procedures and instru-ments used to collect, abstract, adjust, and sum-marize results from economic studies reporting

on cost, cost-effectiveness, cost-benefit or cost-utility of

selected interventions for the Guide to Community Pre-ventive Services: Systematic Reviews and Evidence-Based Rec-ommendations (the Guide). Methods were developed sothat studies using disparate analytical methods can beconsistently compared. Although these methods weredeveloped specifically for the Guide, they are sufficientlycomplete and general enough to be readily adapted toa range of systematic reviews of economic evaluations inhealth care and public health.

Evidence-based reviews of effectiveness (e.g., theGuide to Clinical Preventive Services1) usually have notreported data regarding costs and financial benefits ofpreventive interventions. Users of evidence-based effec-tiveness reviews often do not have enough informationto identify among effective interventions, those provid-ing the greatest amount of health or financial benefitper dollar of resource used. This might be because ofthe absence of economic evaluation studies or prob-

From the Division of Prevention Research and Analytic Methods,Epidemiology Program Office (Carande-Kulis, Briss, Zaza, Truman,Messonnier, Harris), Centers for Disease Control and Prevention(CDC), Atlanta, Georgia; Health Partners Research Foundation(Maciosek), Bloomington, Minnesota; Merck & Co., Inc. (Teutsch),West Point, Pennsylvania; Division of Global Health (Pappaioanou),CDC, Atlanta, Georgia; UCLA Schools of Public Health and Medi-cine, and County of Los Angeles Department of Health Services(Fielding), Los Angeles, California

The names and affiliations of the Task Force members are listed onpage v of this supplement and at http://www.thecommunityguide.org

Address correspondence and reprint requests to: Vilma G. Car-ande-Kulis MS, PhD, Economist, Community Preventive ServicesGuide Activity, Epidemiology Program Office, Centers for DiseaseControl and Prevention, 4770 Buford Highway MS-K73, Atlanta, GA30341: E-mail: [email protected].

75Am J Prev Med 2000;18(1S) 0749-3797/00/$–see front matter© 2000 American Journal of Preventive Medicine PII S0749-3797(99)00120-8

lems of comparability among available studies. Al-though little can be done on a short-term basis toincrease the quantity of available studies, systematicreviews of economic studies can address problems ofcomparability. In addition, they can improve the use-fulness of existing studies just as systematic reviewsbring together and interpret a body of evidence of theeffectiveness of interventions.

The lack of accepted, or even openly debated, meth-ods and instruments for systematic reviews of economicstudies is significant in light of both the need for suchmethods and the precedent set for such methods in theeffectiveness literature. Inconsistencies in the methodsemployed in published, peer-reviewed economic evalu-ation studies make results not comparable and lead toskepticism2 regarding the validity of the results of suchstudies. The lack of standardization in the design,analysis and reporting of economic evaluations is amatter of concern because changes in design andanalysis produce substantially different results.

Independent published reviews regarding cost-effec-tiveness of clinical and public health interventions havebeen conducted by several authors. Tengs et al.3 con-ducted a comprehensive review on the cost-effective-ness of 500 life-saving interventions in the UnitedStates, including motor vehicle safety devices, controlof environmental hazards, cancer screening, heart dis-ease screening and treatment, vaccination for infec-tious diseases and interventions to promote smokingcessation. Interventions ranged from those that arecost-saving to those that cost $10 billion per life-yearsaved, with a median cost of $42,000 per life-year saved(1993 USD). Ramsberg et al.4 conducted a reviewregarding the cost-effectiveness of 165 life-saving inter-ventions in Sweden, including screening and treatmentfor hypertension, road safety, smoking cessation andfire protection. Interventions ranged from those thatare cost-saving to ones costing $4.9 billion per life-yearsaved, with a median of $19,500 per life-year saved(1993 USD). Graham et al.5 reported on the cost-effectiveness of 40 interventions in the United States,including cancer, coronary heart disease and injuryprevention interventions. Cost-effectiveness ratiosranged from those costing less than $1,000 per quality-adjusted life-year (QALY) to ones costing $1 million(1995 USD) per QALY. Most of the cost-effectivenessratios reviewed were clustered in the range of $10,000to $100,000 per QALY.

In the process of providing the public health com-munity with lists of the cost-effectiveness of variousinterventions expressed in consistent units, the previ-ously mentioned reviews (and other reviews concentrat-ing on a more restricted set of health conditions6–8),have also provided the first steps toward the standard-ization of systematic reviews of published economicevaluations. In addition to published lists of cost-effec-tiveness ratios, these studies have reported general

inclusion and adjustment criteria. However, they usu-ally do not provide an explicit or full description of theprocedure used to adjust from the results of the pri-mary studies to the results they report, nor do theyprovide the instruments used in the abstraction andadjustment of data. In addition, they leave the compa-rability of the results from various studies in questionbecause they (1) make a limited number of adjustmentsto published ratios; and (2) report point estimates ofcost-effectiveness ratios without sensitivity analysis ofthe adjustments.

Therefore, to our knowledge, standardized instru-ments for systematically translating economic data intocomparable economic information have not beenwidely debated and are not available in the publicdomain.

The Guide is being developed by the Task Force onCommunity Preventive Services (the Task Force) incollaboration with many public and private partnersunder the auspices of the U.S. Department of Healthand Human Services.9 The Guide will include systematicreviews of a variety of issues, including effectiveness,applicability of effectiveness, harm, other positive andnegative effects, barriers to intervention implementa-tion, and economic evaluations of selected interven-tions for which evidence of effectiveness is strong orsufficient based on explicit criteria. Standardized meth-ods and instruments for searching, including and ab-stracting studies of effectiveness were developed for theGuide10 to reduce inconsistencies within and amongchapters. A similar process was undertaken to developmethods and instruments for the systematic review ofeconomic evaluations with the purpose of reducingerror and bias in the abstraction and adjustment ofresults and making them comparable acrossinterventions.

Methods

The methods for reviewing economic evaluationsinvolve (1) applying explicit criteria for decidingwhich evaluation studies were to be included in thereviews; (2) using a standard abstraction form torecord individual study characteristics, abstract data,and adjust reported results; and (3) interpreting andsummarizing economic information from relatedstudies of each effective intervention assessed.

Finding and Selecting Relevant Studies

For each chapter of the Guide, candidate economicstudies are identified by systematic searches of theliterature. A systematic search requires study inclusioncriteria. Systematic reviews of economic evaluations inthe Guide will, as much as possible, use consistentcriteria across chapters (Table 1). Each chapter alsorequires specific inclusion criteria relevant to the inter-

76 American Journal of Preventive Medicine, Volume 18, Number 1S

ventions reviewed in that chapter. For example, for theReviews of Evidence for Interventions to Improve Vac-cination Coverage in Children, Adolescents andAdults,12 a study was eligible if it met the standardcriteria and addressed selected interventions to im-prove coverage levels for universally recommendedvaccinations for children, adolescents, and adults.a

Accounting for MethodologicalDifferences in Primary Studies

A key decision in the design of the review process waswhether or not to devise a method for scoring thequality of the study as was done for the review ofeffectiveness literature in the Guide.10 A quality scoringwas not done for several reasons. First, although somedifferences in methods in economic studies couldreasonably be labeled as differences in quality, manydifferences in methods are attributable to differentobjectives or reasonable differences in opinion regard-ing the most appropriate methods. Second, even incases where differences in methods could be reasonablylabeled as quality related, the results of the study mightstill be informative in cases where the results indicatelarge cost-savings or very high cost per health benefit.Third, differences in methods affect the results of somestudies more than others. For example, the differencebetween the use of 5% and 10% discount rates in astudy with mostly short-term costs and benefits mightbe negligible, whereas the difference would be substan-tial in studies where most costs accrue many years in thefuture. Finally, the number of economic studies avail-able from which to select is extremely limited for most

community preventive services. Therefore, the TaskForce and chapter development teams decided to in-clude the studies that met the inclusion criteria and todevelop a method for adjusting the results of studiesrather than excluding some studies because of qualitylimitations. Each study is abstracted by two reviewers tofacilitate more thorough and balanced reviews. Differ-ences of opinion about the adjustments are resolved byconsensus.

Abstracting Economic EvaluationResults from Primary Studies

The second step in performing reviews is abstractingresults from included studies and adjusting them toimprove comparability across studies. The abstractionform provides an explicit instrument for performing thesetasks. The major sections of the form are summarized inTable 2; selected sections of the form are presented inAppendix A. The entire form including instructions canbe obtained from the corresponding author or from theGuide’s Internet home page at the following address:http://www.web.health.gov/communityguide.

Essential information including intervention defini-tion, methods and data sources of the primary study isrecorded in Sections I–IV of the abstraction form.Section I records information about the topic andintervention under consideration in the Guide and thetype of analytic method and summary measure re-ported in the study. Section II compares the interven-tion, as defined in the Guide, with the interventiondescribed in the study. Section III assists the reviewer inlocating the main elements of analysis in the article andSection IV allows descriptive study information such asperspective, comparator, analytic horizon, data sourcesand type of costs to be recorded. Sections V–VII dealwith adjustments to results of economic evaluation andare discussed below.

Adjusting Results of Primary Studies

Published results are adjusted to reflect what the resultsof the study would have been had the study followed achosen set of standards. Adjustments are made sequen-tially. The reference case of the Panel on Cost Effec-tiveness in Health and Medicine (PCEHM)13 is thestandard with which cost-effectiveness and cost-utilitystudies are compared. The PCEHM reference caseprovides an explicit and well-justified set of rules forconducting and reporting cost-effectiveness studies in amanner that allows the results of different studies to becompared with one another. Although perhaps not theoriginal intent of the PCEHM, the reference case alsoprovides a reasonable set of rules with which to stan-dardize the results of existing cost-effectiveness studies.For cost-analysis and cost-benefit studies, all standardspertaining to costs in the reference case also apply. The

a For example, measles, mumps, and rubella vaccinations are recom-mended for young children; hepatitis B vaccinations are recom-mended for adolescents; and annual influenza vaccinations arerecommended for adults aged $65.

Table 1. Standard inclusion criteria for economicevaluation studies for the Guide to Community PreventiveServices

Criteria Description

A Evaluated interventions that the Task Force hasalready classified as “strongly recommended”or “recommended”

B Used an analytical method (e.g., cost-analysis,cost-effectiveness, cost-benefit, or cost-utility)

C Provided sufficient detail regarding methodsand results to enable use and adjustment ofthe study’s data and results

D Was a primary study rather than, for example, aguideline or review

E Had a publication date appropriate to theparticular subject matter

F Was conducted in one or more EstablishedMarket Economies (EME) as defined by theWorld Bank11

G Was written in English

Am J Prev Med 2000;18(1S) 77

choice of the PCEHM reference case does not meanthat the Task Force necessarily believes the correspond-ing methods are better than alternative methods. Thischoice was made because the reference case provides aconvenient, thoroughly considered and generally ac-cepted standard for conducting and reporting cost-effectiveness studies. Of note is the fact that thePCEHM does not recommend these methods as thesole means of conducting cost-effectiveness studies.Rather, the PCEHM recommends the reporting of thereference case, when possible, to facilitate comparisonsamong studies.

Section V of the abstraction form enables the re-viewer to adjust the costs and health outcome measuresreported in the study. The first set of adjustments areintended to convert data expressed in various curren-cies and base years to USD in a consistent base year.First, costs expressed in a foreign currency are con-verted to USD for the base year reported in the studyusing purchasing power parity rates. Purchasing powerparity rates are estimates of the exchange rate at whichan equivalent amount of goods or services could bepurchased in two different countries, regardless of thecurrency being cited. Purchasing power parity rates areused for this purpose because they are less susceptibleto financial flows and governmental exchange ratemanipulation than are market exchange rates. Al-though purchasing power parity rates are less problem-atic than market exchange rates in converting the valueof resources used from one currency to another, con-versions can be inaccurate.14 Therefore, studies that arenot based on US prices and not reported in USD areconsidered only when similar USD-denominated stud-ies are not available.

Second, costs are adjusted to the 1997 base year usingeither the consumer price index (CPI) or the medicalconsumer price index (MCPI).15 Although the choice

of price index might not be particularly important forstudies that report results in a base year one or twoyears before 1997, it can be very important for studiespublished earlier. For example, for studies reported in1985 USD, the use of the CPI to convert to 1997 USDwould inflate the numerator of a cost-effectiveness ratioby 49%, whereas the use of the MCPI would inflate thenumerator by 106%. Therefore, we developed rules(Table 3) on when to apply the CPI or MCPI ratherthan relying on the use of one index in all cases.

Any rule for adjusting the value of the summarymeasureb is inherently inferior to re-estimation of anew value. For example, even if enough detail werepresented in a study to allow the adjustment of individ-ual prices by respective inflation rates, it is very unlikelythat the weighted average prices of the interventionwould increase at the estimated rate of inflation for thegeneral economy (as would be required to accuratelyupdate prices using the CPI) or that the weightedaverage of medical prices would increase at the esti-mated rate of medical inflation. Though imperfect, webelieve these rules are an improvement on standardiza-tion approaches using the same price index to adjust allcost-effectiveness ratios because they risk large errors inupdating studies.

After base year, costs are adjusted for discount rate.The rate used to discount costs and health outcomesaffects the value of the summary measure. The discountrate adjustment is based on the reference case of thePCEHM.13 The recommended discount rate is 3% forboth effectiveness and costs. The discount rate adjust-ment will be possible if the timing and amount offuture intervention costs, costs of illness and healthbenefits are known or can be approximated from data

b Summary measure refers to cost, net cost, cost-utility or cost-effectiveness ratio.

Table 2. Elements of the abstraction form used for reviewing economic evaluation studies selected for the Guide toCommunity Preventive Services

Section Purpose Completed by

I. Classificationinformation

Record tracking information, analytic method, and summarymeasure used in study

Guide staff and checkedby reviewers

II. Comparisonintervention

Record differences between intervention reviewed in the Guideand intervention analyzed in the study


III. Identifyinginformation

Identify key result from study and key data elements reported instudy


IV. Study information Record detailed study information, including location, audience,setting, perspective, data sources, etc.

Reviewers

V. Costs and healthoutcome measureadjustments

Adjust key result to 1997 USD, adjust for discount rate, add orremove costs, remove productivity losses, adjust health outcomemeasure, and assess uncertainty

Reviewers

VI. Nonadjustabledifferences

Summarize differences between study methods and standardmethods for which the study’s key result cannot be adjusted

Reviewers

VII. Applicability Summarize key parameters that might affect the interpretation ofthe results if intervention is applied in other settings

Reviewers

VIII. Summary table Condense findings of review and allow for revision of abstractedinformation after detailed review of study

Reviewers


provided in the study. If so, the full abstraction formprovides a table listing the effect of discounting oneunit (whether 1 USD or one QALY) over various timeperiods for several common discount rates (0%, 3%,5%, 7% and 10%).

The next set of adjustments ensures, to the extentpossible, that the costs and effects included in theanalysis are consistent from study to study. The addi-tional costs sections provide entry for those administra-tive, clinical preventive services, follow-up or patienttime and transportation costs missing from the analysis.The societal perspective requires that all costs andbenefits be considered, regardless of who bears thecosts or receives the benefits. In some instances, costestimates used in the study might not represent thesetting. For example, authors sometimes intentionallyuse high or low costs to obtain “conservative” estimatesof the summary measure. The PCEHM reference casedoes not include a valuation of time lost as a result ofchronic illness, long-term disability or death in thenumerator (“productivity”) to avoid double counting ofquality of life which is already accounted for in thedenominator. Many studies report results with and

without this valuation of time, therefore allowing pro-ductivity to be subtracted.

The conversion of the health outcome measure toQALYs addresses the need to improve comparability ofresults by using a common metric for health outcome.It is difficult, for example, to compare the cost-effec-tiveness of different studies in which results are ex-pressed as dollars per case of disease prevented, dollarsper life-year saved or dollars per QALY saved. Infollowing the PCEHM reference case whenever possi-ble, we use QALYs for this metric. QALYs capturemortality in terms of life-years lost and capture morbid-ity as a product of years in a reduced health state timesthe value of that reduction from full health. The healtheffect measure adjustment will not be done for thosecases where the study considered only intermediateoutcomes or other health outcome measures (e.g.,disability-adjusted life years [DALYs]). In the first case,an intermediate summary measure will be chosen de-pending on the chapter under consideration. In thesecond case, the summary measure will be expressed inUSD per DALY as originally reported in the article andreported in a separate table.

Table 3. Criteria for base-year adjustments for the Guide to Community Preventive Services

Rules Condition Index Justification

1 If both program costs and costs averteda areprimarily non-medical, or if both programcosts and costs averted are a mixture ofmedical and non-medical resources

CPI Reflects the fact that the CPI is already partiallyweighted by estimates of medical careinflation

2A If program costs are primarily non-medical,cost averted are primarily medical, and netcosts are negative (cost-savingintervention)

MCPI Although a mixture of costs exists, the negativenet costs indicate that the medical prices inthe costs averted are likely to play a largerrole in changes to the net costs over time

2B If program costs are primarily non-medical,cost averted are primarily medical, netcosts are positive, and the reviewer candetermine that the ratio of costs averted toprogram costs is ,0.25a

CPI General price inflation is likely to be a moreimportant factor in changes of the net costsover time

2C If program costs are primarily medical, netcosts are positive, and the reviewer candetermine that the ratio of costs averted toprogram costs is .0.75a

MCPIb Medical costs are playing a substantial role, andnet costs are likely to be decreasing over timerather than increasing because medicalinflation is historically greater than generalinflation

2D If program costs are primarily non-medical,costs averted are primarily medical, netcosts are positive, and (a) the ratio of costsaverted to costs incurred falls between 0.25and 0.75a or (b) the ratio cannot bedetermined with a reasonable amount ofcertainty

No adjustment Ratio is either relatively stable over time or thedirection in which the adjustment should bemade is too unpredictable

a “Program costs” (CP) include all positive costs due to the intervention and “costs averted” (CA) include all saved costs resulting from theintervention. When CA are considerably smaller than CP, the CA/CP ratio is relatively small (, 0.25), and the percent change in net costs (CP2 CA) over time due to inflation approaches the economy-wide inflation rate. When CA approaches the magnitude of CP, the CA/CP ratio isrelatively large (. 0.75) and the percent change in net costs over time due to inflation approaches negative infinity. The percent change in netcosts will be approximately equal to the medical inflation rate when the CA/CP ratio equals 0.75. Reviewers are cautioned that the actual percentchange in net cost may be significantly larger (in absolute value) than the medical inflation rate in cases where rule 2C applies and that inflationmay cause net costs to decrease to zero and become negative. When the CA/CP ratio equals 0.5, the percent change in net costs over time dueto inflation is approximately zero. Using either rule 2B or rule 2C (adjusting for inflation with either the CPI or MCPI) is not likely to improvethe estimate of net costs when CA/CP is between 0.25 and 0.75.b Ratio or net costs should be decreased using the MCPI rather than increased.

Am J Prev Med 2000;18(1S) 79

The results of studies addressing a service which pre-vent significant morbidity and express results in terms ofdollars per life-year saved must be adjusted to betterreflect the total health benefits of the service. In mostcases, these adjustments are made by scaling time spent inillness or disability to 70% of the value of time spenthealthy. With this approximation, each year of illness ordisability prevented by this service reduces 0.3 years ofquality of life for most chronic and acute diseases ordisabling conditions. The estimate of 0.3 is the differencebetween perfect health, valued at 1.0 on health utilityscales, and a range of approximately 0.55 to 0.85, (mid-point, 0.7) for persons living with most chronic and acutediseases or disabling conditions as reported in publishedutility scales16,17 (12 0.7 5 0.3).c

Although making adjustments with QALY weightsthat vary according to the specific condition would bepreferable, the simple rule of 0.3 QALYs lost annuallyfor most conditions was adopted because no publishedscales include all conditions and health states that arelikely to be needed in a cross-cutting review of preven-tive services. Regardless of the weights chosen, sensitiv-ity analysis on the weights used to adjust health benefitsis needed because no single means of defining weightsis clearly more correct that others. For many condi-tions, mortality tends to dominate the estimate of QALYand the adjustment will not be sensitive to the QALYweight for morbidity.

Table 4 contains an illustration of how the healthoutcome measure adjustment might be made. Thereviewer can use a higher weight if the 0.3 value isinsufficient to represent the averted impact of majormorbidities such as mental impairment, severe mentalillness or blindness. If an alternative weight is used, the

reviewer states the reason on the form. Adjustmentsshould be made whenever possible and in fact, approx-imations with wide ranges in sensitivity analysis arepreferable to determining that differences are nonad-justable. However, reviewers are asked to record differ-ences for which adjustments are not possible in SectionVI of the abstraction form, “Nonadjustable Differenc-es.” This section addresses differences in effect size,characteristics of the target population and frequencyand intensity of delivery of the intervention evaluatedin the study under review and the “typical” interventionclass evaluated in the Guide review.

Improving Comparability andConsistency of Base Adjustments

The adjustments noted previously are only as precise asthe underlying data and the reporting of the originalstudy allow. However, three additional steps are takento improve the quality of reviews. First, the adjustmentsare subjected to sensitivity analysis. Under the section“Predicted Degree of Error (Uncertainty of Adjust-ment),” reviewers are asked to assess the reliability ofindividual adjustments using alternate values for eachof the adjustments. This assessment is virtually identicalto a single-variable sensitivity analysis on the finaladjusted value of the summary measure. Based uponthis sensitivity analysis, reviewers are asked to determineto which of four predefined categories of uncertaintyeach adjustment belongs. The ranges of uncertaintyassociated with the categories (60%–10%, 11%–30%,31%–50% and .50%) are defined as a percentage ofthe final adjusted ratio, rather than as a percent of theratio resulting from each individual adjustment, so thatthe sensitivity of each adjustment is evaluated on acommon basis and the range of uncertainty has under-lying meaning. No theoretical or empirical basis is

c Based on the fact that when mortality is measured, years of life arevalued at 1, not at 0.9 (the rough average reported for those with nodiseases or disabilities).

Table 4. Example of a cost-effectiveness ratio conversion from dollars per year of life saveda to dollars per quality adjustedlife year (QALY) for an immunization program to immunize infants against Hepatitis B

Item Parameter Source Valuea

A Cost of program for infants, 1993 USD, millions Table 4, chronic, medical costs18 47.0B Medical cost without program, 1993 USD, millions Table 4, chronic, medical costs18 9.5C Medical cost without program, 1993 USD, millions Table 4, chronic, medical costs18 3.1D Medical cost averted, 1993 USD, millions (B 2 C) 6.4E Net cost, 1993 USD, millions (A 2 D) 40.6F Net cost, discounted, USD, millions (E) Discounted 50 years at 5% 3.7G Years of life saved, number Table 518 18,879H Years of life saved, discounted, number (G) Discounted 50 Years at 5% 1,699I Chronic infections prevented, number Table 518 4,702J Quality adjusted life years from morbidity, number (I) 3 10 years of chronic infection 3 0.3 QALYs 14,106K Quality adjusted life years from morbidity,

discounted, number(J) Discounted 50 years at 5% 1,270

L Quality adjusted life years from morbidity andmortality, number

(H 1 K) 2,969

M Cost-effectiveness ratio in 1993 dollars per QALYs (F/L) 1,246aRounded to better reflect precision of adjustments.


currently available for defining these ranges; therefore,the ranges were chosen to identify interpretable de-grees of uncertainty.

Second, in Section VI, “Nonadjustable Differences,”reviewers describe differences in methods and defini-tions of interventions where adjustments are not possi-ble because of lack of information. This section allowsthe reviewer to indicate whether the final summarymeasure from Section V might under- or overstate thevalue that would have been observed had the economicmethods in the study not differed from the referencecase or had the specific intervention evaluated notdiffered from a more typical case for the interventionsunder review.

Third, on the basis of their experience in reading thearticle and making adjustments, reviewers identify vari-ables to which the results are particularly sensitive andwhich are likely to change if the intervention is appliedin predictable, but specialized situations. Section VII,“Applicability,” is designed to record information thatcould be used to alert users of the Guide to predictableand substantial changes in the value of the summarymeasure. Variables in this section needing particularattention from reviewers include baseline prevalence(if applicable), population characteristics, effective-ness, size of the population and costs in the case of amore intensive application of an intervention.

Section VIII, “Summary Table,” condenses the find-ings of the study review and adjustments. It is com-pleted only after the full instrument has been com-pleted and allows for revision of abstraction andadjustments. This summary table lists the main compo-nents of the study and the sequential adjustments madeto the original ratio, costs, or cost-saving value. Thistable also facilitates the process of reconciling differ-ences and reaching consensus in the abstraction andadjustments by the two reviewers.

Summarizing Results: Example for theChapter on Vaccine-Preventable Diseases

An excerpt of the economic summary table for one ofthe interventions included in the Reviews of Evidencefor Interventions to Improve Vaccination Coverage inChildren, Adolescents and Adults12 is provided in Ap-pendix B. Additional examples are shown in thatreport. Cost-effectiveness was expressed in this particu-lar case as the cost of the program per additionalvaccination above baseline coverage or the cost perfully vaccinated child. Baseline vaccine coverage is thecoverage that would have occurred in the absence ofthe intervention. Average cost was defined as the cost ofthe program per person served by the program. Aver-age cost is provided to give program planners a startingpoint for estimating the costs of new initiatives. Thecost-effectiveness of interventions to increase vaccina-

tion rates is not summarized in terms of health out-comes (e.g., QALYs) because (1) effectiveness studiesreviewed in the Guide considered only intermediateoutcomes; and (2) the health benefits vary by vaccine.

The table in Appendix B summarizes informationregarding the study and study results including type ofanalytic methods used in the study or used by thereviewer to obtain the summary measure from datareported in the study. It also lists the type of summarymeasure, original currency, costs included in the anal-ysis, the results of the study before and after adjust-ments, characteristics of the study population andestimates of effectiveness used in the evaluation.

Results from the table are interpreted and summa-rized in the body of the Guide for each particularintervention under the “Economic Evidence” subhead-ing. The economic information is available to usersalthough they do not affect the recommendations. Oneof the summaries from the report on vaccine-prevent-able diseases follows:

The search identified 11 economic evaluations ofclient reminder/recall interventions. Nine studiesprovided 12 cost-effectiveness ratios for single-component reminder/recall interventions and3 cost-effectiveness ratios for multicomponent in-terventions that include reminder/recall. Ad-justed cost-effectiveness ratios for the single-com-ponent interventions based on those studiesranged from $3 to $46/additional vaccinationwith a median of $9/additional vaccination. Theadjusted cost-effectiveness ratios for multicompo-nent interventions were $4/additional vaccina-tion for a combination of client and providerreminders; $51/additional vaccination for a com-bination of reminders and a lottery-type incentive;and $43/additional vaccination for a combinationof mailed reminders and free vaccinations. Ad-justed average costs based on two available studiesvaried from $0.65 to $5.75/child. The lowerboundary is an underestimate because the costs ofthe in-kind contribution of volunteer time werenot included and the upper boundary might bean overestimate because it includes costs of clini-cal time to provide vaccinations.

Discussion

At present, the body of economic evidence available tocompare costs and returns of interventions to improvehealth is substantially limited in both quantity andcomparability. Methods to review, abstract and summa-rize economic evaluations need to be developed anddebated if economic information is going to be usefulto managers, policy makers, program planners andresearchers.

The development of the procedures and instrumentsfor economic evaluation in the Guide required balanc-

Am J Prev Med 2000;18(1S) 81

ing several competing needs. The abstraction formneeds to be flexible enough to allow for review ofstudies using various analytic methods (e.g., cost, cost-effectiveness, cost-benefit and cost-utility) and interven-tions. However, the abstraction process must be consis-tent to ensure comparability of data acrossinterventions, control uncertainty and limit the num-ber of subjective judgements which reviewers need tomake.

The Task Force recognizes that no process of adjust-ment or other means of reviewing existing economicevaluations is flawless. However, data that is made morecomparable is thought to be preferable to alternativesthat would (1) ignore economic information entirely;(2) attempt to use noncomparable data; or (3) adjust inways that are not systematic or explicit. These methodswill continue to evolve with time, experience and inputfrom interested readers. Further application and test-ing will determine whether the process performs well ina variety of studies and disease topic areas and whetherthe instrument could be streamlined.

The systematic review of the economics of interven-tions in health care and public health calls for standard-ized and explicit methodology. The efforts of thePCEHM were an important step toward achieving thisgoal. The PCEHM recommends that, in addition toother results, study authors report results on the basisof the reference case whenever possible. However,methods used to compare the large number of previ-ously published studies, which employ disparate meth-odologies, are still needed. Although the comparabilityof published studies may have improved since thepublication of the PCEHM recommendations, methodsfor comparing published studies will continue to beneeded as some study authors face data constraints,funding constraints, editorial constraints and studyobjectives that are partially at odds with the reporting ofreference case results.

The provisional methods presented here representsan initial effort to meet these current and future needsin the Guide to Community Preventive Services. This paperdescribes the methods used to review economic evalu-ations in the Guide with the hope that others (1) willprovide recommendations for improving the methodsfor the Guide; (2) will adapt these methods to the needsof other reviews of economic evaluations; (3) willengage in a process of developing systematic methodsfor evaluating economic evaluations literature thatequals efforts to develop methods for the systematicreview of effectiveness literature; and (4) will encour-

age economic evaluation studies of interventions forwhich economic information is not available.

The authors are particularly grateful to Marthe R. Gold MD,MPH, C. Kay Smith-Akin, MEd, Robert Deuson, PhD, ReginaPana-Cryan, PhD, and Scott Grosse PhD for their contribu-tions and advice in the development of the economic evalu-ation abstraction form.

References1. US Preventive Services Task Force. Guide to clinical preventive services.

2nd ed. Baltimore: Williams & Wilkins, 1996.2. Kassirer JP, Angell M. The Journal’s policy on cost-effectiveness analyses.

N Engl J Med 1994;331:669–70.3. Tengs TO, Adams ME, Pliskin JS, et al. Five hundred life-saving interven-

tions and their cost-effectiveness. Risk Anal 1995;15(3):369–90.4. Ramsberg JAL, Sjoberg L. The cost-effectiveness of life-saving interventions

in Sweden. Risk Anal 1997;17:467–78.5. Graham JD, Corso PS, Morris JM, Segui-Gomez M, Weinstein MC. Evalu-

ating the cost-effectiveness of clinical and public health measures. Ann RevPub Health 1998;18:125–53.

6. Schrappe M, Lauterbach K. Systematic review on the cost-effectiveness ofpublic health interventions for HIV prevention in industrialized countries.AIDS 1998;12(suppl):231–7.

7. van der Weijden T, Knottnerus JA, Ament AJ, Stoffers HE, Grol RP.Economic evaluation of cholesterol-related interventions in general prac-tice: an appraisal of the evidence. J Epidemiol Community Health 1998;52:586–94.

8. Brown ML, Fintor L. Cost-effectiveness of breast cancer screening: prelim-inary results of a systematic review of the literature. Breast Cancer Res Treat1993;25:113–8.

9. Pappaioanou M, Evans C. Development of the Guide to CommunityPreventive Services: a US Public Health Service initiative. J Public HealthMgmt Practice 1998;4:48–54.

10. Zaza S, Wright-De Aguero LK, Briss PA, et al. and the Task Force onCommunity Preventive Services. Data collection instrument and procedurefor systematic reviews in the Guide to Community Preventive Services. Am JPrev Med 2000;18(suppl 1):44–74.

11. Murray CJL, Lopez AD. Global health statistics. Boston: Harvard UniversityPress, 1996.

12. Briss PA, Rodewald L, Hinman A, et al. and the Task Force on CommunityPreventive Services. Reviews of evidence for interventions to improvevaccination coverage in children, adolescents and adults. Am J Prev Med2000;18(suppl 1):97–140.

13. Gold M, Siegel JE, Russell LB, Weinstein MC, eds. Cost-effectiveness inhealth and medicine. New York: Oxford University Press, 1996: AppendixA304–11.

14. Kanavos P, Mossialos E. International comparisons of health care expen-ditures: what we do know and what we do not know. J Health Services ResPolicy 1999;4:122–6.

15. Statistical Abstract of the United States. Washington, DC: US Bureau of theCensus, 1997:492–510.

16. Fryback DG, Dasbach EJ, Klein R, Klein BE, Dorn N, et al. The Beaver Damhealth outcomes study: initial catalog of health-state quality factors. MedDecis Making 1993;13:89–102.

17. Gold MR, Franks P, McCoy KI, Fryback DG. Toward consistency incost-utility analyses: using national measures to create condition-specificvalues. Med Care 1998;36:778–92.

18. Margolis HS, Coleman PJ, Brown Re, Mast EE, Sheingold SH, Arevalo JA.Prevention of hepatitis B virus transmission by immunization: an economicanalysis of current recommendations. JAMA 1995;274:778–92.


Appendix ASelected Sections of the Economic Evaluation Form

Am J Prev Med 2000;18(1S) 83


Am J Prev Med 2000;18(1S) 85


Am J Prev Med 2000;18(1S) 87

Appe

ndix

BSa

mpl

eEc

onom

icEv

alua

tion

Sum

mar

yTa

ble


Am J Prev Med 2000;18(1S) 89


Refe

renc

esto

Appe

ndix

B1.

Buc

hn

erD

M,

Lar

son

EB

,W

hit

eR

F.In

flue

nza

vacc

inat

ion

inco

mm

unit

yel

derl

y:a

con

trol

led

tria

lof

post

card

rem

inde

rs.

JA

mG

eria

trSo

c19

87;3

5:75

5–60

.2.

Ch

iuT

T,

Bar

ata

SL,

Un

sick

erD

M,

Bre

nn

anL

.C

omm

unit

ym

obili

zati

onfo

rpr

esch

ool

imm

uniz

atio

ns:

the

“Sh

ots

byT

wo”

proj

ect.

Am

JPu

blic

Hea

lth

1997

;87:

462–

3.3.

Fram

eP,

Zim

mer

J,W

erth

P,H

all

J,E

berl

yS.

Com

pute

rize

d-ba

sed

vs.

man

ual

hea

lth

mai

nte

nan

cetr

acki

ng.

Arc

hFa

mily

Med

1994

;3:5

81–

8.4.

Fran

kJW

,M

cMur

ray

L,

Hen

ders

onM

.In

flue

nza

vacc

inat

ion

inth

eel

derl

y:th

eec

onom

ics

ofse

ndi

ng

rem

inde

rle

tter

s.C

anM

edA

ssoc

J19

92;1

32:5

16–

8.5.

Hut

chis

onB

G,

Shan

non

HS.

Eff

ect

ofre

peat

edan

nua

lre

min

der

lett

ers

onin

flue

nza

imm

uniz

atio

nam

ong

elde

rly

pati

ents

.J

Fam

ilyPr

acti

ce19

91;3

3:18

7–9.

6.G

rabe

nst

ein

JD,

Har

tzem

aA

G,

Gue

ssH

A,

Joh

nst

onW

P,R

itte

nh

ouse

BE

.C

omm

unit

yph

arm

acis

tsas

imm

uniz

atio

nad

voca

tes.

Cos

t-eff

ecti

ven

ess

ofa

cue

toin

flue

nza

vacc

inat

ion

.M

edC

are

1992

;30:

503–

13.

7.L

ieu

TA

,B

lack

SB,

Ray

P,et

al.

Com

pute

r-ge

ner

ated

reca

llle

tter

sfo

run

deri

mm

uniz

edch

ildre

n:

how

cost

-eff

ecti

ve?

Pedi

atr

Infe

cD

isJ

1997

;16:

28–3

3.8.

Lie

uT

A,C

apra

AM

,Mak

olJ,

Bla

ckSB

,Sh

inefi

eld

HR

.Eff

ecti

ven

ess

and

cost

-eff

ecti

ven

ess

ofle

tter

s,au

tom

ated

tele

phon

em

essa

ges,

orbo

thfo

run

deri

mm

uniz

edch

ildre

nin

ah

ealt

hm

ain

ten

ance

orga

niz

atio

n.P

edia

tric

s19

98;1

01:E

3.9.

McL

eod

D,

Bow

ieR

D,

Klja

kovi

cM

.C

ost

ofch

ildh

ood

imm

uniz

atio

nin

gen

eral

prac

tice

.N

ZM

edJ

1998

;111

:73–

6.10

.M

oran

WP,

Nel

son

K,

Wof

ford

JL,

Vel

ezR

,C

ase

LD

.In

crea

sin

gin

flue

nza

imm

uniz

atio

nam

ong

hig

h-r

isk

pati

ents

:ed

ucat

ion

orfi

nan

cial

ince

nti

ve.

Am

JM

ed19

96;1

01:6

12–

62.

11.

Nex

oeJ,

Kra

gstr

upJ,

Ron

ne

T.

Impa

ctof

post

alin

vita

tion

san

dus

erfe

eon

infl

uen

zava

ccin

atio

nra

tes

amon

gth

eel

derl

y.A

ran

dom

ized

con

trol

led

tria

lin

gen

eral

prac

tice

.Sc

and

JPr

imar

yH

ealt

hC

are

1997

;15:

109

–12.

12.

Ros

ser

WW

,H

utch

ison

BG

,M

cDow

ell

I,N

ewel

lC

.U

seof

rem

inde

rsto

incr

ease

com

plia

nce

wit

hte

tan

usbo

oste

rva

ccin

atio

n.

Can

Med

Ass

ocJ

1992

;146

:911

–17.

13.

McD

owel

lI,

New

ell

C,

Ros

ser

W.

Com

pari

son

ofth

ree

met

hod

sof

reca

llin

gpa

tien

tsfo

rin

flue

nza

vacc

inat

ion

.C

anM

edA

ssoc

J19

86;1

35:9

91–7

.

Am J Prev Med 2000;18(1S) 91

Reprinted by permission of Elsevier Science from: Methods for Systematic Reviews of Economic Evaluations for the Guide to Community Preventive Services. Vilma Carande-Kulis, Michael V. Maciosek, Peter A. Briss et al., American Journal of Preventive Medicine, Vol 18 No 1S, pp 75-91, Copyright 2000 by American Journal of Preventive Medicine

Documents

Methods for Systematic Reviews of Economic Evaluations for ...€¦ · Methods: The following steps were taken for systematic reviews of economic evaluations: (1) systematic searches