R

MCC

FHa

b

c

d

a

ARRAA

KCEEIPV

C

JM

0d

Vaccine 29 (2011) 9171– 9176

Contents lists available at ScienceDirect

Vaccine

jou rn al h om epa ge: www.elsev ier .com/ locate /vacc ine

eview

ethods for developing evidence-based recommendations by the Advisoryommittee on Immunization Practices (ACIP) of the U.S. Centers for Diseaseontrol and Prevention (CDC)

aruque Ahmeda,∗, Jonathan L. Temteb, Doug Campos-Outcalt c,olger J. Schünemannd, for the ACIP Evidence Based Recommendations Work Group (EBRWG)1

Immunization Services Division, Centers for Disease Control and Prevention, 1600 Clifton Rd. NE, Mailstop A-19, Atlanta, GA 30333, USADepartment of Family Medicine, University of Wisconsin School of Medicine and Public Health, 777 South Mills Street, Madison, WI 53715, USADepartment of Family & Community Medicine, University of Arizona College of Medicine, 550 E. Van Buren Street, Phoenix, AZ 85004, USADepartment of Clinical Epidemiology and Biostatistics, McMaster University Health Sciences Centre, Room 2C10B, 1200 Main Street West, Hamilton, ON L8N 3Z5, Canada

r t i c l e i n f o

rticle history:eceived 12 April 2011eceived in revised form 28 July 2011ccepted 1 August 2011vailable online 11 August 2011

eywords:enters for Disease Control and Preventionvidence-based medicine

a b s t r a c t

The Advisory Committee on Immunization Practices (ACIP) provides expert external advice and guidanceto the Director of the Centers for Disease Control and Prevention and the Secretary of the U.S. Departmentof Health and Human Services on use of vaccines and related agents for control of vaccine-preventabledisease in the United States. During the October 2010 ACIP meeting, the ACIP voted to adopt a newframework for developing evidence-based recommendations that is based on the Grading of Recom-mendations, Assessment, Development and Evaluation (GRADE) approach. Key factors considered in thedevelopment of recommendations include the balance of benefits and harms, type of evidence, valuesand preferences of the people affected, and health economic analyses. Category A recommendations will

vidence-based practicemmunizationreventive health servicesaccination

be made for all persons in an age- or risk-factor-based group. Category B recommendations will be madefor individual clinical decision making; category B recommendations do not apply to all members of anage- or risk-factor-based group, but in the context of a clinician–patient interaction, vaccination may befound to be appropriate for a person. Evidence tables will be used to summarize the benefits and harmsand the strengths and limitations of the body of evidence. The new evidence framework will enhance theACIP’s decision-making process by making it more transparent, consistent and systematic.

Published by Elsevier Ltd.

ontents

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91722. Guiding principles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9172

2.1. Focus on transparency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91722.2. Use of evidence of varying strengths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91722.3. Individual and community health considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91722.4. Availability of existing evidence evaluation systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9172

3. Framework for developing evidence-based recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91723.1. Overview of GRADE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91723.2. New ACIP framework for evaluating evidence and developing recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91733.3. Formulating recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9173

3.4. Balance of benefits and harms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3.5. Type or quality of evidence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.6. Indirect evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

∗ Corresponding author. Tel.: +1 404 639 8827; fax: +1 404 639 8626.E-mail address: fahmed@cdc.gov (F. Ahmed).

1 Members of the EBRWG include Jonathan Temte, MD, PhD (chair), Faruque Ahmed, Pane Gidudu, MD, MPH, Jiangcheng Huang, MD, MS, Gail Janes, PhD, MS, Linda Kinsinger,

D, MPH, Amir Qaseem, MD, PhD, MHA, William Schaffner, MD, Holger J. Schünemann, M

264-410X/$ – see front matter. Published by Elsevier Ltd.oi:10.1016/j.vaccine.2011.08.005

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9173. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9173. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9174

hD, Robert L. Beck, JD, Ned Calonge, MD, MPH, Doug Campos-Outcalt, MD, MPA,MD, MPH, Joanne M. Langley, MD, Edgar K. Marcuse, MD, MPH, Virginia Moyer,

D, PhD, MSc, FRCP(C), Jean Clare Smith, MD, MPH, and Litjen (LJ) Tan, PhD, MS.

9

1

eStt(psevnrd[

MnPtefana

Bdteo(gA

2

euveo

2

dtsps

172 F. Ahmed et al. / Vaccine 29 (2011) 9171– 9176

3.7. Overall type of evidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91743.8. Values and preferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91743.9. Health economic data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91743.10. Evidence tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91753.11. Format for presenting recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9175

4. Identifying population subgroups at higher risk of disease . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91755. Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9175

Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9176References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9176

. Introduction

The Advisory Committee on Immunization Practices (ACIP),stablished in 1964 by the Surgeon General of the U.S. Public Healthervice, provides expert external advice and guidance to the Direc-or of the Centers for Disease Control and Prevention (CDC) andhe Secretary of the U.S. Department of Health and Human ServicesDHHS) on use of vaccines and related agents for control of vaccine-reventable disease in the U.S. civilian population. The ACIP’structure, role, and procedures are described elsewhere [1–3]. Keylements for making ACIP recommendations include vaccine safety,accine efficacy/effectiveness, and burden of disease. Health eco-omic data and other factors are also considered when making aecommendation. These key elements play an important role in theevelopment of vaccination policies in a number of other countries4–8].

ACIP recommendations are published in CDC’s Morbidity andortality Weekly Report and are summarized in the form of Immu-

ization Schedules for children, adolescents, and adults [9]. Healthyeople, which provides 10-year national objectives for improvinghe health of all Americans, includes objectives for preventing dis-ase, disability, and death from infectious diseases as well as targetsor vaccine coverage levels [10]. Data from surveillance programsnd surveys are used to monitor vaccination rates, impact of vacci-ation on morbidity and mortality, and safety of vaccines after theyre licensed [9].

Responding to the review and synthesis of the ACIP’s Evidenceased Recommendations Work Group (EBRWG), the ACIP voteduring the October 2010 meeting to recommend to CDC/DHHShe adoption by the ACIP of a new framework for developingvidence-based recommendations that is based on the Gradingf Recommendations Assessment, Development and EvaluationGRADE) approach [11]. In this article, we present the EBRWG’suiding principles and the new framework that will be used by theCIP.

. Guiding principles

The EBRWG’s guiding principles in the development of anxplicit evidence-based framework include: focus on transparency;se of evidence of varying strengths; consideration of both indi-idual and community health; adoption or adaption of an existingvidence-based system; and a need for continuous improvementf the process.

.1. Focus on transparency

The diverse nature of the elements within ACIP recommen-ations and the diverse nature of ACIP stakeholders require that

2.2. Use of evidence of varying strengths

The types of evidence available will vary based on the type ofquestion and the context. For example, the types of evidence avail-able or obtainable in an emergency or emerging situation may bequite limited. At times, it may be unethical, impossible, or not fea-sible to conduct randomized controlled trials. Recommendationsshould be based on the best available evidence.

2.3. Individual and community health considerations

Vaccine efficacy/effectiveness and safety can apply for boththe individual and community health perspectives. From theindividual-health perspective, efficacy/effectiveness and safetyrefer to health outcomes among vaccinated persons. Communityhealth refers to the impact of immunization on the community as awhole, including unvaccinated persons. Examples of communityhealth benefits of immunization include interrupting transmis-sion of an infectious disease, achieving herd immunity, creatinga “cocoon” effect (vaccinating close contacts of a vulnerable unvac-cinated person), decreasing outbreaks, and eliminating disease[12,13]. Safety considerations include the possibility of illness inimmunocompromised contacts as a result of transmission of liveattenuated vaccine-strain agents, or a potential increase in dis-ease caused by non-vaccine serotypes (e.g., serotype replacementassociated with the use of pneumococcal conjugate vaccine) [12].Other potential harms might include reduced emphasis on otherpreventive interventions due to emphasis on only immunization.

2.4. Availability of existing evidence evaluation systems

The EBRWG reviewed several evidence evaluation systems[11,14–19]. Because evidence evaluation systems exist in states oftransition or change to improve and refine their methodology, theACIP system must be presented as a “work in progress.” The intentis to continually strive to improve the process.

3. Framework for developing evidence-basedrecommendations

The ACIP unanimously voted to adopt a framework for develop-ing evidence-based recommendations that is based on the GRADEapproach. The GRADE approach has been incorporated into theevidence-based systems used by a number of organizations, includ-ing the American Academy of Family Physicians, American Collegeof Physicians, American College of Chest Physicians, AmericanEndocrine Society, American Thoracic Society, Infectious DiseasesSociety of America, UpToDate, Agency for Healthcare Research andQuality, Cochrane Collaboration, and the World Health Organiza-tion [11,20–22].

he methodology used to reach recommendations be explicitlytated and transparent. The approach must be sufficiently trans-arent to merit the confidence of the greatest number of ACIPtakeholders.

3.1. Overview of GRADE

The GRADE approach is based on a sequential assessment ofthe quality of evidence (high, moderate, low, very low), followed

cine 29

bid[memsttooib

3r

ca

•

•

•

rfa

rm

(

(

(

(

RcillRfiupap

3

oami

F. Ahmed et al. / Vac

y judgment about the balance between desirable (benefits, sav-ngs) and undesirable (harms, costs) effects, and subsequentecision about the strength of a recommendation (strong, weak)11,21–23]. Key considerations in the formulation of recom-

endations include: balance of benefits and harms; quality ofvidence; values and preferences; and costs. Separating the judg-ents regarding the quality of evidence from judgments about the

trength of recommendations is a critical and defining feature ofhe GRADE system. The steps include formulating specific ques-ions to be answered by a recommendation, identifying importantutcomes for every question, summarizing evidence for importantutcomes, assessing quality of evidence for each outcome, assess-ng underlying values related to outcomes, judging the balance ofenefits and harms, and formulating recommendations.

.2. New ACIP framework for evaluating evidence and developingecommendations

Labeling vaccine recommendations as strong or weak was notonsidered to be appropriate. The ACIP recommendation categoriesre:

Category A: Recommendation that applies to all persons in an age-or risk-based group.Category B: Recommendation for individual clinical decision mak-ing.No recommendation/unresolved issue.

The suggested wordings for category A recommendations areecommend, recommend against, should, and should not; and thoseor category B recommendations are words like may and suggestgainst.

The body of evidence will be categorized into four types thatepresent a general hierarchy reflecting the confidence in the esti-ated effect of vaccination on health outcomes:

1) Randomized controlled trials (RCTs), or overwhelming evi-dence from observational studies.

2) Randomized controlled trials with important limitations, orexceptionally strong evidence from observational studies.

3) Observational studies, or randomized controlled trials withnotable limitations.

4) Clinical experience and observations, observational studieswith important limitations, or randomized controlled trialswith several major limitations.

Randomization minimizes potential bias and confounding, andCTs are considered the gold standard for assessing vaccine effi-acy. However, observational studies may provide more relevantnformation than RCTs in certain situations (e.g., for assessingong-term health outcomes or outcomes that are rare) [24]. Aftericensure of a vaccine, it may not be feasible or ethical to conductCTs to assess the effect of vaccination on health outcomes (bene-ts or possible harms), and such assessments are usually conductedsing observational studies [25]. Observational studies, such ashase 4 or post-marketing studies, can provide useful informationbout the effect of vaccination under the conditions of everydayractice in large populations.

.3. Formulating recommendations

Category A recommendations will apply to all persons in an age-

r risk-factor-based group, with the exception of persons who have

contraindication. Category B recommendations do not apply to allembers of an age- or risk-based subgroup of the population, but

n the context of a clinician-patient interaction, vaccination may

(2011) 9171– 9176 9173

be found to be appropriate for a person. The category B recommen-dation is similar to what was previously referred to by the ACIPas permissive recommendation. In some instances, the ACIP maydecide not to make a recommendation if further information isneeded.

The recommendation category depends on the balance betweendesirable (benefits, savings) and undesirable (harms, costs) effectsof vaccination. A category A recommendation is one for which thedesirable effects outweigh the undesirable effects (recommenda-tion for) or that the undesirable effects outweigh the desirableeffects (recommendation against). Key factors that can lead to acategory B recommendation include smaller net benefit (e.g., lowbaseline risk, small relative or absolute effects); lower confidencein the estimated effect of vaccination on health outcomes (e.g.,wide confidence intervals); variability in values attributed to ben-efits and harms; and lower cost-effectiveness or uncertainty aboutwhether the net benefits are worth the costs (e.g., because of lackof data on input assumptions that substantially affect the resultsof economic models). Examples of category A recommendationsinclude the ACIP recommendation to vaccinate all U.S. infants withrotavirus vaccine; recommendation to administer pneumococcalpolysaccharide vaccine to all children aged 2 years or older with cer-tain underlying medical conditions; and recommendation againstuse of the 2010–2011 Afluria influenza vaccine among childrenaged 6 months through 8 years [9]. An example of a category B rec-ommendation is the recommendation that the 2010–2011 Afluriavaccine may be used for a child aged 5–8 years with a medical con-dition that increases the child’s risk for influenza complications ifno other age-appropriate, seasonal influenza vaccine is available;providers should discuss with the parents or caregivers the benefitsand risks of Afluria vaccination before use of the vaccine [9].

ACIP members consider all relevant factors for making rec-ommendations. ACIP Work Groups include ACIP members andmultiple external experts and stakeholders in the preparation ofoptions for immunization to present to ACIP. ACIP meetings areopen to the public, and experts and stakeholders from liaisonorganizations and other entities provide input during the ACIPdecision-making process. Recommendations to the CDC/HHS arepassed by majority vote among committee members; recommen-dations are provisional pending acceptance by the Director ofCDC/Secretary of HHS [3]. Existing ACIP recommendations areupdated on a periodic basis.

3.4. Balance of benefits and harms

When balancing the desirable and undesirable effects, theimportance of the outcomes needs to be considered. The burden ofdisease, assessed from research studies or from surveillance data,plays an important role. For example, the number needed to vac-cinate to prevent one case is a function of the incidence of diseasein the absence of vaccination (baseline risk) and vaccine efficacy. Ifthe baseline risk substantially influences the magnitude of benefitor harm, and there are specific population subgroups at high or lowbaseline risks, separate recommendations may be made based onrisk groups or risk factors.

3.5. Type or quality of evidence

The evidence evaluation process includes conducting a system-atic search of the literature (including unpublished informationfrom vaccine manufacturers and surveillance systems), specifyingcriteria for including/excluding studies, and evaluating the body of

evidence for each outcome.

Categorization of the type of evidence for each outcome beginswith the study design, which is classified broadly as randomizedcontrolled trials (RCTs) or observational studies (e.g., cohort, case

9 cine 29

ctara

•

•

•

•

•

•

•

•

vtauweicebetrvbfitspillicbut

174 F. Ahmed et al. / Vac

ontrol, controlled before–after, interrupted time series). Five cri-eria are applied to assess limitations and three criteria are used tossess special strengths of the body of evidence [23]. These crite-ia determine the final classification into the four types describedbove.

The criteria to assess limitations of a body of evidence are:

Risk of bias or methodological limitations (major limitations forthe majority of studies in the review, such as inappropriate selec-tion of vaccinated and unvaccinated groups, failure to adequatelymeasure or control for confounding, lack of blinding, or high lossto follow-up).Inconsistency of results across studies (heterogeneity that can-not be explained by differences in populations, interventions,comparator, or outcomes).Indirectness of evidence (the question being addressed is dif-ferent from the available evidence regarding the population,intervention, comparator, or an outcome; or indirect comparisonof vaccines when studies are available that compare each vaccineagainst placebo but head-to-head comparisons are not available).Imprecision of estimates (wide confidence interval around thepooled or best estimate of effect).Publication bias (systematic underestimate or overestimate ofbenefits or harms due to selective publication of studies).

The criteria to assess special strengths of a body of evidence are:

Strength of association (the stronger the association, as measuredby the magnitude of the relative risk or odds ratio, the less likelyit is that all of the apparent benefit or harm can be explained byresidual confounding or bias).Dose–response (includes scenarios of increasing vaccine efficacywith increasing number of doses, and declining disease incidencewith increasing population vaccination rates).Direction of all plausible residual confounding or bias (plausiblebiases reduce the demonstrated effect or increase the effect whenno effect is observed).

Well-performed observational studies may be viewed as pro-iding exceptionally strong evidence if: (a) results from at leastwo studies show relative risk of approximately >2 or <0.5; (b)

dose–response gradient is observed; or (c) all plausible resid-al confounding or bias would reduce a demonstrated effect, orould suggest a spurious effect when results show no effect. For

xample, publicity about a potential adverse event might resultn increased spontaneous reporting of the adverse event in vac-inated persons compared to that in unvaccinated persons; ifpidemiological studies find no association despite the potentialias associated with differential reporting due to publicity, thevidence of lack of association may be considered to be excep-ionally strong. Well-performed observational studies showingelative risk of approximately >5 or <0.2 may be viewed as pro-iding overwhelming evidence [26,27]. Observational studies maye classified as having important limitations based on any of theve criteria for assessing limitations. The criteria may be addi-ive. For example, well-performed observational studies showingtrong association as well as dose–response may be viewed asroviding overwhelming evidence. RCTs may be classified as hav-

ng important limitations, notable limitations, or several majorimitations depending on the total number and weight of theimitations. For example, RCTs with non-trivial methodological lim-tations (e.g., incomplete accounting of study subjects) may be

lassified as RCTs with important limitations, whereas RCTs withoth methodological limitations and indirectness of evidence (e.g.,se of immunogenicity as surrogate for disease outcomes whenhere are no well-established standard correlates of protection)

(2011) 9171– 9176

may be categorized as RCT(s) with notable limitations. It shouldbe noted that the absence of a dose–response relationship does notnecessarily reduce confidence in the observed effect, and that a rel-ative risk of >2 by itself may not indicate strong evidence if thereare methodological limitations (e.g., failure to adjust for multiplecomparisons). The strength of association criterion will not apply tonon-inferiority studies or safety studies showing absence of harm.

3.6. Indirect evidence

The indirectness criterion can be used to evaluate indirectevidence. Such scenarios include applicability of evidence to popu-lations that were not included in RCTs; applicability of evidencefrom an old vaccine to a new formulation of the vaccine; andapplicability of evidence for shorter-term outcomes to long-termoutcomes (e.g., hepatitis B infection vs. liver cancer; rubella vs. con-genital rubella syndrome). Biologic information may be taken intoaccount when evaluating indirect evidence. For example, RCTs haveassessed the efficacy of rotavirus vaccine among healthy infants,but efficacy data are not available for infants with chronic gas-trointestinal tract diseases [28]. If there is reason to believe thatthe vaccine efficacy may be different for infants with chronic gas-trointestinal diseases compared to that for healthy infants, it wouldbe considered indirect evidence and therefore the evidence type forefficacy of vaccine among infants with chronic gastrointestinal tractdiseases may be categorized as RCTs with important limitations(indirectness).

3.7. Overall type of evidence

For purposes of providing an overall evidence type when theevidence type varies across outcomes that are critical for decision-making, the evidence type that conveys lower certainty of theeffect of vaccination for any of the critical outcomes determinesthe overall evidence type. An exception to this guidance is when theevidence type differs across critical outcomes but the associationis similar in direction. For example, if a vaccine reduces incidenceof disease and hospitalization and this is based on well-performedRCTs (i.e., the highest level of evidence), but there are only ecolog-ical studies assessing protection associated with herd immunity,and all associations indicate similar effects, the overall evidencetype may be categorized as RCTs even if protection due to herdimmunity is considered to be a critical outcome.

3.8. Values and preferences

Values can be described as the relative importance of out-comes related to benefits, harms, and costs. Values, as well asethical considerations, play a key role in developing recommen-dations. The values should reflect those of the people affected,including the general population, patients, clinicians, and policy-makers. For example, for assessing new information on the risk offebrile seizures among children aged 12–23 months after adminis-tration of the combination measles, mumps, rubella, and varicella(MMRV) vaccine, the ACIP considered values of parents and clini-cians regarding the risks of febrile seizures [29]. When values areparticularly important for the interpretation of recommendations,the key values that are considered in making a recommendationshould be described.

3.9. Health economic data

Economic analysis is an important factor that informsjudgments in formulating recommendations (e.g., cost-benefit,cost-utility, cost-effectiveness). Use of a fixed cut-off thresholdsuch as $50,000 or $100,000 per quality adjusted life year (QALY) for

F. Ahmed et al. / Vaccine 29 (2011) 9171– 9176 9175

Table 1Benefits and harms.a

Outcome No. of subjects(# studies)

Incidence incontrols

Incidence invaccinated

Vaccine efficacy orrelative risk (95% CI)

Risk difference per1000 (95% CI)

Number needed totreat (NNT)

Rotavirus (RV) diarrhea 5627 (2 RCTs) 12.9% 3.5% 73% (66, 78) −94 (−85, −100) 11Severe RV diarrhea 5627 (2 RCTs) 2.0% 0.1% 97% (86, 99) −19 (−17, −20) 52Hospitalization for RV diarrhea 57,134 (1 RCT) 0.5% 0.02% 96% (91, 98) −5 (−5, −5) 205Intussusception 70,139 (3 RCTs) 1.4 per 10,000 1.7 per 10,000 1.20 (0.37, 3.93) 0.03 (−0.1, 0.4) –Other serious adverse events 70,139 (3 RCTs) 2.3% 2.2% 0.96 (0.87, 1.06) −1 (−3, 1) –

RCT, randomized controlled trial.a Incidence for diarrhea outcomes are from ≥14 days after the third dose of rotavirus vaccine through the first full rotavirus season after vaccination; incidence for

intussusception and other serious adverse events are within 42 days after any dose. For the fifth column, vaccine efficacy is presented for benefits and relative risk ispresented for harms. Incidence in controls = weighted incidence among placebo recipients across studies. Incidence in vaccinated = incidence in controls × pooled relativerisk. Vaccine efficacy = (1 − pooled relative risk) × 100. Risk difference = incidence in vaccinated − incidence in controls. NNT = 1/risk difference. NNT is shown when the riskdifference is statistically significant. Pooled relative risks were computed using the RevMan software (http://ims.cochrane.org/revman).

Table 2Type of evidence.

Outcome Design (# studies) Risk of bias Inconsistency Indirectness Imprecision Other considerationsa Evidence typeb

Rotavirus (RV) diarrhea RCT (2) No serious No serious No serious No serious None 1Severe RV diarrhea RCT (2) No serious No serious No serious No serious None 1Hospitalization for RV diarrhea RCT (1) No serious No serious No serious No serious None 1Intussusception RCT (3) No serious No serious No serious No serious None 1Other serious adverse events RCT (3) No serious No serious No serious No serious None 1

a Strength of association, dose–response, direction of all plausible residual confounding, publication bias.ence

s notabi

dnthoffrrPt

3

tTepimvuv

3

c

TS

b Evidence type: 1 – randomized controlled trials (RCTs), or overwhelming evidtrong evidence from observational studies; 3 – observational studies, or RCTs withmportant limitations, or RCTs with several major limitations.

etermining cost-effectiveness, however, ignores other determi-ants of value [30]. The methodology described above for assessinghe type or quality of evidence is not intended to be applied toealth economic analyses based on modeling. The ACIP Work Groupn Economic Analysis has published a document titled Guidanceor Health Economic Studies Presented to the ACIP that provides aramework for the description and presentation of the methods andesults, and that stipulates technical review by anonymous peereviewers of any economic materials presented to the ACIP [31].resentation of health economic data should be undertaken usinghe guidelines in that document.

.10. Evidence tables

Information on the magnitude of the benefits and harms andhe type of evidence are to be summarized in evidence tables.ables 1–3 provide examples of evidence tables for the body ofvidence for selected outcomes for the human-bovine reassortantentavalent rotavirus vaccine (RotaTeqTM). For this example, we

ncluded phase 3 studies that were available when the ACIP recom-ended vaccination of U.S. infants with three doses of the rotavirus

accine [32–35]. We excluded phase 1 and phase 2 studies thatsed a different vaccine formulation as well as studies of rotavirusaccines formulated with other rotavirus strains.

.11. Format for presenting recommendations

A recommendation is to be followed by the recommendationategory and evidence type in parentheses, and should include a

able 3ummary of evidence.

Comparison Outcome Study design(# studies)

Fin

Rotavirus vaccinationvs. no vaccination

Rotavirus (RV) diarrhea RCT (2) DeSevere RV diarrhea RCT (2) DeHospitalization for RV diarrhea RCT (1) DeIntussusception RCT (3) NoOther serious adverse events RCT (3) No

from observational studies; 2 – RCTs with important limitations, or exceptionallyle limitations; 4 – clinical experience and observations, observational studies with

Remarks section that summarizes the key thought process behindthe recommendation. For example, the recommendation for use ofpentavalent rotavirus vaccine may be presented as follows:

“Recommendation: ACIP recommends vaccination of U.S. infantswith three doses of rotavirus vaccine administered orally at ages 2,4, and 6 months (recommendation category: A, evidence type: 1).

Remarks: Nearly every child in the U.S. is infected withrotavirus by age 5 years, resulting in approximately 410,000 physi-cian visits, 205,000–272,000 emergency department visits, and55,000–70,000 hospitalizations each year. Benefits of vaccinationare large compared to potential harms.”

4. Identifying population subgroups at higher risk ofdisease

The GRADE framework is intended to be used for assessing thequality of evidence pertaining to the effect of an intervention onhealth outcomes. However, it may be necessary to evaluate thequality of studies assessing whether specific population subgroupsare at higher risk of disease in the absence of vaccination. The cri-terion risk of bias may be used for such assessments.

5. Summary

Adoption of the GRADE framework that is used by a numberof organizations will facilitate communication and collaboration.Categorizing the evidence involves judgments that are inherentto any evidence evaluation system. The GRADE system does not

dings Evidence type Overall evidencetype

creased risk among vaccinated infants 1 1creased risk among vaccinated infants 1creased risk among vaccinated infants 1

difference 1 difference 1

9 cine 29

grdiedvltbfitttc

A

aaoC

R

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

176 F. Ahmed et al. / Vac

uarantee reproducible judgments but one of its strengths is that itequires explicit judgment that is made transparent to users so thatisagreement can be resolved. Future ACIP recommendations will

nclude the recommendation category and the evidence type forach key recommendation. Key factors for developing recommen-ations include the balance of benefits and harms, type of evidence,alues and preferences, and health economic analyses. Rather thanabeling recommendations as strong or weak, ACIP recommenda-ions will be labeled as A or B. Category A recommendations wille made for all persons in an age group or for all persons in a risk-actor-based group. Category B recommendations will be made forndividual clinical decision making. Evidence tables will be usedo summarize the benefits and harms and the strengths and limi-ations of the body of evidence. The new framework will enhancehe ACIP’s decision-making process by making it more transparent,onsistent and systematic.

cknowledgments

We gratefully acknowledge contributions of Dr. Daniel Fishbeinnd other individuals who participated in the initial work groupctivities. The findings and conclusions in this article are thosef the authors and do not necessarily represent the views of theenters for Disease Control and Prevention.

eferences

[1] Smith JC, Snider DE, Pickering LK. Immunization policy development in theUnited States: the role of the Advisory Committee on Immunization Practices.Ann Intern Med 2009;150(1):45–9.

[2] Smith JC. The structure, role, and procedures of the U.S. Advisory Committeeon Immunization Practices (ACIP). Vaccine 2010;28(Suppl 1):A68–75.

[3] Charter of the Advisory Committee on Immunization Practices (ACIP). CDC.http://www.cdc.gov/vaccines/recs/acip/charter.htm [accessed 12.09.10]. Reftype: Generic.

[4] Bryson M, Duclos P, Jolly A, Cakmak N. A global look at national ImmunizationTechnical Advisory Groups. Vaccine 2010;28(Suppl 1):A13–7.

[5] Bryson M, Duclos P, Jolly A, Bryson J. A systematic review of national immu-nization policy making processes. Vaccine 2010;28(Suppl 1):A6–12.

[6] Erickson LJ, De WP, Farand L. An analytical framework for immunization pro-grams in Canada. Vaccine 2005;23(19):2470–6.

[7] Houweling H, Verweij M, Ruitenberg EJ. Criteria for inclusion of vaccinationsin public programmes. Vaccine 2010;28(17):2924–31.

[8] Piso B, Wild C. Decision support in vaccination policies. Vaccine2009;27(43):5923–8.

[9] Centers for Disease Control and Prevention. Vaccines & immunizations. Avail-able from: www.cdc.gov/vaccines [accessed on 11.18.10]. Ref type: Generic.

10] U.S. Department of Health and Human Services. Healthy People. Available from:www.healthypeople.gov [accessed 11.18.10]. Ref type: Generic.

11] Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, onso-Coello P, et al.GRADE: an emerging consensus on rating quality of evidence and strength ofrecommendations. BMJ 2008;336(7650):924–6.

12] Direct and indirect effects of routine vaccination of children with 7-valentpneumococcal conjugate vaccine on incidence of invasive pneumococ-

cal disease—United States, 1998–2003. MMWR Morb Mortal Wkly Rep2005;54(36):893–7.

13] Samandari T, Bell BP, Armstrong GL. Quantifying the impact of hep-atitis A immunization in the United States, 1995–2001. Vaccine2004;22(31–32):4342–50.

[

(2011) 9171– 9176

14] American Academy of Pediatrics Steering Committee on Quality Improvementand Management. Classifying recommendations for clinical practice guidelines.Pediatrics 2004;114(3):874–7.

15] Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. Grading qual-ity of evidence and strength of recommendations. BMJ 2004;328(7454):1490.

16] Briss PA, Zaza S, Pappaioanou M, Fielding J, Wright-De AL, Truman BI,et al. Developing an evidence-based Guide to Community PreventiveServices—methods. The Task Force on Community Preventive Services. Am JPrev Med 2000;18(1 Suppl):35–43.

17] Guirguis-Blake J, Calonge N, Miller T, Siu A, Teutsch S, Whitlock E. Currentprocesses of the U.S. Preventive Services Task Force: refining evidence-basedrecommendation development. Ann Intern Med 2007;147(2):117–22.

18] Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al. Currentmethods of the US Preventive Services Task Force: a review of the process. AmJ Prev Med 2001;20(3 Suppl):21–35.

19] Evidence-based recommendations for immunization—methods of the NationalAdvisory Committee on Immunization. An Advisory Committee Statement(ACS). Can Commun Dis Rep 2009;35(ACS-1):1–10.

20] GRADE Working Group. Grading of Recommendations Assessment, Develop-ment and Evaluation (GRADE). Available from: www.gradeworkinggroup.org;[accessed 01.12.10]. Ref type: Generic.

21] Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ.What is “quality of evidence” and why is it important to clinicians? BMJ2008;336(7651):995–8.

22] Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ, et al.Going from evidence to recommendations. BMJ 2008;336(7652):1049–51.

23] Schunemann H, Brodek J, Oxman A, editors. GRADE handbook for grad-ing quality of evidence and strength of recommendation. Version 3.2[updated March 2009]. The GRADE Working Group; 2009. Available from:http://ims.cochrane.org/gradepro [accessed 15.07.11]. Ref type: Generic.

24] Methods reference guide for effectiveness and comparative effectivenessreviews. Agency for Healthcare Research and Quality; 2007. Available from:http://effectivehealthcare.ahrq.gov/repFiles/2007 10DraftMethodsGuide.pdf[cited 13.06.08].

25] Glanz JM, McClure DL, Xu S, Hambidge SJ, Lee M, Kolczak MS, et al. Four dif-ferent study designs to evaluate vaccine safety were equally validated withcontrasting limitations. J Clin Epidemiol 2006;59(8):808–18.

26] Bross ID. Pertinency of an extraneous variable. J Chronic Dis1967;20(7):487–95.

27] Glasziou P, Chalmers I, Rawlins M, McCulloch P. When are randomised trialsunnecessary? Picking signal from noise. BMJ 2007;334(7589):349–51.

28] Cortese MM, Parashar UD. Prevention of rotavirus gastroenteritis among infantsand children: recommendations of the Advisory Committee on ImmunizationPractices (ACIP). MMWR Recomm Rep 2009;58(RR-2):1–25.

29] Marin M, Broder KR, Temte JL, Snider DE, Seward JF. Use of combinationmeasles, mumps, rubella, and varicella vaccine: recommendations of theAdvisory Committee on Immunization Practices (ACIP). MMWR Recomm Rep2010;59(RR-3):1–12.

30] Grosse SD, Teutsch SM, Haddix AC. Lessons from cost-effectiveness research forUnited States public health policy. Annu Rev Public Health 2007;28:365–91.

31] Centers for Disease Control and Prevention. Guidance for presentation of eco-nomic studies to the Advisory Committee on Immunization Practices. MMWR2008;57(5):125–6.

32] Food and Drug Administration. Clinical Review of New Biologics LicenseApplication STN #125122 RotaTeq. 2006. Available from: http://www.fda.gov/BiologicsBloodVaccines/Vaccines/ApprovedProducts/ucm094063.htm;[accessed 26.03.10]. Ref type: Generic.

33] Block SL, Vesikari T, Goveia MG, Rivers SB, Adeyi BA, Dallas MJ, et al. Efficacy,immunogenicity, and safety of a pentavalent human-bovine (WC3) reassortantrotavirus vaccine at the end of shelf life. Pediatrics 2007;119(1):11–8.

34] Parashar UD, Alexander JP, Glass RI. Prevention of rotavirus gastroenteritisamong infants and children: recommendations of the Advisory Committee

on Immunization Practices (ACIP). MMWR Recomm Rep 2006;55(RR-12):1–13.

35] Vesikari T, Matson DO, Dennehy P, Van DP, Santosham M, Rodriguez Z, et al.Safety and efficacy of a pentavalent human-bovine (WC3) reassortant rotavirusvaccine. N Engl J Med 2006;354(1):23–33.