Cd

MVFa

b

c

d

e

Uf

g

a

ARRAA

KDVDMFHI

p(LC

h0

Vaccine 33 (2015) 1206–1217

Contents lists available at ScienceDirect

Vaccine

j o ur na l ho me page: www.elsev ier .com/ locate /vacc ine

onceptual frameworks and key dimensions to support coverageecisions for vaccines

arien González-Lorenzoa,∗, Alessandra Piatti c, Liliana Coppolac, Maria Gramegnac,ittorio Demichelid, Alessia Melegaroe, Marcello Tirania, Elena Parmelli f,g,rancesco Auxiliaa, Lorenzo Mojaa,b, the Vaccine Decision Group1

Dipartimento di Scienze Biomediche per la Salute, Università degli Studi di Milano, ItalyUnità di Epidemiologia Clinica, IRCCS Istituto ortopedico Galeazzi, Milano, ItalyDirezione Generale Salute, UO Governo della Prevenzione e Tutela Sanitaria, Regione Lombardia, Milano, ItalyServizio Regionale di Riferimento per l’Epidemiologia, SSEpi-SeREMI - CochraneVaccines Field, Azienda Sanitaria Locale ASL AL, Alessandria, ItalyDipartimento di Analisi delle Politiche e Management Pubblico & Centro Dondena per la ricerca sulle Dinamiche Sociali e Politiche Pubbliche,niversità Bocconi, Milano, ItalyDipartimento di Epidemiologia del Servizio Sanitario Regionale del Lazio, Roma, ItalyCentro Cochrane Italiano, Università degli Studi di Modena e Reggio Emilia, Italy

r t i c l e i n f o

rticle history:eceived 29 August 2014eceived in revised form 8 December 2014ccepted 9 December 2014vailable online 19 December 2014

eywords:ecision-makingaccinationecision aidodel

rameworkealth policy

mmunisation programmes

a b s t r a c t

Background: Health policy makers often have to face decisions on whether and how to incorporate newvaccines into immunisation plans. This study aims to review and catalogue the relevant current frame-works and taxonomies on vaccines and connect these to the DECIDE Evidence to Decision framework(EtD), a general framework based on evidence-based criteria to guide decision-making on interventionadoption.Methods: We systematically searched MEDLINE, EMBASE, Cochrane Library and funding agency websitesfrom 1990 to 2013. We included systematic reviews and primary studies presenting decision-makingtools for community vaccine adoption. We qualitatively summarised the reports by purpose, targetedcountry, principal results, and decisional models. We then extracted and compared the dimensionsadopted by vaccine frameworks across studies.Results: Fourteen studies (five systematic reviews and nine primary studies) were included. Several factorsfrequently influenced decision-makers’ views on vaccines: the most frequent political-context factorsconsidered were Importance of illness or problem, Vaccine characteristics, Resource use, and Feasibility.Others such as Values and preferences and Acceptability were less consistently reported. We did not findevidence on the reasons why a framework for vaccine adoption differs from that for decisions on theadoption of an intervention in general, such as the EtD. There are limited data on how dimensions areexplained in practical factors and directly linked to coverage decisions.

Conclusions: This review summarises conceptual models and taxonomy of a heterogeneous and evolvingarea in health policy decisions. A shared and comprehensive framework on vaccine coverage remainsto be achieved with its single dimensions (epidemiologic, effectiveness, economic, and social) valueddifferently across studies. A generic tool such as the EtD conceptualises all relevant dimensions, andmight reduce inconsistencies.

© 2014 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-SA

∗ Corresponding author at: Dipartimento di Scienze Biomediche per la Salute, UniversitàE-mail addresses: marien.gonzalez@unimi.it, mariengonzalezlorenzo@gmail.com (M.

1 The Vaccine Decision Group: Marien González-Lorenzo (marien.gonzalez@uniola (liliana coppola@regione.lombardia.it), Maria Gramegna (Maria Gramegna@regalessia.melegaro@unibocconi.it), Marcello Tirani (marcello.tirani@unimi.it), Elena Parmorenzo Moja (lorenzo.moja@unimi.it), Vanna Pistotti (vanna.pistotti@marionegri.it)(IRCochrane), Alessandro Zanetti (alessandro.zanetti@unimi.it)(Dipartimento di Scienze Bio

ttp://dx.doi.org/10.1016/j.vaccine.2014.12.020264-410X/© 2014 The Authors. Published by Elsevier Ltd. This is an open access article un

license (http://creativecommons.org/licenses/by-nc-sa/3.0/).

degli studi di Milano. Via Carlo Pascal 36, Milano 20133, Italy. Tel.: +39 02 50315119.González-Lorenzo).mi.it), Alessandra Piatti (alessandra piatti@regione.lombardia.it), Liliana Cop-ione.lombardia.it), Vittorio Demicheli (vdemicheli@aslal.it), Alessia Melegaro

elli (elena.parmelli@unimore.it), Francesco Auxilia (francesco.auxilia@unimi.it),CS Istituto di ricerche farmacologiche Mario Negri, Milano, Italy; Network Italianomediche per la Salute, Università degli Studi di Milano, Italy).

der the CC BY-NC-SA license (http://creativecommons.org/licenses/by-nc-sa/3.0/).

/ Vacc

1

ecoHiam

pvdo

nsfgwmraci

ua[ofpf(op

S(ddo(opieicnfaffdtus

pvd(tf

M. González-Lorenzo et al.

. Background

Over the past decade, several randomised controlled trialsvaluating the safety or the efficacy of new vaccines have beenompleted [1]. One approved by drug regulatory agencies, somef these vaccines, such as those for human papilloma virus and1N1 influenza A, have been introduced into low, middle, and high-

ncome countries. Health care systems are discussing the potentialdoption of other vaccines such as rotavirus, pneumococcal andeningococcal vaccines [2,3].New vaccines might represent major medical breakthroughs in

reventive care, while others may have scarce or limited innovativealue. In both cases, the production and commercialisation of newlyeveloped vaccines are likely to be more expansive than previousnes.

From a public health perspective, health policy makers willeed to answer questions regarding dosage, general effectiveness,afety, impact, and expected compliance of vaccines. They musturther make decisions between new vaccines and alternative pro-rammes (e.g., interventions for stopping the spread of infection),eighing the costs and expected health benefits in the develop-ent of national immunisation programmes [4]. Limited financial

esources should be distributed in a fair and effective manner tochieve the best possible outcomes within local rather than globalonditions, considering all direct and indirect consequences of themmunisation programme.

In 2005, the World Health Organisation (WHO) released the doc-ment entitled “Vaccine introduction guidelines. Adding a vaccine to

national immunisation programme: decision and implementation”5]. The document has since been updated [6]. This publicationffers decision-makers a structured and comprehensive frameworkor determining the impact of one or more vaccine options, androvides answers to various policy questions. Other frameworksollowing a similar approach to the WHO have also been proposede.g. Global Immunisation Strategic Framework 2011–2015) [7],utlining useful criteria, indicators, sources of data and validationrocesses.

The Developing and Evaluating Communication Strategies toupport Informed Decisions and Practice Based on EvidenceDECIDE) project is a collaborative research project aimed toevelop and evaluate communication strategies to support evi-ence informed decisions building on the work of the Gradingf Recommendations Assessment Development and EvaluationGRADE) Working Group and the Cochrane Applicability and Rec-mmendations Methods Group [8–10]. As a part of the DECIDEroject, a general framework for communicating evidence to

nform coverage decisions called Evidence to Decision (EtD) for cov-rage is being developed [9,11]. In this context, “coverage” decisions defined as the potential provision of the intervention (i.e., vac-ine) at no cost to the user and at the expense of the regional orational health system, health insurance organisations or health

oundations and charities (e.g., the GAVI Alliance). This frameworkims to be general, encompassing a wide spectrum of decisionsrom preventive to therapeutic or diagnostic interventions. Theramework is divided into three parts: the first presents the con-ition and background information; the second involves a tablehat provides information identified as essential to make a partic-lar type of decision (i.e., dimension and criteria used); the thirdummarises the components used to make the decision [9].

The aim of our study was to explore specific aspects of healtholicy decision frameworks on vaccines. We a priori assumed thataccines differ from other drugs at least in the types of recommen-

ations or decisions involving its uptake by health care systemse.g., clinical recommendations from an individual patient perspec-ive or from a population perspective. Thus, we thought a genericramework such as the EtD may not be suitable for vaccines. When

ine 33 (2015) 1206–1217 1207

a generic framework is applied to support specific decisions (i.e.,vaccines), its validity and sensitivity can be limited. To fully addressthe issues relevant specifically to vaccines, we decided to system-atically identify, summarise, and catalogue current frameworksand taxonomies on vaccines. We sought to then connect thesedimensions to a general framework for communicating evidence toinform coverage decisions (i.e., the EtD framework). We exploreddifferences among the existing frameworks specific to vaccines andcompared these with the wider EtD framework in order to informthe EtD framework with specific and comprehensive dimensionsand criteria related to the adoption or modification of immunisa-tion policies.

2. Methods

The research was structured in three phases. In the first phase,we developed a comprehensive literature search to identify con-ceptual frameworks on vaccines. In the second phase, we identifiedand synthesised the main dimensions and constructs suggested. Inthe third phase, we provided specific suggestions on target vaccineadoption decisions to inform the EtD framework.

2.1. Criteria for considering studies for inclusion

We included systematic reviews, which summarised frame-works for vaccine adoption decision-making. We considered asa systematic review any review of a clearly formulated ques-tion that uses systematic and explicit methods to identify, select,and critically appraise relevant research, and to collect and ana-lyse data from the studies that are included in the review [12].We excluded studies presenting: (a) no frameworks for vaccineadoption decision-making; (b) providing a narrow focus on a sin-gle dimension (e.g., cost-effectiveness studies); (c) basic scientificresearch on vaccine development; and (d) data on non-human vac-cinations.

To ensure the comprehensiveness of our review, we alsoincluded primary studies (i.e., conceptual studies describing orproposing a set of decision criteria or a decision-making tool),which were not included in the selected reviews. To identify theseprimary studies providing a new approach not previously describedin the included reviews, we crosschecked the reference lists of allincluded reviews against the output of the search strategies.

All relevant studies were included, regardless of their languageor publication status.

2.2. Search methods for identification of eligible studies

We searched for systematic reviews published between January1990 and March 2013 on the following bibliographic databases:MEDLINE, EMBASE, The Cochrane Library (i.e., Cochrane Databaseof Systematic Reviews, Database of Abstracts of Reviews of Effects,Health Technology Assessment Database, and NHS Economic Eval-uation Database). Our search strategy featured the followingkeywords: decision-making, vaccination, decision aid, model, frame-work, health policy, and immunisation programmes. In order toconsider primary studies that were not included in reviews, we per-formed a sensitive search strategy on MEDLINE, EMBASE and TheCochrane Central Register of Controlled Trials between March 2010and March 2013 (the final date of the searches reported in Burchettet al. [13], the most recent systematic review on our subject).

In addition to bibliographic databases, we searched the fol-

lowing websites: WHO, National Institutes of Health, Centers forDisease Control and Prevention, European Centre for Disease Pre-vention and Control and Pan American Health Organisation. Toidentify papers that were not indexed in the above-mentioned

1208 M. González-Lorenzo et al. / Vaccine 33 (2015) 1206–1217

Full-text articles excluded, with reasons:

- No decisional frameworks (n = 53)- The framework (for primary studies) wasincluded in a SR (n = 9)- A vaccine economic analyses (n = 10)- Not stud y design of interest (n = 13)- Focused on no n-hu man vacc inati ons (n = 1)

Records identified through primary study databases searching (2010 -2013)

(n = 99 7)

Records identified through systematic reviews (SRs) databases searching

(199 0-2013 )(n = 292 0)

Full-text articles assessed for eli gibilit y

SRs (n = 23)Primary studies (n = 75)

Stud ies i nclud ed i n qu alit ati ve synthesis(n = 14)

- 5 SRs (2 found by hand sea rches)- 9 primary stud ies

low d

df

2

2

robAb

2

pamitt

2

ofb

cfodmefs

Fig. 1. F

atabases, we examined the reference lists of all included studiesor additional literature.

.3. Data collection and analysis

.3.1. Study selectionTwo authors independently examined the titles and abstracts

etrieved by the search strategy and assessed the full-text articlesf the potential relevant studies. Any disagreement was resolvedy discussion. We documented the reasons for excluding studies.

study flow diagram according to the PRISMA statement [14] cane found in Fig. 1.

.3.2. Data extraction and managementWe extracted information on the purpose of the study, the

ublication date, the origin and targeted country, primary resultsnd decisional frameworks and taxonomy used. All doubtful infor-ation were presented to a third author and discussed before

nclusion. Given the heterogeneity of study designs, their descrip-ive nature and lack of a standard methodology, we did not assesshe methodological quality of the included studies.

.4. Development and analysis of the conceptual framework

We first recorded all of the components about coverage rec-mmendations proposed in the included studies, assessing therequency of each. We were inclusive, as we did not exclude itemsased on their relevance or frequency.

We then used the EtD model to structure the identifiedomponents [8,9], aligning the terminology used in the vaccinerameworks to corresponding ones of the EtD. In particular, werganised our vaccine framework to present the information byimensions and criteria. In EtD, the broader headings involve the

ain domains to be assessed (i.e., burden of illness or problem, ben-

fits and harms, values and preferences, resources use, equity, andeasibility). The criteria for each domain are then presented asubheadings (e.g., incidence, efficacy, cost-benefit, etc.) placed at

iagram.

subsequently narrow levels of the proposed hierarchy. Again, weretained all components found across included studies, althoughwe might have modified the relationship between the domains andthe criteria as intended by the original study authors.

This entire process was carried out by a multi-disciplinary groupof 12 review authors, including methodologists, policy makers,economists, and medical editors. The authors participated in exten-sive face-to-face and electronic correspondence. A few authors metin person on several occasions to consider all comments and refinethe conceptual framework. We regularly circulated the documentamong authors who approved the final decisional framework.

3. Results

3.1. Search results

We identified 2920 reference citations after excluding the dupli-cates. Among them, 98 potentially relevant publications wereretrieved in full text (Fig. 1). Eighty-six publications were excludedbecause they: lacked a decisional framework for community vac-cine adoption (n = 53) or an appropriate study design (e.g., narrativereviews) (n = 13); provided a narrow focus on a single dimension(e.g., economic analysis of vaccines) (n = 10); focused on non-human vaccinations (n = 1), or addressed a framework alreadydescribed in an included review(n = 9)(see additional file 1). Weidentified three additional systematic reviews [13,15,16] as wellas nine primary studies [17–25] that were not included in thesystematic reviews. We selected an additional two reviews throughhand searches [26,27], leading to a total of 14 included publications.

3.2. Description of systematic reviews and primary studiesincluded

All systematic reviews included were published after the year2000. All were in English and the locations of the primary authorswere from Canada [26], United Kingdom [13], Italy [27], Austria

/ Vacc

[s

waenTTtwtrtVc

sp[tcasGcao

r

3

rs

“iIaApqiID

towasapa

iDtviodNv

M. González-Lorenzo et al.

15], and Mexico [16]. Each review included a range of 5–85 primarytudies.

Two of the reviews focused on developing a theoretical frame-ork to support rational vaccination decision-making based on the

vailable scientific literature [15,27], while the reviews of Brysont al. [26] and Burchett et al. [13] reviewed the literature onational decision-making regarding the adoption of new vaccines.he former included the characteristics of National Immunizationechnical Advisory Groups (NITAGs), which provide expert adviceo government decision-makers. The latter analysed the frame-orks included using a grounded theory approach to search for

hemes and categories that emerged from the criteria included. Theeview by Tapia-Conyer et al. [16] assessed the evidence-basis ofhe Commission for the Future of Vaccines in Latin America (COF-AL) and feasibility in order to discuss each recommendation in theontext of existing vaccine-preventable diseases control strategies.

The nine primary studies were in English language. Of these,even targeted different geographic and cultural contexts: twoublications focused on a middle-income country (South Africa)18,21], one on low to middle-income countries [24], two onhe national immunisation policy of developed and high-incomeountries (United States, South Korea) [17,25], while two waspplied to malaria-endemic countries [19,22]. The remaining twotudies focused on accelerating the adoption of new vaccines inlobal Alliance for Vaccines and Immunization (GAVI) eligibleountries [23] as well as a proposal for embracing the GRADEpproach in the development of immunisation related WHO rec-mmendations [20].

We present a qualitative description of the five systematiceviews and the nine primary studies in additional files 2 and 3.

.3. The conceptual framework

For each publication, we extracted the dimensions and the crite-ia proposed as well as the use of a methodologically rigorousystem to develop the framework (e.g., GRADE approach) (Table 1).

We then removed redundant terms of similar concepts (e.g.,economical and financial issues” or “economic data”) across stud-es and identified 10 dimensions repeated across the frameworks:mportance of illness or problem, Vaccine characteristics (benefitsnd harms), Values and preferences, Resource use, Impact of vaccine,cceptability, Feasibility, Equity and ethical considerations, Legal andolitical considerations, and Decision-making. We quantified the fre-uency of each dimension across the conceptual frameworks in the

ncluded studies (Table 2). The most common dimensions were:mportance of illness or problem, Vaccine characteristics, Resource use,ecision-making and Feasibility.

We extracted the criteria reported in the frameworks, organisedhem into the 10 dimensions identified, and assessed the frequencyf each (Table 3). The most common criteria report across frame-orks was Health economic analyses. The criteria Vaccine efficacy

nd effectiveness and Vaccine safety, which were under the dimen-ion Vaccine characteristics (benefits and harms), were reported inlmost all frameworks. In the dimension Importance of illness orroblem, the criteria Incidence, Prevalence, Mortality, Social impactnd Specific risk groups were most frequently mentioned.

Regarding the use of a methodologically rigorous system tonform the frameworks in included studies, Ahmed et al. [17],uclos et al. [20], Piatti [27] and Tapia-Conyer et al. [16], proposed

he use of GRADE approach [10] for the information about theaccine effectiveness and safety. Piatti [27] and Blecher et al. [18]ndicated how they obtained the data to inform each dimension

f the framework. For example, Blecher et al. [18] noted that theirimension Burden of disease was informed by data reported by theational Department of Health; their dimension Effectiveness of theaccine was informed by data from published studies in reputable

ine 33 (2015) 1206–1217 1209

international journals, if possible, meta-analysis or Cochranereviews (see Table 1).

Eight of the frameworks analysed in Table 1 [16–23,25–27]reported that they considered guidance and recommendationsfrom WHO guidelines [5].

After carefully reviewing the frequency, hierarchy, and recip-rocal relationship of the vaccine framework components reportedacross studies, and standardizing the terminologies to develop our10 dimensions and criteria, we proceeded to link the 10 dimensionsto those of the general EtD framework, grouping some dimensionstogether. In fact, we were able to consider dimensions specifi-cally proposed for vaccines to relate to other interventions suchas drugs. The concepts outlined in the EtD were comprehensive,allowing us to align constructs and criteria proposed in the vac-cines frameworks with those of the EtD. Various dimensions suchas Importance of illness or problem, vaccine characteristics and impactof the immunisation programme, and resource use were consistentlyrepeated across vaccine frameworks, despite small differences inthe exact terms used to describe each. For instance, variations todescribe the dimension Importance of illness or problem includedburden of illness, seriousness of the problem, and number of peo-ple affected; the terms vaccine benefits and harms and impact ofvaccination or immunisation programmes were used interchange-ably to signify vaccine characteristics. These terms were groupedunder the umbrella of the EtD dimension named vaccine charac-teristics and impact of the immunisation programme. In other cases,the relationship between vaccine frameworks and the EtD frame-work was less straightforward. For instance, Acceptability, Legaland political considerations, and Decision-making were placed underthe umbrella of Feasibility. Thus, our final framework may havealtered the original constructs intended by authors in the originalpapers.

Table 4 presents the six dimensions of the EtD adapted to thevaccine context. The dimensions are Burden of disease, Vaccinecharacteristics and impact of immunisation programme, Valuesand preferences, Resource use, Equity and Feasibility. Each dimen-sion is followed by a brief description and its corresponding criteria.For example, the main criteria for the Burden of disease dimensionincludes Frequency of the disease/condition (e.g., incidence), Sever-ity of the disease/condition (e.g., mortality), and Social impact of thecondition (e.g., hospitalisation rate).

4. Discussion

This review analysed existing frameworks on the adoption ofnew vaccines in order to develop a comprehensive strategy thatincludes the most relevant and critical components for decision-makers. The dimensions proposed across vaccine frameworksaimed to inform and support coverage decisions (e.g., decisions bythird party payers about whether and how much to pay for vac-cines). The taxonomy of different frameworks featured a sufficientlevel of overlap. Overall, Burden of disease, Vaccine characteris-tics, Resource use, Decision-making, and Feasibility were frequentlyreported across frameworks as the key factors to be consideredin vaccine adoption decision-making, while Values and preferencesand Acceptability were less consistently reported. These dimensionslargely overlapped with those recommended for the assessmentof any technology by policy-makers [28]. We a priori thoughtthat a framework on vaccines would differ from that of otherinterventions such as drugs. However, after creating the frame-work and comparing it with EtD, we found that there were large

areas of overlap such that the EtD was helpful in structuring thedimensions of the vaccine framework. It is important that thedimensions address pragmatic factors and directly link to coveragedecisions.

1210 M. González-Lorenzo et al. / Vaccine 33 (2015) 1206–1217

Table 1Conceptual frameworks and empirical approaches analysed.

Studies General description Dimensions and criteria

Ahmed et al., 2011 [17] The framework is based on theGrading of RecommendationsAssessment, Development andEvaluation (GRADE) approach [10].

- Balance of benefits and harms: review of the baseline risk for disease and theexpected relative and absolute effects on vaccination on health outcomes.- Type of evidence: evidence is grouped into four categories, with the order reflectingthe level of confidence in the estimated effect of vaccination on health outcomes.- Values and preferences: relative importance of outcomes related to benefits, harmsand cost.- Health economic analyses: cost-benefit, cost-utility, cost-effectiveness.

Blecher et al., 2012 [18]Ngcobo and Cameron2012 [21]

The framework considers guidanceand recommendations from WHO[5]. The decision to introduce anew vaccine in South Africa isbased on local data.

- Disease burden and public health priority: incidence, morbidity and mortality of thecondition; the public health significance of the condition (Data reported by theNational Department of health).- Efficacy and safety (Published studies in reputable international journal, if possiblemeta-analysis or Cochrane reviews).- Cost-effectiveness: local studies are usually required, given very different coststructures across countries.- Total cost and affordability: depends on fiscal space, prioritisation, success in pricenegotiations and contracting.- Feasibility of implementation and availability of a credible implementation plan (Ifthere are doubts about feasibility, pilot studies may be useful).- International guidelines and advice of the South African National Advisory Group onImmunisation (NAGI) and other local and international experts.- Political process: besides the technical aspects, the budget process also involvescommunication between the Ministers of Health and Finance and approval by a widercommittee of Ministers, the national Cabinet and Parliament.

Brooks and Ba Nguz, 2012[19]Milstien et al., 2010 [22]

The framework considers guidanceand recommendations from WHO[5] for introducing new vaccinesfor malaria.

- Malaria disease burden: reported and confirmed cases by age group; reportedmalaria-related deaths by age group; malaria epidemiology profile by district; malariacases in pregnant women and HIV + population.- Other malaria interventions: impact of current malaria interventions; coverage ofcurrent malaria interventions; cost-effectiveness estimates of current malariainterventions.- Malaria vaccine impact: impact on mortality and morbidity by age group.- Economical and financial issues: cost-effectiveness estimates of malaria vaccine.- Malaria vaccine efficacy, quality and safety: adverse events; interaction with othervaccines; efficacy.

Bryson et al., 2010 [26] Factors considered by countrieswhen making recommendationsby presence of NITAGs.

- Burden of disease- Economic evaluation- Feasibility of local vaccine production- Feasibility of recommendation- Recommendations of other countries- Public perception- Vaccine safety and vaccine effectiveness

Burchett et al., 2012 [13] Nine broad categories of criteriawhich may influence decisions onvaccine adoption.

- Importance of the health problem: burden of disease data, political priority, costs ofdisease, perceptions of importance.- Vaccine characteristics: efficacy, effectiveness, safety, deliver issues.- Immunisation programme considerations: feasibility, supply.- Acceptability- Accessibility, equity and ethics.- Financial/economic issues: economic evaluation, incremental costs, funding sources,vaccine price, financial sustainability, affordability.- Impact: impact on health outcomes and on non-health outcome, effect ofco-administration, risks of serotype replacement.- Alternative interventions: cost–effectiveness of alternatives, effectiveness ofalternatives.- Decision-making process: Evidence sources/quality of evidence, actors involved,procedures, cues to action.

Cho, 2012 [25] The framework considers guidanceand recommendations from WHOfor introducing new vaccines inKorea [5].

- Disease burden in Korea: clinical characteristics of the disease, incidence, mortality,and case fatality rates.- Analyzes data on the efficacy, effectiveness, and safety of the vaccine: Sources ofinformation on the vaccine include clinical trials conducted both in Korea and in othercountries, WHO position papers, recommendations published by the U.S Centers forDisease Control and Prevention and the European Centre for Disease Prevention andControl.- Economic data: the cost, affordability, and financial sustainability of implementingthe new vaccine programme, as well as the vaccine’s cost-effectiveness.- Recommendations by sub-committees and the KCDC: isolation of the patients, theprophylactic management among the patient’s contacts, the diagnostic methods, thedisease surveillance and the immunisation.

M. González-Lorenzo et al. / Vaccine 33 (2015) 1206–1217 1211

Table 1 (Continued)

Studies General description Dimensions and criteria

Duclos et al., 2012 [20] The framework considers guidanceand recommendations from WHO[5] adopting the GRADE approach[10].

- Epidemiologic features of the disease: disease burden (including age specificmortality, morbidity, and social impact), specifics risk groups, epidemic potential,disease occurrence over time (i.e., secular trends), serogroup or serotype distribution(for serogroup or serotype specific vaccines), changes in epidemiological features overtime.- Clinical characteristics of the targeted disease: clinical management, disease severityand fatality, primary/secondary/tertiary care implications, long-term complicationsand medical care requirements.- Vaccine and immunisation characteristics: efficacy, effectiveness and populationimpact of the vaccine (including herd immunity), safety, indirect effects, cold chainand logistical concerns, vaccine availability, vaccine schedule, social andprogrammematic acceptability of the schedule, ability to reach the target populations,ability to monitor programme impact.- Economic considerations: cost of illness, vaccine and vaccine delivery costs, potentialfor vaccine price reductions, cost-effectiveness of immunisation programmes,affordability of immunisation.- Health system considerations: possible interactions with other interventions andcontrol strategies, possible impact of vaccine adoption on the wider health system.- Social impacts- Legal considerations- Ethical considerations

Levine et al., 2010 [23] This is proposed framework basedon observations of the process anddrivers of new vaccine adoption inGlobal Alliance for Vaccines andImmunisation (GAVI) eligiblecountries. Considers guidance andrecommendations from WHO [5].

Establish and organise evidence:- Epidemiology and burden of the disease (including the distribution of serotypes orstrains if relevant to vaccine policies).- Evidence based on the safety, efficacy and relative cost effectiveness of the vaccine asa solution.Establish supportive global policies:- Vaccine recommendations.- Financing policies- Procurement mechanismTranslate polices into local action:- Political will to implement- System to deliver and monitor

Makinen et al., 2012 [24] Principal factors considered indecision-making processes of newvaccine adoption inlower-middle-income countries(LMICs).

- Burden of disease data (e.g., mortality and morbidity)- Cost related drivers: vaccine market information, cost-effectiveness, budget impactand affordability, and available financing.- Other decision-making factors: the experience of neighbouring countries, access toadequate procurement mechanisms and the role played by global/regional bodies toengage countries.- Recommendations include making epidemiological data and vaccine marketinformation accessible to countries, building and reinforcing related analysis capacity,and promoting more efficient procurement mechanisms such as pooling.

Piatti, 2011 [27] The decision-making procedure isdivided in five analytical steps.For each step are providedmethods and indicators, one ofthem is the GRADE approach [10].

- Step (1) Safety: Adverse Events (nature and frequency); Risk factors and groups atrisk; Biological effects (biological disequilibrium) of the vaccine.- Step (2) Medical-Socio-Sanitary Aspects: Burden of disease, including the socialimpact of the disease; efficacy, vaccine coverage.- Step (3) Cost-efficacy analysis: direct and indirect cost, modelling; discounting;vaccine effectiveness; alternative scenario evaluation.- Step (4) Other implementation-related aspects: legal aspects; ethical aspects andequity.- Step (5) Priority: Integration of the above-mentioned points with the sense ofurgency for introducing it.

Piso and Wild, 2009 [15] The decision-making procedure isdivided in seven analytical steps.Elements belonging to the firststep were considered moreimportant and incisive in shapingthe decisional process than thefollowing ones.

- Step (1) Public health relevance and alternative measures, immunisation strategy,conformity of programmes, research questions.- Step (2) Disease considerations: burden of disease, clinical manifestations, currenttreatment, epidemiology, risk groups and risk factors, social impact and otherpreventives measures; Vaccine considerations: vaccine characteristics, supply,administration schedule, immune response, efficacy and utilisation, populationeffectiveness and safety.- Step (3) Cost-effectiveness analysis.- Step (4) Considerations on acceptability and feasibility of the new programme, equityand ethical implications, legal and political considerations, potential side effects.- Step (5) Final decision: decision-making process itself.- Step (6) Implementation.- Step (7) Surveillance of vaccine coverage and utilisation, of epidemiologic changes,the frequency and nature of adverse events, immune surveillance and re-evaluation(revision).

Tapia-Conyer et al. [16] The evidence-basis of theCommission for the Future ofVaccines in Latin America(COFVAL) and feasibility.

- Burden of disease and vaccine coverage- Epidemiological surveillance- National health accounts- Regional vaccination reference schemes- Professionalising immunisation policies and practices- Vaccine Advisory Committees- Innovative financing mechanisms for purchasing vaccines

1212 M. González-Lorenzo et al. / Vacc

Tab

le

2D

imen

sion

s

con

sid

ered

acro

ss

con

cep

tual

fram

ewor

ks

and

emp

iric

al

app

roac

hes

.

Stu

die

sD

imen

sion

s

Imp

orta

nce

ofil

lnes

s

orp

robl

em

Vac

cin

ech

arac

teri

stic

s(b

enefi

ts

and

har

ms)

Val

ues

and

pre

fere

nce

sEc

onom

ic

con

sid

-er

atio

ns/

Res

ourc

eu

se

Imp

act

ofva

ccin

atio

nA

ccep

tabi

lity

Feas

ibil

ity

Equ

ity

and

Eth

ical

con

sid

erat

ion

s

Lega

l an

dp

olit

ical

con

sid

erat

ion

s

Dec

isio

n-

mak

ing

Ah

med

et

al.,

2011

[17]

X

X

XB

lech

er

et

al.,

2012

[18]

Ngc

obo

and

Cam

eron

2012

[21]

X

X

X

X

X

Bro

oks

and

Ba

Ngu

z,

2012

[19]

Mil

stie

n

et

al.,

2010

[22]

X

X

X

X

X

Bry

son

et

al.,

2010

[26]

X

X

X

X

X

XB

urc

het

t

et

al.,

2012

[13]

X

X

X

X

X

X

X

XC

ho

2012

[25]

X

X

X

XD

ucl

os

et

al.,

2012

[20]

X

X

X

X

X

X

XLe

vin

e

et

al.,

2010

[23]

X

X

X

X

X

X

X

XM

akin

en

et

al.,

2012

[24]

X

X

X

XPi

atti

, 201

1

[27]

X

X

X

X

X

XPi

so

and

Wil

d, 2

009

[15]

X

X

X

X

X

X

X

XTa

pia

-Con

yer

et

al.,

2013

[16]

X

X

X

X

X

X

X

Tota

l

11

12

1

12

4

3

7 6

5

10

ine 33 (2015) 1206–1217

Many of the dimensions and criteria presented in the includedsystematic reviews and primary studies were broad, lacking practi-cal details for effective application in the evaluation or comparisonof vaccine strategies to guide vaccine adoption decision-making.The benefit and safety of vaccines, for example, were not suf-ficiently addressed. At the same time, most frameworks weregenerally qualitative: they did not report key issues such as thestudy designs to privilege, how to assess the risk of bias, how to ana-lyse benefits and risks (e.g., which relative and absolute measuresto use), the value to award patient reported outcome measures andthe minimal important differences.

In order to be effective in their implementation, dimensions andcriteria should have supporting documents that present summa-ries of quantitative and qualitative information with graphical andtabular displays.

Governmental immunisation technical advisory panels adviseregional and national authorities on decisions to introduce newvaccines or to adjust existing immunisation strategies. Theseauthorities are commissioned to perform their own analyses (e.g.,effectiveness or economic) based on existing studies that are notfully generalisable to local contexts due to differences such asdisease epidemiology, vaccine-specific issues (e.g., distributionand replacement of serotypes), magnitude of estimated herdimmunity effects, local immunisation strategies, effective cover-age, local health care systems, budgets and vaccination policies[29,30]. The panels are likely to face difficulty re-adapting orre-interpreting data pertaining to relevant vaccine dimensionsand criteria, which may introduce several problems and lowerthe quality of their recommendations. Policy makers must oftensupport recommendations on the introduction of new vaccineswithout a transparent, structured and independent assessmentprocess. This frequently occurs in situations characterised byconsiderable external pressure from the general public and thosewith vested interests [31–33]. Expert opinions may vary greatly asthe unstructured approach leads to selective reporting of dimen-sions that confirm a priori beliefs [34]. A structured approach isnecessary to guide stakeholders through the process, promptingthem to consider all available evidence, including less comfortabletopics such as the harms of the intervention [35].

The DECIDE project aims to provide empirically-based infor-mation on how recommendations are formulated and performedto support coverage decisions [8]. This approach is building onthe substantial experience and knowledge of the GRADE Work-ing Group [10]. In our study, we compared the DECIDE approachto other frameworks, revising it to specifically target vaccine adop-tion decisions. There is limited empirical research on how to conveydetailed information to policy makers on adopting and implemen-ting new vaccinations. No studies have examined the impact ofa decisional framework on vaccines for policy makers. However,the lack of any structured framework and mechanism to ensureequitable access to vaccines and other benefits from research oninfectious diseases raised controversies and tensions over the pastyears [36–38]. The limited results produced by these negotiationshave stimulated calls for a structured global framework to improveequitable access to vaccines [39].

4.1. Limitations

The main limitation of our study is the absence of a criticalappraisal of studies included in our framework. Thus, some of theconclusions of this review may come from studies that are sus-ceptible to bias. We also analysed the frequencies of dimensions

combining systematic reviews and primary studies. Dimensionsproposed by systematic reviews might have greater value thanthose proposed in primary studies. Moreover, we focused only onvaccine frameworks. The age of the primary studies included in the

M.

González-Lorenzo

et al.

/ V

accine 33

(2015) 1206–1217

1213

Table 3Criteria considered across conceptual frameworks and empirical approaches.

Dimension Criteria Studies mentioning criteria and frequency

Ahmedet al., 2011[17]

Blecheret al., 2012[18]

Ngcobo andCameron2012 [21]

Brooks andBa-Nguz2012 [19]Milstienet al., 2010[22]

Brysonet al. 2010[26]

Burchettet al. 2012[13]

Cho 2012[25]

Ducloset al., 2012[20]

Levineet al., 2010[23]

Makinenet al., 2012[24]

Piatti2011[27]

Piso andWild 2009[15]

Tapia-Conyeret al. 2013[16]

Total

Importance ofillness orproblem

Burden of disease data x x x x x x 6Prevalence x x x x 4Incidence x x x x x 5Mortality/Case fatalityrates

x x x x x 5

Morbidity x x x 3Social Impact x x x x x 5Hospitalisations x 1Risk factors x x 2Specific risk groups x x x x x 5Disease occurrence overtime (i.e., epidemic,secular trends)

x 1

Serogroups or serotypesdistribution

x x 2

Clinical features of thedisease

x x x 3

Clinical management x x 2Cost of disease x x 2Perceptions ofimportance

x x 2

Other preventivesmeasures

x 1

Vaccinecharacteristics(benefits andharms)

Vaccine characteristics x x 2Biological effects of thevaccine

x x x 3

Vaccine efficacy andeffectiveness

x x x x x x x x x x x 11

Vaccine coverage x 1Population impact of thevaccine

x 1

Herd immunity x 1Vaccine safety x x x x x x x x x x x 11Interaction with othervaccines

x 1

Indirect effects x 1Cold chain and logisticalconcerns

x x 2

Vaccine administrationschedule

x x x 3

Social and programmaticacceptability of theschedule

x 1

Values andpreferences

Relative importance ofoutcomes related tobenefits

x 1

1214

M.

González-Lorenzo

et al.

/ V

accine 33

(2015) 1206–1217

Table 3 (Continued)

Dimension Criteria Studies mentioning criteria and frequency

Ahmedet al., 2011[17]

Blecheret al., 2012[18]

Ngcobo andCameron2012 [21]

Brooks andBa-Nguz2012 [19]Milstienet al., 2010[22]

Brysonet al. 2010[26]

Burchettet al. 2012[13]

Cho 2012[25]

Ducloset al., 2012[20]

Levineet al., 2010[23]

Makinenet al., 2012[24]

Piatti2011[27]

Piso andWild 2009[15]

Tapia-Conyeret al. 2013[16]

Total

Relative importance ofoutcomes related toharms

x 1

Relative importance ofoutcomes related to cost

x 1

Economicconsidera-tions/Resourceuse

Health economicanalyses (e.g.,cost-benefit, cost-utility,cost-effectiveness)

x x x x x x x x x x x x 12

Direct and indirect cost x 1Vaccine and vaccinedelivery costs

x x x x x 5

Potential for vaccineprice reductions

x 1

Affordability ofimmunisation

x x x 3

Cost-effectiveness ofalternativesinterventions

x x x 3

Incremental costs x 1Funding sources x x 2Financial sustainability x x x 3

Impact ofvaccination

Impact on healthoutcomes

x x x 3

Impact on non-healthoutcomes

x x 2

Effect ofco-administration

x 1

Risk of serotypereplacement

x 1

Other impact x 1

Acceptability Acceptability of thevaccine

x x 2

Public perception x 1

Feasibility Feasibility of theimplementation of theprogramme

x x x x 4

Feasibility of localvaccine production andvaccine availability

x x x 3

Feasibility ofrecommendation

x x 2

Ability to reach the targetpopulations

x x 2

M.

González-Lorenzo

et al.

/ V

accine 33

(2015) 1206–1217

1215

Impact of vaccineadoption on the widerhealth system.

x x 2

Ability to monitorprogramme impact (i.e.,surveillance)

x x x x 4

Epidemiological changesof the disease (aftervaccine introduction)

x 1

Equity andEthicalconsiderations

Ethical considerations x x x 3Accessibility, equity andethics

x x x 3

Legal and politicalconsiderations

Legal considerations x x x x x 5

Decision-making

Decision-making process x x x x 4Evidence sources/qualityof evidences

x x 2

Impacts and coverage ofother interventions forthe disease

x x x 3

Population coverage x 1Priority x 1Actors involved x 1Procedures x 1Cues to action x 1Recommendations ofother countries

x 1

International guidelinesand expert advice

x 1

Political process x x 2

1216 M. González-Lorenzo et al. / Vaccine 33 (2015) 1206–1217

Table 4Proposed conceptual framework to support vaccine adoption and coverage decisions in a health system.

Dimensions Description Questions Criteria

Burden of disease Description of epidemiology,clinical features and sequelae ofthe disease/condition in terms ofpublic health consequences.

Is the disease/condition severe?Is the disease/condition frequent?Is the vaccination a priority?

- Frequency of the disease/condition (e.g., incidence,prevalence, secular trends).- Severity of the disease/condition (e.g., mortality,morbidity).- Social impact of the disease/condition (e.g.,hospitalisation rate, sickness absenteeism, high-riskgroups, clinical features, perception of importance,other preventive measures)

Vaccine characteristicsand impact ofimmunisationprogramme

Description of the effects andadverse events of the vaccine;using the GRADE method.Overall quality of the availableevidence of effects across all of theoutcomes, which are critical tomaking a decision.

Are the desirable anticipatedeffects large?Are the undesirable anticipatedeffects small?What is the net benefit of thevaccination?What is the overall certainty of thisevidence (e.g., how confident weare about the net benefit of thevaccination)?

- Vaccine characteristics or properties (e.g.,components, types, target population, posology).- Efficacy (e.g., immunogenicity, strain coverage,capacity to reduce the disease incidence, capacity todisrupt carriage, duration of protection, serotypereplacements).- Safety (e.g., reactogenicity, adverse events,interaction with other vaccines)

Values and preferences Consideration of values andpreferences of patients/care giversabout the balance betweendesirable and undesirable effects ofthe vaccine.

How certain is the relativeimportance of the desirable andundesirable outcomes?Would patients/caregivers feel thatthe benefits outweigh the harmsand burden?What is the appreciation and valueof the vaccination in thepopulation?

- Values and preferences of citizens about the balancebetween desirable and - undesirable effects of thevaccine.- Perspectives and perceptions of the citizens andhealth professionals about the disease and the vaccine.

Resource use All the information about costs, useof resources and health outcomesgained.

Is the incremental cost smallrelative to the net benefits?Is the total cost (impact on budget)small?What are the costs of thevaccination and are they limitedcompared to the benefits?

- Vaccination costs (e.g., costs of the vaccine,administration costs, costs arising from potentialadverse effects).- Budget impact and financial sustainability.- Health economic analyses (e.g., cost-effectivenessanalysis, cost-benefit analysis, cost-utility analysis)- Direct and indirect costs.

Equity Impact on health inequities andethical considerations.

What would be the impact onhealth inequities?Would some part of the populationtaking advantage from thevaccination compared to othergroups?

- Ethical considerations- Equity (e.g., accessibility; equal distribution ofresources, benefits risks, costs, etc. related to thevaccination programme).

Feasibility Information on applicability andpossible barriers, acceptability,organisational impact, alternativescenarios, control system.

Is the option feasible to adoption inthe actual setting?Which vaccination barriers orfacilitators act at the system level?

- Acceptability of the vaccination among thepopulation and health care professionals.- Feasibility of the implementation of the programme(e.g., vaccination coverage, ability to reach thepopulation target, vaccine availability and supply,recommendation).- Alternative interventions (e.g., effectiveness andcost-effectiveness of alternatives).- Surveillance system.

sdomaadrvdFmTtdG

ew

ystematic reviews of frameworks is limited. Most studies wereeveloped only in the last 15 years. Most reviews incorporatedverlapping contents with a narrow focus. However, decision-akers may differ in the way they conceptualise their problems

nd approach decisions given the availability of support tools suit-ble for use in other clinical areas as well. If vaccines are notifferent from other interventions, particularly drugs, a systematiceview with a broader perspective might have added other rele-ant elements. Our work serves as a starting point to define andevelop a usable framework to inform vaccine coverage decisions.uture directions involve addressing the above conceptual andethodological issues and increasing their broader applicability.

his process should include key stakeholders as well interna-ional groups leading the development of support tools for clinicalecision-making such as the GRADE and the DECIDE Working

roups [8,10].

Another consideration is that decisional frameworks quicklyvolve, shaped by the changes in professional and social contexts asell as secular trends. For example, the EtD framework we used as

a comparison tool was released as an interim version in December2013 [8]; the finalised version has yet to be released and mightdiffer from the interim version.

5. Conclusions

The prospect for a framework to support coverage decisionson vaccines is promising as national interests favour equitableaccess to vaccines and drugs and reduce the threat posed by pre-ventable diseases. Our study identified a number of frameworkswith a variety of dimensions. Burden of illness or problem, vac-cine characteristics, values and preferences, resource use, equityand feasibility are key factors. Our review highlights limitationsin the dimensions and criteria as well their lack of detailed and

pragmatic instructions for effective application in the evaluationor comparison of vaccine strategies. Moving forward, future stud-ies should evaluate ways to effectively communicate and supportthe uptake of evidence pertaining to the selected dimensions and

/ Vacc

cai

C

A

tdtym

S

asP(

A

Od

A

i1

R

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

[

M. González-Lorenzo et al.

riteria, providing empirically-based and user-tested informationbout alternative ways to present criteria and their perceived utilityn supporting policy decision-making.

onflict of interest

The authors declare that they have no competing interests.

uthors’ contributions

MGL, AP, LC, MG, FA and LM conceived of the study. VP par-icipated in the search strategy. MGL and LM participated in theesign of the study, performed data extraction and coordination ofhe study. MGL, LM, AM, VD, EP, MT and AZ participated in the anal-sis. All authors read, provided feedback, and approved the finalanuscript.

upport

The research leading to these results was primarily supported by grant from the Regione Lombardia, Italy. Elena Parmelli receivedupport from the European Community’s Seventh Frameworkrogramme (FP7/2007–2013) under grant agreement n◦ 258583DECIDE project).

cknowledgements

We thank Koren H. Kwag for their copyediting support, and Andyxman and Stefanos Bonovas for their comments on a previousraft of this essay.

ppendix A. Supplementary data

Supplementary data associated with this article can be found,n the online version, at http://dx.doi.org/10.1016/j.vaccine.2014.2.020.

eferences

[1] Manzoli L, Flacco ME, D’Addario M, Capasso L, De Vito C, Marzuillo C, et al. Non-publication and delayed publication of randomized trials on vaccines: survey.Br Med J 2014;348:3058.

[2] Walker CL, Rudan I, Liu L, Nair H, Theodoratou E, Bhutta ZA, et al. Global burdenof childhood pneumonia and diarrhoea. Lancet 2013;381(9875):1405–16.

[3] Whitney CG, Zhou F, Singleton J, Schuchat A. Benefits from immunization dur-ing the vaccines for children program era—United States, 1994–2013. MorbMortal Wkly Rep 2014;63(16):352–5.

[4] Hutubessy R, Henao AM, Namgyal P, Moorthy V, Hombach J. Results from evalu-ations of models and cost-effectiveness tools to support introduction decisionsfor new vaccines need critical appraisal. Bio Med Cent Med 2011;9:55.

[5] World Health Organisation. Vaccine introduction guidelines. Adding a vaccineto a national immunisation program: decision and implementation; 2010.

[6] World Health Organisation. Principles and considerations for adding a vaccineto a national immunization programme. From decision to implementation andmonitoring; 2014.

[7] CDC global health global immunization strategic framework 2011–2015.〈http://www.cdc.gov/globalhealth/gid/framework/.〉.

[8] Treweek S, Oxman AD, Alderson P, Bossuyt PM, Brandt L, Brozek J, et al. Develop-ing and Evaluating Communication Strategies to Support Informed Decisionsand Practice Based on Evidence (DECIDE): protocol and preliminary results.Implement Sci 2013;8:6.

[9] Parmelli E, Amato L, Saitto C, Davoli M. DECIDE: developing and evaluatingcommunication strategies to support informed decisions and practice basedon evidence. Recenti Prog Med 2013;104(10):522–31.

10] Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al.

GRADE: an emerging consensus on rating quality of evidence and strength ofrecommendations. Br Med J 2008;336(7650):924–6.

11] Work Package 2: Policymaker and manager focussed strategies forcommunicating evidence-based recommendations. 〈http://www.decide-collaboration.eu/WP2.〉.

[

ine 33 (2015) 1206–1217 1217

12] Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for sys-tematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med2009;151(4):264–9. W64.

13] Burchett HE, Mounier-Jack S, Griffiths UK, Mills AJ. National decision-making onadopting new vaccines: a systematic review. Health Policy Plan 2012;27(Suppl.2):ii62–76.

14] Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al.The PRISMA statement for reporting systematic reviews and meta-analyses ofstudies that evaluate healthcare interventions: explanation and elaboration. BrMed J 2009;339:b2700.

15] Piso B, Wild C. Decision support in vaccination policies. Vaccine2009;27(43):5923–8.

16] Tapia-Conyer R, Betancourt-Cravioto M, Saucedo-Martinez R, Motta-MurguiaL, Gallardo-Rincon H. Strengthening vaccination policies in Latin America: anevidence-based approach. Vaccine 2013;31(37):3826–33.

17] Ahmed F, Temte JL, Campos-Outcalt D, Schunemann HJ. Methods for developingevidence-based recommendations by the Advisory Committee on Immuniza-tion Practices (ACIP) of the U.S. Centers for Disease Control and Prevention(CDC). Vaccine 2011;29(49):9171–6.

18] Blecher MS, Meheus F, Kollipara A, Hecht R, Cameron NA, Pillay Y, et al.Financing vaccinations—the South African experience. Vaccine 2012;30(Suppl.3):SC79–86.

19] Brooks A, Ba-Nguz A. Country planning for health interventions under devel-opment: lessons from the malaria vaccine decision-making framework andimplications for other new interventions. Health Policy Plan 2012;27(Suppl.2):ii50–61.

20] Duclos P, Durrheim DN, Reingold AL, Bhutta ZA, Vannice K, Rees H. Devel-oping evidence-based immunization recommendations and GRADE. Vaccine2012;31(1):12–9.

21] Ngcobo NJ, Cameron NA. The decision making process on new vaccines intro-duction in South Africa. Vaccine 2012;30(Suppl. 3):SC9–13.

22] Milstien J, Cardenas V, Cheyne J, Brooks A. WHO policy development processesfor a new vaccine: case study of malaria vaccines. Malar J 2010;9:182.

23] Levine OS, Hajjeh R, Wecker J, Cherian T, O’Brien KL, Knoll MD, et al. Apolicy framework for accelerating adoption of new vaccines. Hum Vaccine2010;6(12):1021–4.

24] Makinen M, Kaddar M, Molldrem V, Wilson L. New vaccine adoption in lower-middle-income countries. Health Policy Plan 2012;27(Suppl. 2):ii39–49.

25] Cho HY. An overview of the national immunization policy making process: therole of the Korea expert committee on immunization practices. Korean J Pediatr2012;55(1):1–5.

26] Bryson M, Duclos P, Jolly A, Bryson J. A systematic review of national immu-nization policy making processes. Vaccine 2010;28(Suppl. 1):SA6–12.

27] Piatti A. Decisional aid tool for vaccine policy in Lombardia, Italy. Milan: Uni-versity of Milan; 2011.

28] Vogel JP, Oxman AD, Glenton C, Rosenbaum S, Lewin S, Gulmezoglu AM, et al.Policymakers’ and other stakeholders’ perceptions of key considerations forhealth system decisions and the presentation of evidence to inform those con-siderations: an international survey. Health Res Policy Syst 2013;11:19.

29] Hamers RL, Schuurman R, van Vugt M, Derdelinckx I, Rinke de Wit TF.AIDS treatment in Africa: the risk of antiretroviral resistance. Ned TijdschriftGeneeskd 2007;151(48):2666–71.

30] Burns JE, Mitrovich RC, Jauregui B, Matus CR, Andrus JK. Descriptive analysisof immunization policy decision making in the Americas. Rev Panam SaludPublica 2009;26(5):398–404.

31] Andrus JK, Toscano CM, Lewis M, Oliveira L, Ropero AM, Davila M, et al. A modelfor enhancing evidence-based capacity to make informed policy decisions onthe introduction of new vaccines in the Americas: PAHO’s ProVac initiative.Public Health Rep 2007;122(6):811–6.

32] Beutels P, Jit M. A brief history of economic evaluation for human papillo-mavirus vaccination policy. Sex Health 2010;7(3):352–8.

33] Haas M, Ashton T, Blum K, Christiansen T, Conis E, Crivelli L, et al. Drugs,sex, money and power: an HPV vaccine case study. Health Policy 2009;92(2-3):288–95.

34] Gotzsche PC, Ioannidis JP. Content area experts as authors: helpful orharmful for systematic reviews and meta-analyses? Br Med J 2012;345:e7031.

35] Ioannidis JP, Evans SJ, Gotzsche PC, O’Neill RT, Altman DG, Schulz K, et al. Bet-ter reporting of harms in randomized trials: an extension of the CONSORTstatement. Ann Int Med 2004;141(10):781–8.

36] Ashbaugh AR, Herbert CF, Saimon E, Azoulay N, Olivera-Figueroa L, Brunet A.The decision to vaccinate or not during the H1N1 pandemic: selecting the lesserof two evils? Plos One 2013;8(3):e58852.

37] Fedson DS, Opal SM. The controversy over H5N1 transmissibility research: anopportunity to define a practical response to a global threat. Hum VaccinesImmunother 2013;9(5):977–86.

38] Kennedy C, Gray Brunton C, Hogg R. Just that little bit of doubt’: Scottish par-

ents’, teenage girls’ and health professionals’ views of the MMR, H1N1 and HPVvaccines. Int J Behav Med 2014;21(1):3–10.

39] Fidler DP. Negotiating equitable access to influenza vaccines: global healthdiplomacy and the controversies surrounding avian influenza H5N1 and pan-demic influenza H1N1. Plos Med 2010;7(5):e1000247.