Specifying and comparing implementation strategies across

RESEARCH Open Access

Specifying and comparing implementationstrategies across seven largeimplementation interventions: a practicalapplication of theoryCynthia K. Perry1* , Laura J. Damschroder2,3, Jennifer R. Hemler4, Tanisha T. Woodson5, Sarah S. Ono5 andDeborah J. Cohen5

Abstract

Background: The use of implementation strategies is an active and purposive approach to translate researchfindings into routine clinical care. The Expert Recommendations for Implementing Change (ERIC) identified anddefined discrete implementation strategies, and Proctor and colleagues have made recommendations for specifyingoperationalization of each strategy. We use empirical data to test how the ERIC taxonomy applies to a largedissemination and implementation initiative aimed at taking cardiac prevention to scale in primary care practice.

Methods: EvidenceNOW is an Agency for Healthcare Research and Quality initiative that funded seven cooperativesacross seven regions in the USA. Cooperatives implemented multi-component interventions to improve hearthealth and build quality improvement capacity, and used a range of implementation strategies to foster practicechange. We used ERIC to identify cooperatives’ implementation strategies and specified the actor, action, target,dose, temporality, justification, and expected outcome for each. We mapped and compiled a matrix of the specifiedERIC strategies across the cooperatives, and used consensus to resolve mapping differences. We then groupedimplementation strategies by outcomes and justifications, which led to insights regarding the use of and linkagesbetween ERIC strategies in real-world scale-up efforts.

Results: Thirty-three ERIC strategies were used by cooperatives. We identified a range of revisions to the ERICtaxonomy to improve the practical application of these strategies. These proposed changes include revisions tofour strategy names and 12 definitions. We suggest adding three new strategies because they encapsulate distinctactions that were not described in the existing ERIC taxonomy. In addition, we organized ERIC implementationstrategies into four functional groupings based on the way we observed them being applied in practice. Thesegroupings show how ERIC strategies are, out of necessity, interconnected, to achieve the work involved in rapidlytaking evidence to scale.

Conclusions: Findings of our work suggest revisions to the ERIC implementation strategies to reflect theirutilization in real-work dissemination and implementation efforts. The functional groupings of the ERICimplementation strategies that emerged from on-the-ground implementers will help guide others in choosingamong and linking multiple implementation strategies when planning small- and large-scale implementationefforts.

(Continued on next page)

© The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, andreproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link tothe Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

* Correspondence: [email protected] of Nursing, Oregon Health & Science University, 3455 SW USVeterans Hospital Rd, Portland, OR 97239, USAFull list of author information is available at the end of the article

Perry et al. Implementation Science (2019) 14:32 https://doi.org/10.1186/s13012-019-0876-4

http://crossmark.crossref.org/dialog/?doi=10.1186/s13012-019-0876-4&domain=pdf

http://orcid.org/0000-0002-8167-5936

http://creativecommons.org/licenses/by/4.0/

http://creativecommons.org/publicdomain/zero/1.0/

mailto:[email protected]

(Continued from previous page)

Trial registration: Registered as Observational Study at www.clinicaltrials.gov (NCT02560428).

Keywords: Implementation strategy mapping, Implementation strategies, Large-scale initiative, Capacity building,Implementation facilitation,

BackgroundIt is well recognized that an active and purposive approachis required to translate research findings into routine clin-ical care. Methods used to disseminate and implement re-search findings into clinical practice are termedimplementation strategies, [1] and interventions designedto facilitate the application of evidence into clinical prac-tice typically use a combination of implementation strat-egies tailored to a particular clinical context. Multipleimplementation frameworks [2, 3] provide guidance onhow to define factors that may explain or predict imple-mentation outcomes and how to assess implementationinterventions (e.g., Consolidated Framework for Imple-mentation Research (CFIR) [4]; Promoting Action on Re-search in Health Services (PARIHS) [5]; NormalizationProcess Theory (NPT) [6]). These frameworks providetheoretical structures and are an indispensable foundationfor developing testable hypothesized models, establishingan understanding of the factors that impact implementa-tion, and building a base of implementation knowledge [7,8]. However, these frameworks may not necessarily pro-vide guidance about how to organize detailed implementa-tion strategies to achieve specific outcomes, may useinconsistent names for implementation strategies, andmay lack adequate strategy descriptions—making under-standing and replication difficult.Powell and colleagues sought to address the problem

of multiple names and definitions for implementationstrategies in 2012 [9]. Through a comprehensive reviewof the literature, they identified 68 discrete strategiesand organized them into six key implementation do-mains. These domains were organized by type, withsimilar implementation strategies grouped into the samedomain. They then updated and expanded this list byengaging a broad group of implementation scientists ina three-phase, modified-Delphi process that refined andexpanded the list to 73 discrete implementation strat-egies, each with a name and definition [10]. These 73strategies were grouped into nine clusters based uponstrategy type, and ratings of importance (high/low) andfeasibility (high/low) were elicited for each strategy [11].This updated compilation is known as the Expert Rec-ommendations for Implementing Change (ERIC).ERIC has made an important theoretical contribution

to identifying a broad range of implementation strategiesand developing a common nomenclature with defini-tions. An important next step toward advancing this

work is to test and refine the taxonomy based on theuse of these implementation strategies in on-the-groundimplementation efforts; Proctor and colleagues [1] rec-ognized this need and have started this work. They haverecommended that implementation strategies be furtherspecified to include such details as the actor (who isenacting the implementation strategy), the actions (spe-cific activities to support implementation), the targets(entity impacted), temporality, dose, expected outcome,and justification. Researchers have used the ERIC andProctor specifications to understand the operationaliza-tion of implementation strategies and have found thisuseful [12–16]. The level of specification has been help-ful in identifying how strategies can be tailored to differ-ent settings [14], fostering comparison across clinicalsites in a multisite study [16], and helping to inform thedevelopment of activity logs to track implementationstrategies [13].Researchers who applied the ERIC and Proctor frame-

works to real-world implementation efforts recom-mended additional implementation strategies for theERIC taxonomy [13]. Others have used ERIC to guidedetailed mapping of implementation strategies duringimplementation study planning [12]; one of these studiesused ERIC to develop a blueprint to guide implementa-tion efforts [15]. As all implementers know, however,there is a big difference between what you propose orplan to do during implementation and what really hap-pens. To date, few studies (and none we could identify)use data from large-scale, multi-site implementation ef-forts to apply and refine the ERIC taxonomy.This paper reports a study we conducted to apply the

ERIC taxonomy and the Proctor reporting recommenda-tions to multiple large-scale studies. We use empiricaldata collected during active implementation with an in-tent of further refining ERIC, if changes were identifiedas necessary.

MethodsSettingThe setting for this study is the EvidenceNOW initiative[17] funded by the Agency for Healthcare Research andQuality (AHRQ). EvidenceNOW focused on rapid dis-semination and implementation of cardiovascular pre-ventive care (appropriate aspirin use, blood pressure andcholesterol management, and smoking cessation—theABCS of heart health) among smaller primary care

Perry et al. Implementation Science (2019) 14:32 Page 2 of 13

https://clinicaltrials.gov/ct2/show/NCT02560428

practices as well as capacity building (e.g., quality im-provement experience, data access, practice leadership).In the request for applications, AHRQ “strongly encour-aged” the use of specific implementation strategies, in-cluding practice facilitation, data review and feedbackand benchmarking, peer-to-peer learning, and expertconsultation [18], and each cooperative was required toproduce clinical quality measures and practice capacitymeasurement to determine change over time in thesedomains. In addition to funding seven cooperatives,AHRQ also funded a national evaluation of Evidence-NOW called Evaluating System Change to AdvanceLearning and Take Evidence to Scale (ESCALATES) toharmonize and coordinate the collection of quantitativedata, to collect additional qualitative data, and to bringtogether the cross-cooperative comparative findings. Formore details on the ESCALATES evaluation, see Cohenet al. [19].

SampleEvidenceNOW funded seven regional cooperatives thatspanned 12 states (Oregon, Washington, Idaho, Colorado,New Mexico, Oklahoma, Wisconsin, Illinois, Indiana,New York City, North Carolina, Virginia). For details onthe regions and states, see the EvidenceNOW website [17]and Ono et al. [20]. Each cooperative engaged over 200primary care practices in this effort (N = 1721). Practiceswere smaller (< 10 clinicians) and varied with respect toownership and geographic location. Accomplishing thegoal of large-scale rapid dissemination and implementa-tion in less than 3 years required a large workforce anduse of a range of implementation strategies.

Data sourcesWe reviewed and triangulated four sources of qualitativedata from years one and two of the EvidenceNOW ini-tiative to identify and define the cooperatives’ implemen-tation strategies. First, we reviewed the cooperatives’study proposals, which provided an outline of broad,overarching implementation strategies. Second, we ex-amined cooperative-level documents, including trainingtools and conceptual models, to further specify contentof these overarching strategies. We also analyzed facilita-tor training documents, which provided insights intospecific activities and implementation strategies that fa-cilitators would use to support practices. For example,templates for facilitators to document and track theirwork provided detailed descriptions of activities that fa-cilitators may perform. The third source of data wereentries written by cooperative team members on the ES-CALATES interactive online diary [21]. The online diary[21] is a web-based platform that allowed our evaluationteam to communicate with each cooperative team abouttheir implementation experiences and clarify details

about on-the-ground implementation support they wereproviding to practices. These entries were made duringactive implementation of the interventions across coop-eratives. The fourth data source analyzed comprised fieldnotes from site visits to cooperatives and transcriptsfrom interviews with cooperative leadership and keyteam members during active implementation. Analyzinginterview transcripts and field notes provided importantinsight into the pragmatic, on-the-ground use of imple-mentation strategies to support practice change.All field notes were prepared soon after the site visit.

Interviews followed a semi-structured guide and wereprofessionally transcribed, checked for accuracy, andde-identified. Field notes, interviews, and other docu-ments were entered into Atlas.ti for data managementand analysis.

Data analysisWe used an iterative and inductive analytic approach[22] to identify and describe cooperatives’ implementa-tion strategies. Guided by Proctor and colleagues’ rec-ommendations for specifying and reporting details onimplementation strategies [1], we constructed an inter-vention table for each cooperative describing the broadimplementation strategies and more detailed activitiesthat cooperatives included in their interventions andspecified the actors, action targets, dose, temporality, ex-pected outcomes, and justification for each. Wemember-checked these tables with the cooperative lead-ership and refined these documents, as needed. (seeAdditional file 1 for an example of one such table). Fromthese seven intervention tables, we identified 266 de-tailed actions that were documented by cooperativemembers to help practices implement changes to im-prove capacity or ABCS.Next, four of the authors (LJD, CKP, JRH, TTW) inde-

pendently mapped each action to a strategy in the ERICcompilation [10] based on alignment of each action withthe definition for each ERIC strategy. We met weeklyover 5 months to independently map and discuss map-pings until we reached consensus. Throughout thisprocess, we operationalized ERIC definitions to helpguide decisions about alignment of ERIC implementa-tion strategies with the EvidenceNOW actions and iden-tified areas where ERIC definition expansion, revision,and/or reorganization was needed to better reflect theactions cooperatives used and how that could help makethe ERIC strategies more pragmatic and easily appliedby researchers and practitioners.Next, we created a consolidated cross-cooperative

matrix that organized implementation strategies basedupon their practical application. We pile-sorted each im-plementation strategy into functionally similar groupsbased on their justification and expected outcomes. We


reviewed and labeled each grouping and considered thecooperatives’ practical applications of strategies withinand across the groupings, recognizing interdependenciesand differences, such as “actors” (who enacts the strat-egies within a grouping; facilitators with health informa-tion technology (HIT) expertise versuscooperative-employed data experts).

ResultsThe 266 actions identified across the seven cooperativesmapped to 33 of the 73 ERIC implementation strategies.We propose refinements for 13 strategies. Table 1 liststhe ERIC strategies, our proposed changes, and our ra-tionale for these changes. Recommended refinements in-clude changes in strategy names for four strategies andchanges in definitions for 12 strategies. Additionally, wepropose adding three new distinct strategies which weidentified as being used in practices among the coopera-tives: assess and redesign workflow, create online learningcommunities, and engage community resources. Defini-tions for these strategies are provided in Table 1. TheERIC strategies taxonomy included “Ancillary Material”that the authors included as an Additional file [10]. Werecommend changes in ancillary material for 15 strat-egies and provide ancillary material for the three newstrategies; details are provided in our Additional file 2.We will refer to ERIC strategies using our new proposedlabels when applicable (e.g., implementation facilitationinstead of ERIC’s original name of facilitation).We grouped the 33 ERIC strategies used by the coop-

eratives into four functional groupings: (1) build healthinformation technology to support data-informed qualityimprovement (QI), (2) build QI capacity and improveoutcomes, (3) enhance clinician and practice memberknowledge, and (4) build community connections andpatient involvement. Tables 2, 3, 4, and 5 list implemen-tation strategies and identify the ERIC cluster to whicheach strategy belongs along with the actor, specific ac-tions, and targets; each table covers of one of the fourgroupings listed above. The following sections describethe four overarching functions and the implementationstrategies within each grouping.

Build health information technology to support data-informed QIMultiple implementation strategies are needed toachieve data-informed QI. A critical aspect of the coop-eratives’ efforts to rapidly disseminate and implementevidence into practice was helping practices gain accessto data to support data-informed QI functions (seeTable 2). All cooperatives included an audit and providefeedback strategy to support data-informed QI. Thisstrategy relied on region- and practice-level HIT infra-structure to deliver trusted data to practices at regular

intervals throughout intervention. Audit functions (e.g.,ability to produce ABCS performance reports) were ne-cessary for feedback functions (e.g., communication ofthe audit to clinicians and staff ), which in turn, were ne-cessary to inform QI efforts. For instance, generated datawere used to identify quality gaps and help to identifyopportunities on which to focus improvement efforts.These data were also used to monitor the impact thatchanges had on ABCS outcomes. Audit and providefeedback, thus, was a key strategy to improve ABCSoutcomes.On the ground, up to seven additional ERIC imple-

mentation strategies were necessary to accomplish build-ing the infrastructure and capacity needed to implementand sustain audit and provide feedback functions, asshown in Table 2. All but one cooperative sought to de-liver audit and provide feedback functions in a way thatcould be sustained past the time of their funded projectperiod. The degree to which these seven additional strat-egies were used by the cooperatives was based on the ro-bustness of existing HIT infrastructure in their region.For example, one cooperative used all seven strategieslisted in Table 2, which required significant time and in-vestment. This cooperative hired HIT experts to connectpractices’ electronic health record (EHR) systems withexternal registries (use data warehousing techniques)and to develop dashboards—available through central-ized portals (develop and organize quality monitoringsystems)—to provide practices with quality reportsneeded to support audit and provide feedback functions.At times, HIT experts (or an HIT-trained facilitator)worked with practices to change how the practice mem-bers documented information from appointments andother sources within their EHR to ensure valid andcomplete data were extracted (change records systems).This step was necessary to get good quality data to acentralized data warehouse, and for the warehouse thento provide the application and cleaned data to supportquality reports, which were provided back to practices.This was necessary because many practices did not havethe capability to generate robust and useable reportswithin their own setting. The local technical assistanceHIT experts provided were extended to assisting prac-tices with translating information from reports to action-able opportunities for improvement related to thegenerated performance reports and monitoring of pro-gress over time.

Build QI capacity and improve outcomesPractice facilitators used a wide range of implementationstrategies to build QI capacity and improve outcomes(see Table 3). Facilitation was the central implementationstrategy used by all cooperatives. Practice facilitators,who were actors in this process, used many different


Table 1 Proposed changes to ERIC strategy labels and/or definitions with rational for changes (proposed changes are in italics)

ERIC name Current ERIC definition Proposed changes to definition Rationale for proposed change

Use data experts Involve, hire, and/or consult experts toinform management on the use of datagenerated by implementation efforts

Involve, hire, and/or consult experts toacquire, structure, manage, report, and usedata generated by implementationefforts

We broadened functions of data expertsbeyond just management of data.

Fund and contractand/or negotiatewith vendors for theclinical innovation

Governments and other payers ofservices issue requests for proposals todeliver the innovation, use contractingprocesses to motivate providers todeliver the clinical innovation, anddevelop new funding formulas thatmake it more likely that providers willdeliver the innovation

None We broadened the name to include therole of negotiation. Having outsideassistance to negotiate with EHRvendors can be valuable in addition topayment.

Provide localtechnical assistance

Develop and use a system to delivertechnical assistance focused onimplementation issues using localpersonnel

Develop and use a system to delivertechnical assistance within local settingsthat is focused on implementation issues

We clarified the definition to indicateany technical assistance provided in thelocal setting, whether provided by localstaff or by other on-site individuals.

Audit and providefeedback

Collect and summarize clinicalperformance data over a specified timeperiod and give it to clinicians andadministrators to monitor, evaluate, andmodify provider behavior

Develop summaries of clinicalperformance over a specific time period,often including a comparator, and give itto clinicians and/or administrators.Summary content (e.g., nature of the data,choice of comparator) and their delivery(e.g., mode, format) are designed tomodify specifically targeted behavior(s) oractions of individual practitioners, teams,or health care organizations

We broadened the definition to includeproviding comparator data (benchmark)and to indicate that goal is to modify atargeted behavior and/or action ofmultiple actors. These facets are listed inancillary materials but were notincluded in the published definition.

Use animplementationadvisor

Seek guidance from experts inimplementation

Seek guidance from experts inimplementation, including providingsupport and training for theimplementation work force

We broadened this definition to includeproviding support and training forfacilitators.

Implementationfacilitation

A process of interactive problem solvingand support that occurs in a context of arecognized need for improvement and asupportive interpersonal relationship

[A] multi-faceted interactive process ofproblem solving, enabling and supportingindividuals, groups and organizations intheir efforts to adopt and incorporateinnovations into routine practices thatoccurs in a context of a recognized needfor improvement and a supportiveinterpersonal relationship

The name was changed to specify“implementation” because facilitation isa very broad concept. Implementationfacilitation includes practice facilitation,a more specific type of implementationfacilitation.We broadened the definition toacknowledge that facilitation is morethan just “interactive problem-solving.”

Assess for readinessand identify barriersand facilitators

Assess various aspects of an organizationto determine its degree of readiness toimplement, barriers that may impedeimplementation, and strengths that canbe used in the implementation effort

Assess various aspects of an organizationto determine its degree of readiness toimplement and identify barriers that mayimpede implementation and strengthsthat can be leveraged to facilitate theimplementation effort

We revised to clarify identification ofbarriers and leveraging facilitators.

Develop animplementationblueprint

Develop a formal implementationblueprint that includes all goals andstrategies. The blueprint should includethe following: (1) aim/purpose of theimplementation; (2) scope of the change(e.g., what organizational units areaffected); (3) timeframe and milestones;and (4) appropriate performance/progress measures. Use and update thisplan to guide the implementation effortover time

Develop a formal implementationblueprint that includes all goals andstrategies. The blueprint should includethe following: (1) aim/purpose of theimplementation; (2) scope of the change(e.g., what organizational units areaffected); (3) timeframe and milestones;and (4) appropriate performance/progress measures. Use and update thisplan to guide the implementation effortover time

We suggest deleting the word “Formal”to include informal as well as formalimplementation blueprints. This willinclude plans that are developed forquality improvement as well as largerformal plans for implementation.Additionally, the definition is expandedto explicitly acknowledge its role inguiding implementation over time.

Organizeimplementationteams and teammeetings

Develop and support teams of clinicianswho are implementing the innovationand give them protected time to reflecton the implementation effort, sharelessons learned, and support oneanother’s learning

Develop and support teams of clinicians,staff, patients and other stakeholders whoare implementing or may be users of theinnovation. Provide protected time forteams to reflect on the implementationprogress, share lessons learned, makerefinements to plans, and support oneanother’s learning

We broadened the name to include allpossible team members and to includeformation of teams as well as meetings.Removing the term clinician allows for amulti-disciplinary team and increases en-gagement among all team members.We broadened definition to be inclusiveof all team members and clarified intent


implementation strategies identified by ERIC to promotepractice change and to improve capacity for conductingregular QI to improve ABCS outcomes. For example, fa-cilitators assisted practices in engaging clinicians, prac-tice members, and leaders in practice change efforts(identify and prepare champions); organized and facili-tated meetings (organize implementation team and team

meetings); assessed and offered suggestions for revisingclinical work flows (assess and redesign workflow);assessed HIT needs and data set-up (assess for readinessand identify barriers and facilitators); discussed andidentified an improvement plan (develop an implementa-tion blueprint); assisted with the change process andsupported the change efforts including pre-visit planning

Table 1 Proposed changes to ERIC strategy labels and/or definitions with rational for changes (proposed changes are in italics)(Continued)

ERIC name Current ERIC definition Proposed changes to definition Rationale for proposed change

of the definition.

Develop educationalmaterials

Develop and format manuals, toolkits,and other supporting materials in waysthat make it easier for stakeholders tolearn about the innovation and forclinicians to learn how to deliver theclinical innovation

Develop and format manuals, toolkits,and other supporting materials to makeit easier for stakeholders to learn aboutthe innovation and for clinicians to learnhow to deliver the clinical innovation.This can include technology-delivered (e.g.,online/smartphone-based static ordynamic) content and health messaging

We expanded to include technology-delivered content and messaging.

Conducteducationaloutreach visits

Have a trained person meet withproviders in their practice settings toeducate providers about the clinicalinnovation with the intent of changingthe provider’s practice

Have a trained person meet withindividuals or teams in their work settingsto educate them about the clinicalinnovation with the intent of changingbehavior to reliably use the clinicalinnovation as designed

We broadened definition to includeteam members beyond providers andclarified language to more clearly statethat this strategy aims to encouragesustained use of the innovation.

Conduct ongoingtraining

Plan for and conduct training in theclinical innovation in an ongoing way

Plan for and conduct training in theclinical innovation in an ongoing way forall individuals involved withimplementation and users of the clinicalinnovation e.g., clinicians, implementationstaff, practice facilitators

We expanded the definition to includeall individuals involved.

Conducteducationalmeetings

Hold meetings targeted toward differentstakeholder groups (e.g., providers,administrators, other organizationalstakeholders, and community, patient/consumer, and family stakeholders) toteach them about the clinical innovation

Hold meetings targeted towardeducating multiple stakeholder groups(i.e., providers, administrators, otherorganizational stakeholders, communitymembers, patients/consumers, families)about the clinical innovation and/or itsimplementation

We revised the definition to addspecificity about the purpose of theeducation (the innovation and/or itsimplementation) and to clarify thateducation is among multiple types ofstakeholders.

Assess and redesignworkflow

Observe and map current workprocesses and plan for desired workprocesses, identifying changes necessaryto accommodate, encourage, orincentivize use of the clinical innovationas designed

New: Added as this work is not reflectedin current ERIC strategies.

Create onlinelearningcommunities

Create an online portal for clinical staffmembers to share and access resources,webinars, and FAQs related to thespecific evidenced-based intervention,and provide interactive features to en-courage learning across settings andteams, e.g., regular blogs, facilitated dis-cussion boards, access to experts, andnetworking opportunities


Engage communityresources

Connect practices and their patients tocommunity resources outside thepractice (e.g., state and county healthdepartments; non-profit organizations; re-sources related to addressing the socialdeterminants of health; and organiza-tions focused on self-management tech-niques and support


Please see Additional file 2 for complete list of proposed changes plus changes to “Ancillary Material” originally included in Additional File 6 published withPowell et al. (10)


and outreach (implementation facilitation); and helpedrevise plans based upon results of change cycles. As re-ported above, practice facilitators were also involved inhelping set up HIT infrastructure.

Enhance clinician and practice member knowledgeSeveral implementation strategies were employed to en-hance clinician and practice member knowledge ofevidence-based guidelines and interventions designed toimprove ABCS outcomes, measures to assess outcomes,and QI methods (see Table 4). All seven cooperatives in-cluded education implementation strategies (e.g., developeducational materials, distribute educational materials).Cooperatives employed a range of education strategies andmore than one strategy might be used by a cooperative.These also included implementation strategies aimed atstrengthening peer exchange of best practices to buildstronger, sustained supportive professional learning com-munities. In addition, we identified one new implementa-tion strategy used by EvidenceNOW cooperatives—createonline learning communities. Given the great distances

some cooperatives covered to reach practices, online learn-ing techniques were tested to improve connections betweenpractices for distance learning (e.g., online portals with in-formational resources, discussion boards, webinars).

Build community connections and patient involvementMultiple implementation strategies were used to buildcommunity connections and increase patient involve-ment in improving ABCS outcomes (see Table 5). Somecooperatives used implementation strategies to create orexpand community connections with the goal of betterconnecting patients, practices, and community resourcesto improve ABCS. For example, one cooperative devel-oped resource sharing agreements to forge links betweenpractices and state and county health departments. Fivecooperatives worked to engage community resources tobuild connections between practices and health relatedorganizations in the communities, including participa-tion in collaborative projects; this approach is not listedamong the ERIC strategies and is newly proposed. Twocooperatives employed additional strategies to reach out

Table 2 Build health information technology to support data-informed QI. ERIC strategies included in EvidenceNOW interventionswith Proctor specificationsExpected outcomes: Ability to generate ABCS reports sufficient HIT capacity, ability to generate ABCS reports, ABCS documentation, data interoperability and sharing.Sustained performance quality measurement and use of data for QIJustification: Practices need population-level data and interoperability to improve quality and optimize care delivery, but they have little or no HIT capacity for this. Prac-tices may need additional, external data infrastructure to provide robust practice-level, clinician-level, and patient-level data on demand. Practices often need internal sup-port to access, validate, and use data for QI. Practices are motivated to improve upon seeing their own data

No.co-ops

Name it Specify it

ERIC cluster ERIC strategy* Actor Specific action(s) Target

5 Adapt and tailorto context

Use data experts Cooperativeleadership, dataexperts

Hire health informatics technology experts (called “data experts” in thistable) to connect practices to external data infrastructures or hire practicefacilitators skilled in health informatics technology (called HIT-PFs) to assistpractices in generating and understanding EHR data

Cooperative,practice

5 Adapt and tailorto context

Use data warehousing techniques Data expert,HIT-PF

Connect practice EHR to warehouse, repository, and/or other externaldurable infrastructures (i.e., registries and software platforms)

Practice

6 Use evaluativeand iterativestrategies

Develop and implement tools forquality monitoring

Data expert,HIT-PF

Perform the data extraction, data normalization, and “back-end” datavalidation necessary for data warehousing and other data platforms

Practice,datainfrastructure


Develop and organize qualitymonitoring systems

Data expert,HIT-PF, PF

Connect practices to additional data interfaces for receiving ABCS data andother types of metrics (i.e., access to data software platforms likepopHealth or Sharepoint or cooperative dashboards)

Practice

3 Utilize financialstrategies

Fund and contract (and/ornegotiate) with vendors for theclinical innovation

Cooperativeleadership

Reimburse for registry connections for the duration of EvidenceNOW;negotiate with EHR vendors to connect practices to data infrastructure

Practice

2 Changeinfrastructure

Change records systems Data expert Help practices upgrade to new EHR and/or modify existing EHRs forefficiency and accuracy

Practice

6 Provideinteractiveassistance

Provide local technical assistance Data expert,HIT-PF, PF

Audit charts to validate data reports; assist with helping practices improveABCS documentation; help practices run/generate/pull reports;troubleshoot dashboards; help practice staff transition to and learn to usenew EHR, if needed

Practice, PF


Audit and provide feedback Data expert,HIT-PF, PF

Share ABCS data for feedback and monitor improvement over time. Mostused ABCS data for audit and provide feedback, and several providedbenchmarking to similar practices; a few included survey items and othersources of data for feedback

Practice

ABCS aspirin, blood pressure, cholesterol, and smoking cessation; PF practice facilitator; HIT health information technology; HIT-PF health information technologyspecialist practice facilitator; EHR electronic health record; Data expert health informatics technology expert, expert consultant physician faculty or clinical expert; Q& A question and answer; FAQ frequently asked questions; extension agent community-based agent specializing in using community resources to address socialdeterminants of health*Modifications to ERIC labels and new strategies are in italics. Strategy definitions are in Table 1


directly to patients and/or families to promote accept-ance of ABCS evidence-based treatments among theirpatients.

DiscussionWe used the naming and definitions provided by theERIC taxonomy [10] of implementation strategies toharmonize the language across the EvidenceNOW coop-eratives. We expanded on this information by using thedescriptive domains recommended by Proctor and

colleagues to specify details about how each implemen-tation strategy was actually used. The combination ofstandardized language from ERIC and specification ofwhat happened when, by whom, under what conditionsand for what reason provided a means by which toharmonize, understand, and compare implementationstrategies across cooperatives. This is a necessary foun-dation to cross-cooperative comparisons in a nationalevaluation such as ESCALATES. Cooperatives, collect-ively, employed 33 of the ERIC implementation

Table 3 Build QI capacity and improve outcomes. ERIC strategies included in EvidenceNOW interventions with Proctor specifications

Expected outcomes: Improve ABCS outcomes and QI capacity Improved ABCS measures and increased practice capacity to take on new qualityinitiatives. Additional outcomes may vary by cooperative (e.g., joy in practice). Knowledge of evidence-based guidelines and QI techniques and princi-ples is neededJustification: practices are under-resourced for quality improvement and need external support to guide them through the change process. Imple-mentation staff needs a supportive infrastructure to help them help practices achieve their goals

No.co-ops

Name it Specify it


7 Provideinteractiveassistance

Implementation facilitation PF Assist with the change process and support change efforts; reviewworkflows and create actionable plans to put best practices inplace (i.e., huddles, pre-visit planning, medication synchronization,gaps analysis, outreach, decision support use, patient education,etc.)

Practice


Assess for readiness andidentify barriers and facilitators

Dataexpert HIT-PF, PF

Assess HIT needs and data set-up and workflow; assess clinic work-flows; assess use of evidence-based protocols

Practice


Develop an implementationblueprint

PF, HIT-PF Discuss and identify improvement plan; adjust plans according todata; develop actionable plans for implementation of evidence-based protocols and addressing other pain points (i.e. non-compliant patients)

Practice


Conduct cyclical small tests ofchange

PF Develop and use Plan Do Study Act (PDSA), Define MeasureAnalyze Improve and Control (DMIAC), and other tests of changeactivities and processes involved in QI

Practice

5 New Assess and redesign workflow PF Assess and revise clinic workflows using a variety of techniques (i.e.,observations and mapping of workflow, role-redesign exercises) tofacilitate discussion and implementation of evidence-basedprotocols

Practice

4 Supportclinicians

Remind clinicians PF Assist the change process through use of clinical reminders anddecision-support tools

Practice

4 Developstakeholderinterrelationships

Identify and prepare champions PF Assist in creating and engaging practice leaders and others inpromoting efficient care strategies

Practice


Recruit, designate and train forleadership

PF Assist in creating and facilitating practice leaders and others inpromoting efficient care strategies

Practice


Organize implementationteams and team meetings

PF Assist in creating and facilitating QI teams in promoting efficientcare strategies

Practice

4 Engageconsumers

Intervene with patients/consumers to enhance uptakeand adherence

PF Help generate patient lists for outreach from EHRs or registries;recall patients; offer patient education materials

Practice

5 Developstakeholderinterrelations

Use an implementation advisor ExpertconsultantPF-HIT

Support and education for practice facilitators by data experts and/or physician faculty hired as expert consultants

PF

ABCS aspirin, blood pressure, cholesterol, and smoking cessation; PF practice facilitator; HIT health information technology; HIT-PF health information technologyspecialist practice facilitator; EHR electronic health record; data expert health informatics technology expert; expert consultant physician faculty or clinical expert; Q& A question and answer, FAQ frequently asked questions, extension agent community-based agent specializing in using community resources to address socialdeterminants of health*modifications to ERIC labels and new strategies are in italics. Strategy definitions are in Table 1


Table 4 Enhance clinician and practice member knowledge. ERIC strategies included in EvidenceNOW interventions with Proctorspecifications

Expected outcomes: Clinician knowledge of ABCS & QI and exchange best practices Knowledge of clinical guidelines, use of local resources, andincreased peer interaction encouraging continued engagement and participation in QI. Exchange of best practices, peer learning and supportJustification: Clinicians benefit from learning in a variety of formats (peer-to-peer; online learning; in-person one-on-one or group learning) from differ-ent professionals (expert consultants who are also clinical peers, practice facilitators, other healthcare experts). Interactive, collaborative learning andpeer networking help practices engage in QI and reduce burnout

No.co-ops

Name it Specify it


7 Train andeducatestakeholders

Developeducationalmaterials

Cooperative leadership, expertconsultants, PF

Develop webinars, Q & A, toolkit, training modules, andonline resources; compile lists of community resources

Practice


Distributeeducationalmaterials

Expert consultant, communityhealth improvement organization,extension agent, PF

Host didactic webinars and videos and post toolkits,modules, Q & A/FAQ online for asynchronous learning;provide information during in person visits, such as listing ofcommunity resources; support change efforts throughproviding educational materials and working on them withpractices; establish links to community resources

Practice


Conducteducationaloutreach visits

Expert consultants, data experts Hold one-on-one meetings with practice to educate on clin-ical topics; visits to train practice data champion in datareporting/data use training

Practice


Provideongoingconsultation

Expert consultants Provide expert advice by email or in person Practice


Conductongoingtraining

Expert consultants Provide interactive training webinars and “office hours” onclinical topics related to the ABCS; include time for Q & A.Provide training on cardiovascular disease risk calculatorsand other clinical tools

Practice


Create alearningcollaborative

Cooperative leadership, PF,practice

Hold events for practices to learn from and interact withcooperative staff about the clinical interventions as well asinteract with other practices, workshop practice needs, andcreate plans for embarking on or continuing withintervention change processes

Practice

2 New Create onlinelearningcommunities


Create online forum with access to resources, networking,and discussions

Clinicalcommunity


Conducteducationalmeetings

Cooperative leadership, expertconsultants, PF, practice

Hold collective meetings for practices with cooperatives forthe purpose of orienting the practice to the interventionand educating practice members on clinical topics and/orbest practices; practice facilitators may hold collaborativecalls or meetings with practices for training on particulartopics of practices’ choice

Practice


Shadow otherexperts

Cooperative leadership, practice Select exemplar practices for other practices to visit; visitsmay be in person or virtual

Practice


Visit othersites

Cooperative leadership, practice Select exemplar practices for other practices to visit or bringtogether local practices to discuss best practices andintervention successes and challenges; visits may be inperson or virtual

Practice


Promotenetworkweaving


Encourage networking between practices to build a clinicalcommunity for best practices

Practice

5 Developstakeholderinterrelation-ships

Capture andshare localknowledge

Cooperative leadership, expertconsultants, PF, practice

Hold in-person meetings or events or phone calls so thatpractices can learn from each other and engage in discus-sion; visit exemplar practices; facilitate networks and learningevents in the region; host a small group of practices in oneclinic; use online portal for virtual learning

Practice

ABCS aspirin, blood pressure, cholesterol, and smoking cessation; PF practice facilitator; HIT health information technology; HIT-PF health information technologyspecialist practice facilitator; EHR electronic health record; data expert health informatics technology expert; expert consultant physician faculty or clinical expert; Q& A question and answer; FAQ frequently asked questions; extension agent community-based agent specializing in using community resources to address socialdeterminants of health*Modifications to ERIC labels and new strategies are in italics. Strategy definitions are in Table 1


strategies; we identified a number of areas where im-provements could be made to the ERIC taxonomy tobetter align with how implementation happenedon-the-ground within the large-scale EvidenceNOW ini-tiative. In addition, we recommend an alternative way ofgrouping some strategies to better align them with howthey are linked to accomplish real-world goals, for ex-ample, the need for several HIT-related strategies (e.g.,use data experts) to implement and sustain use of auditand provide feedback.Cooperatives documented 266 actions mapped to 33

of 73 total ERIC strategies. This number of strategies isconsistent with other published studies that have re-ported a range of 16–59 strategies used in implementa-tion studies [13, 15, 16]. Likely, the range and number ofstrategies employed in an implementation interventiondepends upon the goals and scale of the project. Wepropose adding three strategies to the ERIC list: engagecommunity resources, create online learning communi-ties, and assess and redesign workflow. Each of thesestrategies encapsulates distinct actions that were notidentified in the existing ERIC taxonomy and are likelyto be used in a broad range of other settings. Other re-searchers have also recommended new strategies: obtainworker feedback about implementation plan and plan foroutcome evaluation [13]. Thus, as ERIC is applied in real

world settings, additional strategies may continue to beidentified expanding the ERIC taxonomy.Audit and provide feedback was a critical implementa-

tion strategy used across cooperatives. Audit and providefeedback consists of two distinct yet interrelated con-cepts: audit, the assessment of performance often com-pared to a specific target or benchmark; and feedback,communication of the performance. Data used to meas-ure and assess performance must be accurate and cred-ible [23–25]. Many of the individual practices within theEvidenceNOW cooperatives had challenges producingthe audit needed for feedback. Published trials typicallyonly describe the feedback function within audit andprovide feedback when reporting findings. In a commen-tary on an updated Cochrane review of audit and pro-vide feedback, the authors list eight characteristics thatcontribute to variability in effectiveness across studies;all eight characteristics pertain to feedback with no men-tion of the role and quality of the audit component [24].Other publications have mentioned audit functions butdo not articulate the importance of reliable, sustainableprocesses to obtain the audits [26] or provide guidanceon the infrastructure needed to obtain valid and mean-ingful audits [25]. Without valid and reliable audits, it ischallenging, if not impossible, to provide the feedback,ultimately impacting the capability to reliably identify

Table 5 Build community connections and patient involvement. ERIC strategies included in EvidenceNOW interventions withProctor specifications

Expected outcomes: Improved community connections and patient involvement Improved referrals to accessible patient resources, improved patientengagement in their own care. Increased patient activation and knowledge leads to acceptance of new evidence-based practices and increased qual-ity of careJustification: Local resources help practices and patients improve cardiovascular preventive care and self-management. Patient feedback and involve-ment in QI promotes patient-clinician communication, tailored health messaging, and patient adherence

No.co-ops

Name it Specify it


5 New Engage communityresources

PF,extensionagent

Build links between practices and health resources or organizationsembedded in communities; varies from meetings with communityorganizations and participating in collaborative project to participatingin referral programs

Practice

1 Support clinicians Develop resource sharingagreements

Cooperativeleadership

Connect to state and county departments of health; connect practicesto local health related organizations to participate in collaborativeprojects

Practice

1 Engageconsumers

Involve patients/consumers and familymembers

Cooperativeleadership,PF

Include patients on QI teams Practice


Obtain and use patients/consumers and familyfeedback

Cooperativeleadership,PF

Assess and use results of patient/consumers and family feedback frompatient engagement surveys; patient focus group, patient and familyadvisory councils

Practice

1 Engageconsumers

Prepare patients/consumers to be activeparticipants

PF Invite members of the community to participate in tailoring ABCS andCVD messaging for their communities

Practice

ABCS aspirin, blood pressure, cholesterol, and smoking cessation; PF practice facilitator; HIT health information technology; HIT-PF health information technologyspecialist practice facilitator; EHR electronic health record; data expert health informatics technology expert; expert consultant physician faculty or clinical expert; Q& A question and answer; FAQ frequently asked questions; extension agent community-based agent specializing in using community resources to address socialdeterminants of health*Modifications to ERIC labels and new strategies are in italics. Strategy definitions are in Table 1


and monitor progress of QI efforts. Furthermore,pay-for-performance initiatives, such as the Centers forMedicare and Medicaid Quality Payment Program ne-cessitate accurate reporting of clinical quality measures.Thus, there is increasing pressure on primary care prac-tices to generate reliable data in a sustainable manner.The EvidenceNOW cooperatives described seven strat-

egies that were needed in differing combinations, de-pending on the local context, to build the necessary HITinfrastructure to support audit and provide feedbackand to then engage in data-informed QI to improveABCS outcomes. In the ERIC taxonomy, these sevenstrategies are spread across four different clusters. Wegroup these HIT strategies based on their shared justifi-cation and expected outcomes as this better reflects howthese strategies may need to be sequenced to accomplishthe intended outcome. Five of the seven strategies tobuild HIT infrastructure were rated as low-feasibilitywithin the ERIC taxonomy [11] largely because of thedifficulty and cost of building HIT infrastructure. Sinceall but one of the cooperatives planned to sustain auditand provide feedback functionality beyond this initiativeand because of its focus and substantial funding, Eviden-ceNOW makes a unique contribution to the ERIC tax-onomy by providing insight into what is needed to buildthe infrastructure to support audit and provide feedbackfunctions.A second clear grouping of strategies is those aimed at

helping practices build QI capacity and improve ABCSoutcomes, supported foundationally by implementationfacilitation, the core strategy described by all seven co-operatives. Implementation facilitation was used to sup-port practices in accomplishing their QI goals on thepath to improving ABCS outcomes; it involves perform-ing interrelated and complex roles and skillfully applyingdiverse strategies in a flexible and dynamic manner tomeet the local needs and priorities of each primary carepractice [27–31]. Thus, it is both a role (facilitator) anda strategy (implementation facilitation). In Evidence-NOW, practice facilitators provided nearly all of the ex-ternal support to practices. The facilitators, includingfacilitators with HIT skill, were responsible for perform-ing, either solely or in conjunction with other actors, 27of the 33 strategies, spanning all four overarching func-tions within EvidenceNOW. Using this broad array ofstrategies requires diverse and sophisticated skills in-cluding interpersonal (including emotional intelligence),technical (including adept use of EHR platforms),organizational, communication, leadership, and peda-gogical skills [29, 32, 33]. The broad scope of strategiesused by facilitators highlights the challenge and com-plexity of facilitation and the critical role facilitators playin supporting practices in implementing evidence intopractice and building capacity for QI.

Regrouping the ERIC strategies from the current nineclusters to a clustering that is more pragmatic may makethe ERIC taxonomy easier for practitioners to apply andmight be helpful in implementation planning. Weintentionally developed these groupings to better reflecthow strategies were employed across the diverse con-texts of EvidenceNOW. Leeman and colleagues havealso proposed an alternative organizing structure forERIC, proposing five groups, based upon actor and tar-get for each strategy [34]. They suggest that this classifi-cation approach, grounded in relevant theory, may aid inunderstanding the mechanisms by which strategies bringabout change [34]. We suggest that the pragmaticgrouping we developed may be useful to implementersand recognize the value that analysis of the practical ap-plications of ERIC can have for informing the refinementof this taxonomy for different audiences and users.In our analysis, we specified further each cooperative’s

implementation strategies following Proctor and col-leagues’ recommendations. Specifying actor highlightedthe broad array of strategies performed by the practicefacilitators. A priori specification of dose and temporal-ity, however, was less useful for EvidenceNOW. Moststrategies were designed to be used “as needed” in re-sponse to local needs of practices. On the one hand, thiswas a unifying theme across the cooperatives, and onthe other hand, specifying these dimensions for each in-dividual strategy was unnecessary. However, the cooper-atives did track the activities of their facilitators atvarying levels of detail; analysis of this data is outsidethe scope of this current work. Other researchers havehighlighted the challenges with specifying dose: it isresource-intensive to specify and track often because ofits ambiguity [12, 15]. However, a priori specification oftemporality can be useful in guiding the timing for spe-cific implementation strategies within a more structuredmulti-site implementation, as highlighted by Huynh andcolleagues [15].

LimitationsData were collected early in the cooperatives’ implemen-tation phase; they reflected strategies that were includedor just started to be employed during the implementa-tion phase of each of the cooperatives’ respectivelarge-scale multi-component interventions. Some coop-eratives may have under-reported their strategies be-cause they may have been tacit or intuitive and thus notexplicitly reported. This limitation was mitigatedthrough online diary postings by cooperative team mem-bers including practice facilitators and early site visitswhere we observed practice facilitators and others whowere delivering implementation strategies, rather thanrelying exclusively on cooperatives’ self-reported data.The cooperatives did track dose and temporality of


strategies used (i.e., when use of a strategy started andended), but these data were nearly impossible toharmonize across the cooperatives because of the vary-ing level of detail and inconsistencies within how doseand temporality were tracked. It is important to notethat this work includes strategies that were specified bycooperatives to provide external support to primary carepractices; it does not include micro-level strategies usedwithin individual practices.

ConclusionsBased on empirical experiences across seven large coop-eratives participating in the AHRQ-funded large-scaleEvidenceNOW initiative, we recommend refinements tothe ERIC list of strategies, which if adopted, would bet-ter align ERIC with real-world implementation needsand improve its utility for implementers and researchers.Combined use of the refined ERIC taxonomy and thespecifications suggested by Proctor et al. add transpar-ency and thus promote replicability by identifying, creat-ing common nomenclature, and harmonizing apotentially diverse array of implementation strategiesused by multiple large multi-site dissemination and im-plementation initiatives such as EvidenceNOW.

Additional files

Additional file 1: Example from a single cooperative’s table ofinterventions. (DOCX 23 kb)

Additional file 2: ERIC strategy names and definitions. (DOCX 52 kb)

AbbreviationsABCS: Aspirin use for high-risk patients, blood pressure control, cholesterolmanagement, smoking cessation counseling; AHRQ: Agency for HealthcareResearch and Quality; EHR: Electronic health record; ERIC: ExpertRecommendations for Implementing Change; ESCALATES: Evaluating SystemChange to Advance Learning and Take Evidence to Scale, a nationalevaluation of EvidenceNOW; HIT: Health information technology; QI: Qualityimprovement

AcknowledgementsWe are extremely grateful to all of the EvidenceNOW Cooperatives who havemade this work possible. In addition, the entire ESCALATES study teamwillingly contributed their thoughtful feedback to this manuscript. We areparticularly thankful for the extra support we received from BenjaminCrabtree, PhD; Leif Solberg, MD; William Miller, MD, MA, Donna Shelley, MD,MPH and Claire Diener, BA as well as editorial assistance from AmandaDelzer Hill, BA.

FundingThis research was supported by a grant from Agency for Healthcare Researchand Quality 1R01HS023940-01 (PI: Cohen).

Availability of data and materialsThe datasets generated and/or analysed during the current study are notpublicly available because the study is still in progress; data will not bepublicly available until after the national evaluation is complete, but may beavailable from the corresponding author upon reasonable request.

Authors’ contributionsCKP, LJD, JRH, and TTW analyzed and interpreted data, mapped to ERICtaxonomy, and prepared the manuscript; SSO analyzed and interpreted dataand prepared the manuscript; and DJC analyzed and interpreted data andprepared manuscript. All authors read and approved the final manuscript.

Ethics approval and consent to participateStudy was approved by the Oregon Health & Science University InstitutionalReview Board. Participants provided informed consent prior to participation.

Consent for publicationNot applicable.

Competing interestsThe authors declare that they have no competing interests.

Publisher’s NoteSpringer Nature remains neutral with regard to jurisdictional claims inpublished maps and institutional affiliations.

Author details1School of Nursing, Oregon Health & Science University, 3455 SW USVeterans Hospital Rd, Portland, OR 97239, USA. 2Implementation Pathways,LLC, Ann Arbor, MI, USA. 3VA Center for Clinical Management Research, AnnArbor, MI, USA. 4Department of Family Medicine and Community Health,Rutgers University--Robert Wood Johnson Medical School, 112 PatersonStreet, New Brunswick, NJ 08901, USA. 5Department of Family Medicine,Oregon Health & Science University, 3181 SW Sam Jackson Park Rd, Portland,OR 97239, USA.

Received: 8 January 2019 Accepted: 28 February 2019

References1. Proctor EK, Powell BJ, McMillen JC. Implementation strategies:

recommendations for specifying and reporting. Implement Sci. 2013;8:139.2. Nilsen P. Making sense of implementation theories, models and frameworks.

Implement Sci. 2015;10:53.3. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and

practice: models for dissemination and implementation research. Am J PrevMed. 2012;43(3):337–50.

4. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC.Fostering implementation of health services research findings into practice:a consolidated framework for advancing implementation science.Implement Sci. 2009;4:50.

5. Harvey G, Kitson A. PARIHS revisited: from heuristic to integrated frameworkfor the successful implementation of knowledge into practice. ImplementSci. 2016;11:33.

6. May CR, Mair F, Finch T, MacFarlane A, Dowrick C, Treweek S, et al.Development of a theory of implementation and integration: normalizationprocess theory. Implement Sci. 2009;4:29.

7. Davidoff F, Dixon-Woods M, Leviton L, Michie S. Demystifying theory and itsuse in improvement. BMJ Qual Saf. 2015;24(3):228–38.

8. Foy R, Ovretveit J, Shekelle PG, Pronovost PJ, Taylor SL, Dy S, et al. The roleof theory in research to develop and evaluate the implementation ofpatient safety practices. BMJ Qual Saf. 2011;20(5):453–9.

9. Powell BJ, McMillen JC, Proctor EK, Carpenter CR, Griffey RT, Bunger AC,et al. A compilation of strategies for implementing clinical innovations inhealth and mental health. Med Care Res Rev. 2012;69(2):123–57.

10. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM,et al. A refined compilation of implementation strategies: results from theExpert Recommendations for Implementing Change (ERIC) project.Implement Sci. 2015;10:21.

11. Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL,et al. Use of concept mapping to characterize relationships amongimplementation strategies and assess their feasibility and importance:results from the Expert Recommendations for Implementing Change (ERIC)study. Implement Sci. 2015;10:109.

12. Boyd M, Powell B, Endicott D, Lewis C. A method for trackingimplementation strategies: an exemplar implementing measurement-basedcare in community behavioral health clinics. Behav Ther. 2018;49(4):525–37.


https://doi.org/10.1186/s13012-019-0876-4

https://doi.org/10.1186/s13012-019-0876-4

13. Bunger AC, Powell BJ, Robertson HA, MacDowell H, Birken SA, Shea C.Tracking implementation strategies: a description of a practical approachand early findings. Health Res Policy Syst. 2017;15(1):15.

14. Gold R, Bunce AE, Cohen DJ, Hollombe C, Nelson CA, Proctor EK, et al.Reporting on the strategies needed to implement proven interventions: anexample from a “real-world” cross-setting implementation study. Mayo ClinProc. 2016;91(8):1074–83.

15. Huynh AK, Hamilton AB, Farmer MM, Bean-Mayberry B, Stirman SW, Moin T,et al. A pragmatic approach to guide implementation evaluation research:strategy mapping for complex interventions. Front Public Health. 2018;6:134.

16. Rogal SS, Yakovchenko V, Waltz TJ, Powell BJ, Kirchner JE, Proctor EK, et al.The association between implementation strategy use and the uptake ofhepatitis C treatment in a national sample. Implement Sci. 2017;12(1):60.

17. Agency for Healthcare Research and Quality: EvidenceNOW: AdvancingHearth Health in Primary Care. https://www.ahrq.gov/evidencenow.Accessed 4 Jan 2019.

18. Accelerating the Dissemination and Implementation of PCOR Findings intoPrimary Care Practice (R18). Rockville: Agency for Healthcare Research andQuality. https://grants.nih.gov/grants/guide/rfa-files/RFA-HS-14-008.html.Updated December 2014. Accessed 4 Jan 2019.

19. Cohen DJ, Balasubramanian BA, Gordon L, Marino M, Ono S, Solberg LI,et al. A national evaluation of a dissemination and implementation initiativeto enhance primary care practice capacity and improve cardiovasculardisease care: the ESCALATES study protocol. Implement Sci. 2016;11(1):86.

20. Ono SS, Crabtree BF, Hemler JR, Balasubramanian BA, Edwards ST, Green LA,et al. Taking innovation to scale in primary care practices: the functions ofhealth care extension. Health Aff (Millwood). 2018;37(2):222–30.

21. Cohen D, Leviton L, Isaacson N, Tallia A, Crabtree B. Online diaries forqualitative evaluation gaining real-time insights. Am J Eval. 2006;27:163–84.

22. Miller W, Crabtree B. The dance of interpretation. In: CBaM WL, editor. Doingqualitative research. 2nd ed. Thousand Oaks: Sage Publications; 1999. p.127–43.

23. Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD,et al. Audit and feedback: effects on professional practice and healthcareoutcomes. Cochrane Database Syst Rev. 2012;13(6):CD000259.

24. Ivers NM, Grimshaw JM, Jamtvedt G, Flottorp S, O'Brien MA, French SD,et al. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions inhealth care. J Gen Intern Med. 2014;29(11):1534–41.

25. Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ, et al. No more‘business as usual’ with audit and feedback interventions: towards anagenda for a reinvigorated intervention. Implement Sci. 2014;9:14.

26. Brehaut JC, Colquhoun HL, Eva KW, Carroll K, Sales A, Michie S, et al.Practice feedback interventions: 15 suggestions for optimizing effectiveness.Ann Intern Med. 2016;164(6):435–41.

27. Berta W, Cranley L, Dearing JW, Dogherty EJ, Squires JE, Estabrooks CA. Why(we think) facilitation works: insights from organizational learning theory.Implement Sci. 2015;10:141.

28. Dogherty EJ, Harrison MB, Graham ID. Facilitation as a role and process inachieving evidence-based practice in nursing: a focused review of conceptand meaning. Worldviews Evid-Based Nurs. 2010;7(2):76–89.

29. Lessard S, Bareil C, Lalonde L, Duhamel F, Hudon E, Goudreau J, et al. Externalfacilitators and interprofessional facilitation teams: a qualitative study of theirroles in supporting practice change. Implement Sci. 2016;11:97.

30. Nagykaldi Z, Mold JW, Aspy CB. Practice facilitators: a review of theliterature. Fam Med. 2005;37(8):581–8.

31. Taylor EF, Machta RM, Meyers DS, Genevro J, Peikes DN. Enhancing theprimary care team to provide redesigned care: the roles of practicefacilitators and care managers. Ann Fam Med. 2013;11(1):80–3.

32. Grumbach K, Bainbridge E, Bodenheimer T. Facilitating improvement inprimary care: the promise of practice coaching. Issue Brief CommonwealthFund. 2012;15:1-14.

33. Harvey G, Loftus-Hills A, Rycroft-Malone J, Titchen A, Kitson A, McCormack B,et al. Getting evidence into practice: the role and function of facilitation. JAdv Nurs. 2002;37(6):577–88.

34. Leeman J, Birken SA, Powell BJ, Rohweder C, Shea CM. Beyond“implementation strategies”: classifying the full range of strategies used inimplementation science and practice. Implement Sci. 2017;12(1):125.


https://www.ahrq.gov/evidencenow

https://grants.nih.gov/grants/guide/rfa-files/RFA-HS-14-008.html

Documents

Specifying and comparing implementation strategies across