SYSTEMATIC REVIEW Key characteristics of successful quality improvement … · patient care and system performance outcomes. The realist review approach offers one approach to decon-struct

Key characteristics of successfulquality improvement curricula inphysician education: a realist review

Anne C Jones,1,2,3,4 Scott A Shipman,2,3,5 Greg Ogrinc1,2

▸ Additional material ispublished online only. To viewplease visit the journal online(http://dx.doi.org/10.1136/bmjqs-2014-002846).

1Veterans Affairs MedicalCenter, White River Junction,Vermont, USA2Geisel School of Medicine atDartmouth, Hanover,New Hampshire, USA3The Dartmouth Institute forHealth Policy and ClinicalPractice, Lebanon,New Hampshire, USA4Gannett Health Services,Cornell University, Ithaca,New York, USA5Association of AmericanMedical Colleges, Washington,DC, Washington,USA

Correspondence toDr Anne C Jones,Assistant Medical Director,Gannett Health Services,110 Ho Plaza, Cornell University,Ithaca, NY 14853, USA;[email protected]

Received 15 January 2014Revised 20 August 2014Accepted 30 August 2014Published Online First30 September 2014

To cite: Jones AC,Shipman SA, Ogrinc G. BMJQual Saf 2015;24:77–88.

ABSTRACTPurpose Quality improvement (QI) is a commoncompetency that must be taught in all physiciantraining programmes, yet, there is no clear bestapproach to teach this content in clinicalsettings. We conducted a realist systematicreview of the existing literature in QI curriculawithin the clinical setting, highlighting examplesof trainees learning QI by doing QI.Method Candidate theories describingsuccessful QI curricula were articulated a priori.We searched MEDLINE (1 January 2000 to 12March 2013), the Cochrane Library (2013) andWeb of Science (15 March 2013) and reviewedreferences of prior systematic reviews. Inclusioncriteria included study design, setting,population, interventions, clinical andeducational outcomes. The data abstraction toolincluded categories for setting, population,intervention, outcomes and qualitativecomments. Themes were iteratively developedand synthesised using realist reviewmethodology. A methodological quality toolassessed the biases, confounders, secular trends,reporting and study quality.Results Among 39 studies, most were before–after design with resident physicians as theprimary population. Twenty-one describedclinical interventions and 18 describededucational interventions with a meanintervention length of 6.58 (SD=9.16) months.Twenty-eight reported successful clinicalimprovements; no studies reported clinicaloutcomes that worsened. Characteristics ofsuccessful clinical QI curricula include attentionto the interface of educational and clinicalsystems, careful choice of QI work for thetrainees and appropriately trained local faculty.Conclusions This realist review identifiedsuccess characteristics to guide trainingprogrammes, medical schools, faculty, trainees,accrediting organisations and funders to furtherdevelop educational and improvement resourcesin QI educational programmes.

BACKGROUNDStemming from the Institute ofMedicine’s reports, To Err Is Human in2000 and Crossing the Quality Chasm in2001, improvement in patient outcomesand reduction in medical errors are focifor healthcare institutions around theworld.1 In 2003, the AccreditationCouncil for Graduate Medical Education(ACGME) and American OsteopathicAssociation responded by integratingsystems-based practice (SBP) and practice-based learning and improvement (PBLI)as two of the six core competencies ofmedical education.2 Quality improvement(QI) teaching encompasses content ofboth SPB and PBLI. However, uncer-tainty remains about which methods arethe most effective, and in what circum-stances, for improving educational andclinical outcomes.3–5

Reviews of QI teaching in undergradu-ate and graduate medical education havefound some improvement in educationaloutcomes but little effect on patient out-comes.6–8 Wong et al9 10 identified threecategories of QI education, most ofwhich fall into the first category: (1)formal curricula that teach concepts ormethods intended to facilitate trainees’participation in QI activities; (2) educa-tional activities that impart specificrelated skills and (3) QI initiatives thatinvolve trainees as active or passive parti-cipants. Many others worldwide havedeveloped clinical teaching of QI, aimingto engage physician trainees to improvethe care of the patients they serve and thefunction of the system in which theypractice.10

Although helpful in summarising thenovel approaches to QI education, priorsystematic reviews have been limited.They appropriately sought to answer the

SYSTEMATIC REVIEW

Jones AC, et al. BMJ Qual Saf 2015;24:77–88. doi:10.1136/bmjqs-2014-002846 77

on April 13, 2020 by guest. P

rotected by copyright.http://qualitysafety.bm

j.com/

BM

J Qual S

af: first published as 10.1136/bmjqs-2014-002846 on 30 S

eptember 2014. D

ownloaded from

http://dx.doi.org/10.1136/bmjqs-2014-002846



http://crossmark.crossref.org/dialog/?doi=10.1136/bmjqs-2014-002846&domain=pdf&date_stamp=2014-09-30

http://qualitysafety.bmj.com/

http://www.health.org.uk/


question of whether or not QI educational interven-tions had an impact on physician trainees’ ability togain knowledge and sought to identify themes asso-ciated with successful QI curricula, but did not assessthe specific mechanistic and contextual factors thatpredicted success, especially for improvement ofpatient care and system performance outcomes. Therealist review approach offers one approach to decon-struct such complex interventions to assess key successcharacteristics and develop recommendations.11 12

In this study, we define key characteristics of suc-cessful QI curricula in medical education. This realistreview determines how the teaching of QI in the clin-ical setting enhances patient care and system perform-ance while increasing trainee knowledge and skills.

METHODSReview frameworkA realist review is based on the premise that complexinterventions are successful when certain character-istics facilitate the optimal functioning of a system toproduce a particular outcome when a complex inter-vention is applied.11 In a realist review, an iterativeapproach is used to identify the characteristics ofcomplex interventions in the following categories:‘what works,’ ‘for whom,’ ‘under what circumstances’and ‘to achieve what outcomes’.11 The realist reviewbegins with the articulation of candidate theories thatmay explain the characteristics required for interven-tions to be successful.11 Next, identification and selec-tion of studies is achieved through standard systematicreview approach.11 Once relevant studies are chosenfor inclusion, data are systematically abstracted fromthe studies and the studies are read and reread toidentify themes.11 An iterative approach is used toidentify data, quotations, tables and figures that eithersupport or refute the candidate theories articulated atthe outset. Theories are refined as more data are gath-ered from the articles.11

We began by searching the literature for existingtheories which explained teaching of QI in the clinicalsetting. We evaluated the prior systematic reviews onthe topic of QI medical education,7–9 13 spoken withexperts in the field,7 9 and prepared a candidate con-ceptual framework (see online supplement 1) andaccompanying theory for review.7 14 Our candidatetheory hypothesised that the process of educatingphysicians begins with a curriculum and is impactedby characteristics of the learner, teacher, communityand others, all encompassed within the educationalcontext. From within the educational context emergesengaged learners and teachers, who produce improvededucational and clinical outcomes. The combinationof these successes produces physicians who arecapable of and believe that it is their job to do theirwork and improve their work.The second step of the realist review is development

of inclusion criteria, search strategies, a data

abstraction tool and methodological quality assess-ment for review of the literature and analysis ofincluded studies. Throughout the above process, thecandidate theories were tested and refined and newtheories added. As the studies were evaluated, themesemerged that were based on the predetermined theor-ies. Each theme was assigned a code and linked to aquotation in the study. As a new theme emerged itwas assigned a new code; we then searched for thistheme in all the included studies in the review. Newand revised candidate theories were synthesised intothe set of candidate theories to test in our realistreview.

Study eligibility criteriaIncluded studies had the following criteria:▸ Study design—original journal articles (no commentar-

ies, letters to the editor, editorials or position pieces).▸ Setting—medical schools, residency and fellowship pro-

grammes worldwide.▸ Population—physician trainees (medical students, resi-

dents and/or fellows).▸ Interventions—whether clinical or educational—that

engage trainees in QI work, where they are involved inchanges to the delivery of care to patients within theclinical setting.

▸ Reporting of clinical outcomes (patient care outcomesand system performance improvements) as the primaryoutcome measure.

Search methodsIn collaboration with a professional librarian, onereviewer (ACJ) developed search strategies for the fol-lowing databases: MEDLINE (2000 to 12 March2013), Cochrane Library (2013) and Web of Science(15 March 2013). To locate potentially relevantstudies in MEDLINE, we used exploded MedicalSubject Headings terms and key words to generatesets for the themes of QI and medical education. Wethen used the Boolean term ‘AND’ to find their inter-section. This basic approach was modified as neces-sary to search each electronic database. No languagerestriction was applied. Time limit was applied toobtain articles published after 2000, which corre-sponded with the publication of the Institute ofMedicine reports To Err Is Human and Crossing theQuality Chasm. We excluded commentaries, editorialsand letters. The full search strategy is available uponrequest. Reference reviews of the four earlier system-atic reviews7–9 13 were performed by obtaining allreferences cited and searching forward using Web ofScience to find all papers which cited these reviewsand including them in title and abstract review.

Study selectionOne reviewer (ACJ) independently screened each titleand abstract for eligibility. Then, two non-blindedreviewers (ACJ and GO) independently assessed the

Systematic review

78 Jones AC, et al. BMJ Qual Saf 2015;24:77–88. doi:10.1136/bmjqs-2014-002846



j.com/

BM

J Qual S


eptember 2014. D

ownloaded from


eligibility of each full text record. Discrepancies wereresolved by consensus between both reviewers afterfull text review.

Data collectionOne reviewer (ACJ) abstracted data from the full textarticles. A standardised data collection tool was usedto capture identifying information, intervention sum-maries, details of study protocol, all primary and sec-ondary outcome data and a section to extractquotations from the articles for the realist review (seeonline supplement 2).

AnalysisAnalysis of interventions and outcomesAnticipating that the QI interventions and outcomeswould be complex and different depending on thetraining programme, we used an iterative approach tocategorise the different types of interventions and out-comes described in each study. We focused on anyqualitative or quantitative reports of change in clinicaloutcomes. If outcomes were reported quantitatively,we determined whether statistical analysis was per-formed, either in the form of enumerative statistics oranalytical statistics using statistical process control, amethod of time-ordered analysis for QI.15 For the sec-ondary outcome, we noted the findings of the educa-tional outcomes such as knowledge, skills andattitudes.

Assessment of methodological qualityThe Medical Education Research Quality Instrument16

is a validated instrument for methodological qualityassessment of the medical education literature, butdoes not allow assessment of the methodologicalquality of the QI education literature specifically. Thus,we developed a set of criteria based upon theCochrane Risk of Bias Tool and the Standards forQuality Improvement Reporting Excellence publica-tion guidelines.17 18 Criteria were based on attentionto quality factors in three major categories: the popula-tion, intervention and outcome reporting (see onlinesupplements 3 and 4). Each study was assessed on allthe criteria for quality, taking into account bias, con-founding and study quality.A final methodological quality score was given to

each study, based on a scale ranging from ‘fair,’ ifalmost none of the criteria were met, to ‘good,’ whenminor flaws were found, to ‘very good,’ when high-quality reporting was achieved in at least one criterioneach for population, intervention and outcome report-ing, and ‘excellent,’ when high quality was achievedfor all criteria (see online supplement 5).

The realist reviewAfter completing the methodology of a systematicreview, we used the realist review methodology torigorously test the conceptual framework. We

iteratively identified relevant themes, and throughcontinuous data collection and rereading the articles,we tested the conceptual framework. Specifically, welooked for examples of when the curriculum wasdeveloped at the outset of the intervention to supportthe educational context of the candidate theory. Wealso looked for examples of successful completion ofQI curricula and examples of physicians who arecapable of and believe that it is their job to do theirwork and improve it.

RESULTSResults of searchWe included 39 studies for our final review (figure 1;excluded studies after full text review is availableupon request), most of which were before–afterstudies or case reports in internal medicine or familymedicine residency programmes (table 1). Lesscommon were controlled trials, studies with medicalstudents or reports from subspecialty resident pro-grammes. Of the 29 studies that reported a samplesize, the mean sample size was 5.6 trainees (SD=102),the median sample size was 24 trainees and the rangewas from 3 to 510 trainees.Among the interventions, 21 were primarily clinical

interventions in which the goal was clearly to improvepatient care or system performance in the clinicalsetting with education of the trainee not the primaryfocus of the intervention; 18 studies primarily hadeducational interventions, in which the goal was todeliver a curriculum to trainees focused on learningabout improvement of which a component was toimprove patient care and system performance in theclinical setting. The mean intervention length was6.6 months (SD=9.2).Twenty studies reported system performance out-

comes (e.g., improved documentation) while threereported only patient care outcomes (e.g., haemoglo-bin a1c, blood pressure) and 16 reported both patientcare and system performance outcomes. Among theclinical outcomes, 28 studies reported successfulimprovements, two of which were not sustained. Tenstudies demonstrated improvement in some measuresand two clinical outcome reports were equivocal. Nostudies reported clinical outcomes that worsened.Nineteen studies reported educational outcomes,

using various measures of knowledge, skills and atti-tudes such as the Quality Improvement KnowledgeApplication Tool (QIKAT),19 satisfaction surveys orobjective structured clinical examinations.

Description of studiesThe most common types of interventions were teamprojects and involvement in an existing clinical QI team(table 2). Among the team project interventions, traineesworked together to make improvements in the clinicalsetting; these studies had varied methodological qualityscores. Seven of these studies demonstrated significant

Systematic review




j.com/

BM

J Qual S


eptember 2014. D

ownloaded from


improvement in documentation, critical care measuresand medication adherence20–26; six studies showed nosignificant clinical outcomes.4 27–31

Of the interventions that involved trainees taking arole within an existing clinical interprofessional QIteam, most were of ‘good’32–41 methodologicalquality. Most showed no significant clinical out-comes,33 35 37 40–43 but some showed statistically sig-nificant improvement in diabetes measures,vaccination rates, chronic care measures and criticalcare measures.32 34 36 38 39

The third most common type of intervention used achart audit. Most of these had ‘good’ methodologicalquality. About half these studies showed statisticallysignificant outcomes, specifically in improved diabetescare, preventive care measures, documentation andcritical care measures44–48; and the other half did notshow statistically significant outcomes.49–52

The least common intervention used individual pro-jects in which the trainee worked independently. Most

of these studies were rated ‘good,’53–56 and onereceived a ‘fair’.57 Most of these studies showed nosignificant clinical outcomes,53 55–57 while one studyshowed significant improvement in care for heartfailure patients.54

Methodological qualityA majority of studies (30) had ‘good’ methodologicalquality. Six were ‘fair’ studies and 3 were ‘very good’.There were no ‘excellent’ studies.Among the included studies, all but three studies

described the intervention in sufficient detail so that itcould be replicated. Although all the studies had, byour inclusion criteria, trainees participating in QIwork within the clinical setting, only 14 studies articu-lated educational objectives for trainees. There werefive studies which described a clinical interventionrather than an educational intervention and reportedno educational outcomes (table 2).

Figure 1 Study selection flow diagram.

Systematic review




j.com/

BM

J Qual S


eptember 2014. D

ownloaded from


It is notable that three studies took steps to minimisebias and confounding. Holmboe et al45 matched eachsecond year resident in the intervention track to a thirdyear control. Dysinger and Pappas21 enrolled the entirefourth year medical school class in a required month-long clinical QI rotation over 3 years, resulting in 510students completing the curriculum, and allowing forcomparison over 3 years of data collection to observeand account for secular trends. Asao et al,49 in additionto enrolling only second year residents in the chartaudit curriculum intervention, completed a multivari-ate analysis to account for the trainee’s experience asan auditor, the duration of exposure to the curriculumindependent of training level and number ofcomorbidities in the resident’s patient sample.

Synthesis of results and realist reviewWe identified several major themes through realistreview. These themes are organised by ‘what works,’‘for whom,’ ‘under what circumstances’ and ‘toachieve what outcomes’ (table 3). After synthesisingthe range of interventions, clinical and educationaloutcomes, the methodological quality and realistreview of the 39 studies, we tested the candidatetheory and conceptual framework by iteratively ana-lysing the major themes which emerged from thedata. Specifically, we looked for evidence of a prede-termined curriculum and educational context, as wehypothesised at the outset. However, we failed to findevidence of these mechanistic and contextual factorsas important determinants in producing physicianswho are lifelong learners and improvers. Thus, werevised the candidate theories and developed a con-ceptual framework (figure 2).Many different types of curricula are described in

the included studies, some of which are distinct educa-tional interventions and some of which involve traineesin existing clinical QI in their practices. Among theincluded studies, these interventions fell into four cat-egories, and we found examples of statistically signifi-cant improvements in clinical outcomes in eachcategory. Those studies with ‘fair’ methodologicalquality scores did not report significant results or failedto report numerical results with statistical analysis.Therefore, the higher-quality studies suggest that, withcertain contexts and mechanisms, significant improve-ment in clinical outcomes is achievable when traineesare exposed to QI within the clinical setting.Several success characteristics were common to differ-

ent contexts of clinical QI education (table 3; for illus-trative quotes see online supplement 6). As noted bySockalingham et al56 ‘residents identified workload as amajor barrier to (doing QI work).’ Successful QI teach-ing programmes were consistently clear about the timerequired for trainee work-hour rules, competingdemands and for faculty involvement. Success of spe-cific programmes will also depend on whether it makesmore sense to train all faculty members in QI principlesor to have a dedicated select faculty group in charge ofthe QI curriculum. The availability of data throughinformation systems is also a facilitator to trainee satis-faction and engagement. The sustained improvementreported by Halverson et al36 was achieved through theuse of timely regular data feedback to all providers,including trainees, about the care of patients with dia-betes in the practice. Two studies also highlighted thechallenges that occur when trainees must abstract theirown data, such as is the case in a practice which hasnot implemented an electronic medical record, orwhen data feedback is not timely enough for continuousQI.34 44 Choice of the project topic is also important fortrainees. QI educators need to consider the needs of theclinical setting as well as level of trainee; however, noconsensus emerged as to the best approach.

Table 1 Baseline characteristics of included studies

Characteristic n

Study design

Controlled trials 2

Before–after studies 18

Case reports 10

Time series 7

Interrupted time series 1

Qualitative 1

Population

Residents only 27

Medical students only 3

Fellows only 3

Residents and fellows 3

Medical students and residents 2

Medical students, residents and fellows 1

Specialty

Internal medicine 10

Family medicine 7

Psychiatry 4

Paediatrics 3

Critical care 2

Surgery 2

Neonatology 1

Preventive medicine 1

Radiology 1

Various specialties 5

Medical schools 3

Clinical setting

Outpatient

Primary care 15

Psychiatry 2

Paediatrics 1

Inpatient

General medical unit 2

Adult intensive care unit 2

Neonatal intensive care unit 1

Paediatric emergency department 1

Radiology 1

Systematic review




j.com/

BM

J Qual S


eptember 2014. D

ownloaded from


Table 2 Summary of interventions, outcomes and methodological quality

Author and year Population (n) Intervention descriptionClinicalsetting

Clinical outcomes

Educational outcomesQualityassessment†Finding

Statisticallysignificant*

Chart audit

Gould et al 200244 2nd year medicalstudents (77)

Learners audited random sampleof diabetes charts usingQualidigm, before and after‘Project in a Box’

Outpatientprimary careclinic

Improved rates of Hba1c, foot exam andeye exam documentation

Yes Improved knowledge, skills,attitudes; poor learnersatisfaction

Good

Paukert et al 200348 FM residents (36) Self and peer audits done atseveral time points


Improved, but unsustained, composite‘preventive’ score

Yes NR Good

Holmboe et al 200545 2nd year IMresidents (26)

Learners audited sample of theirown diabetes patients usingQualidigm


Improved patient care (Hba1c, LDL) andsystem performance (monofilamenttesting documentation)

Yes Improved perceived value andmotivation

Very good

Kaddan et al 200646 Residents andmedical students(NR)

Structured session every 24 hrs ofall X-ray and culture reportsamong ED patients

Paediatricemergencydepartment

Decrease in chart requiring attendingcomment and change in treatmentcourse

Yes High learner satisfaction Good

Krajewski et al 200751 Residents (NR) Performed audit on self-identifiedproblem in radiology reporting;presented at M&M conference

Radiologydepartment

Multiple system improvementsimplemented

No NR Fair

Asao et al 200949 2nd and 3rd IM yearresidents (80)

Peer audit performed; feedbackand motivational plan proposed bylearner


Improved composite measures for COPD,CHD, DM, HTN, LVF

No NR Very good

Carek et al 200950 FM residents (20) Chart audit before and aftereducational session


Improvement in oral and mental health,care of underserved populations,antibiotic use, elderly care, preventiveservices

No NR Good

Kirschenbaum et al201047

CC fellows andresidents (NR)

Team audit and analysis of alltransfers from GMU to MICU

MICU Improvement in number of cardiacarrests and deaths, number of timesMET called

Yes NR Good

Smith et al 201252 IM residents (20) Peer RCA on near-miss cases,presented at M&M conference

Variousdepartments

Multiple system improvementsimplemented, some successful

No High learner satisfaction Fair

Participant on clinical QI team

Coleman et al 200334 FM residents (NR) Participation at training site QIteam; learner chose projects andimprovement approaches


Improved microalbuminuria screening,medication list completion, datasummary sheets

Yes NR Good

Mohr et al 200338 Peds residents (8) Resident team chose project forinterprofessional practice teamimprovement

Outpatientpaediatric clinic

Improved vaccination rates for thepractice

Yes NR Good

Landis et al 200637 FM residents (126) Residents involved in statewidelearning collaborative to improvediabetes care


Improved ACIC scores NR NR Good

Continued

Systematic

revie

w

82

JonesAC,etal.BM

JQualSaf2015;24:77

–88.doi:10.1136/bmjqs-2014-002846

on April 13, 2020 by guest. Protected by copyright. http://qualitysafety.bmj.com/ BMJ Qual Saf: first published as 10.1136/bmjqs-2014-002846 on 30 September 2014. Downloaded from


Table 2 Continued


Clinical outcomes



Halverson et al200736

FM residents (NR) Resident served on committee toimprove diabetes care


Improved percentage of patients at goalfor Hba1c, LDL, BP

Yes NR Good

Stapleton et al 200942 Peds senior residents(NR)

RPIW on improving senior residentrotation

Paediatricinpatient service

Improved system performance measures NR NR Fair

Buckley et al 201032 CC fellows andresidents (NR)

Mandatory participation on MICUimprovement team

MICU Improved iatrogenic pneumothorax rates,sepsis-specific mortality, sepsis bundlecompliance

Yes NR Good

Fischman et al 201035 IM residents (4) Controlled trial with involvementon clinical QI team


Improvement in no-show, continuitymeasures, doctor-patient relationships

NR NR Good

Stevens et al 201039 IM, FM, pedsresidents (NR)

Involvement in statewidecollaborative for diabetes care


Improved ACIC scores Yes High learner participation Good

Yu et al 201041 FM residents (6) Resident participation in statewidecollaborative to improve diabetescare


Improved ACIC scores NR Improved % of residentsreviewing performance reports,demonstrating improvedbehaviours

Good

Vidyarthi et al 201143 Fellows andresidents (NR)

Financial incentive (US$1200 pertrainee) for involvement inimprovement work at medicalcentre

Variousdepartments

Several system performance measuresimproved

NR NR Fair

Stueven et al 201240 Residents andmedical students(249)

Residents surveyed for relevantpatient safety and QI improvementissues, attended retreats

Variousdepartments

Several system performance measuresimproved

NR NR Good

Carey et al 201333 Neonatology fellows(3)

Fellows participate on Q and Scommittee, chair working groupwhen issues arise

NICU Improvement in broncho-pulmonarydisease, catheter-associated bloodstreaminfections

NR Learner self-reflections aboutQI, presentations atconferences

Good

Team project

Varkey et al 200631 Fellows, residents,medical students (7)

Multidisciplinary trainee teams intraining hospital

Variousdepartments

Improvement in documentation ofmedication reconciliation

NR Improved QIKAT scores Good

Oyler et al 200823 2nd year IMresidents (34)

Mandatory project while onambulatory rotation


Improvement in documentation ofheight, weights, BMI

Yes NR Good

Varkey et al 200830 Preventive medicine,endocrinologyfellows (9)

QI project taught jointly bymedicine and engineering faculty

Variousdepartments

Improvement in patient understandingon treatment

NR Improved QIKAT scores,learner satisfaction

Fair

Continued

Systematic

review

JonesAC,etal.BM

JQualSaf2015;24:77

–88.doi:10.1136/bmjqs-2014-002846

83



Table 2 Continued


Clinical outcomes



Tomolo et al 200929 IM residents (42) QI project while rotating oninpatient medicine service

Variousdepartments

Multiple sustained system performancemeasures improved (i.e., missing labvalues)

NR High learner satisfaction Good

Varkey et al 200926 Preventive medicinefellows (19)

Fellows develop and implementprojects over 4-week rotation

Variousdepartments

Multiple system performance and patientcare measures improved

Yes Improved QIKAT scores, OSCEperformance, learnersatisfaction

Good

Diaz et al 201020 2nd and 3rd yearFM residents (61)

Clinical scholars curriculumdelivered over 1 year of residency


Multiple system performance and patientcare measures improved

Yes Increase in number ofpublications and presentations

Good

Shiner et al 201025 Psychiatry residents(12)

Aiming to improve care for majordepressive disorder

Outpatientpsychiatry clinic

Improved percentage of patients seenwithin 6 weeks of starting MDDtreatment

Yes NR Good

Clark et al 201128 General surgeryresidents (33)

Team completed needs assessmentand improvement in signoutprocess by template

Inpatient surgeryservice

Improvement in signout processes NR NR Good

Dysinger et al 201121 4th year medicalstudents (510)

Required rotation, putting studentsinto practices focused onimprovement

Variousdepartments

Improved documentation Yes Some improved learnersatisfaction

Very good

Laiteerapong et al201122

IM residents (10) Team project to improvedocumentation


Improvement in height, weight, BMIdocumentation

Yes Improved learner satisfaction Good

Ogrinc et al 20114 2nd year medicalstudents (22)

Health Leadership PracticumElective brings student teams intoclinical settings in need ofimprovement

Variousdepartments

Multiple reported improvements (i.e.,urine samples screened in pregnantwomen)

NR NR Good

Oyler 201124 IM residents (64) Required rotation encouragingteams to develop projects


Improvement in ASA use,documentation on BMI and smokinghistory

Yes NR Good

Arbuckle et al 201327 3rd year psychiatryresidents (12)

Longitudinal QI curriculum over37 weeks, including longitudinalproject

Outpatientpsychiatry clinic

Improvement in monitoring ofdepression symptoms, screening

NR Improved QIKAT scores Good

Individual project

Weingart et al 200457 2nd and 3rd year IMresidents (26)

RCA/QI project on voluntaryelective

Variousdepartments

Improvement in system performance andpatient care measures

NR Improved learner satisfaction Fair

Canal et al 200753 3rd year generalsurgery residents(15)

Mandatory QI project on researchtime

Variousdepartments

Improvement in several systemperformance measures

NR Improved QI curriculum pre/post test

Good

Continued

Systematic

revie

w

84

JonesAC,etal.BM

JQualSaf2015;24:77

–88.doi:10.1136/bmjqs-2014-002846



DISCUSSIONHow can our findings inform a comprehensive modelof teaching, learning and doing QI in medical educa-tion? Our realist analysis and conceptual framework(figure 2) suggest that clinical education, whether pro-viding care to an individual patient or doing QI work,does not begin with the curriculum. It begins with thetrainee and patient at the centre of a healthcare systemthat encompasses many institutional levels. The traineeexists within two overlapping worlds: educational andclinical. The educational world is comprised of teach-ing faculty and together they exist within the trainingprogramme. The clinical world is made up of thepatient and family at the centre of an interprofessionalcare team, of which the trainee is one part. Theoutcome of the educational world is improved traineeknowledge, skills and attitudes. The outcome of theclinical world is improved patient care and system per-formance. While these are often seen as separate, QI isone key area that exposes the inter-related complexitiesof the educational and care-delivery systems. In orderto produce physicians who are capable of and believethat their duty is to do their work and improve theirwork, we should rethink the conventional wisdom

Table2

Continued

Autho

ran

dyear

Popu

latio

n(n)

Interven

tionde

scrip

tion

Clinical

setting

Clinical

outcom

es

Educationa

loutcomes

Qua

lity

assessmen

t†Find

ing

Statistic

ally

sign

ificant*

Sockalingham

etal

2010

562ndand3rdyear

psychiatryresidents

(40)

MandatoryQIprojectwhileon

rotation

Various

psychiatry

departm

ents

Improvem

entinconsult/referralservices

NR

Informationby

focusgroup

andquestionnaire

Good

Oujirietal2011

543rdyear

IMresidents

LearnerschoseQIprojectas

part

ofelectiverotation

Outpatient

primarycare

clinic

Improvem

entinseveralpatient

careand

system

performance

measuresforh

eart

failurepatients

Yes

NR

Good

Reardonetal2011

55Psychiatryresidents

(16)

Learnerassignedto

unitand

completed

mandatoryproject

Various

departm

ents

Improvem

entinconsultp

rocesses

NR

Improved

QIKAT

scores

Good

*If‘Yes’themajority

offindingsintheresults

ofthestudyhadto

beshow

nto

bestatistically

significant,eitherby

pvalueor

statisticalprocesscontrol(SPC)

analysis.

†Wemodified

methodologicalqualityassessmentof

thesestudiesto

take

into

accountthe

important

factorsof

educationalliterature:clearaccountingforsystematicbias,clear

descriptionof

theintervention,

learning

objectivesarticulated

foreducationalinterventions,m

inimisationof

otherexposures

which

couldhave

impacton

theoutcom

e,minimisationof

bias

andconfounding.

QI,quality

improvem

ent;AC

IC,assessm

ento

fchronicillnesscare;A

SA,aspirin;BM

I,Body

MassIndex;BP,blood

pressure;C

C,criticalcare;CH

D,coronaryheartd

isease;CO

PD,chronicobstructive

pulmonarydisease;DM

,diabetes

mellitus;ED,

emergencydepartm

ent;FM

,fam

ilymedicine;G

MU,

generalm

edicalunit;Hb

a1c,haem

oglobina1c;HTN,h

ypertension;

IM,internalm

edicine;LDL,low

density

lipoprotein;LVF,leftventricularfailure;

M&M

,morbidityandmortality;MET,m

edicalemergencyteam

;MICU,

medicalintensivecareunit;

N,n

o;NICU,

neonatalintensive

careunit;

NR,

notreported;

QIKAT,qualityimprovem

entknowledgeassessmenttest;OSCE,

objective

structured

clinicalexamination;RC

A,root

causeanalysis;

RPIW,rapidprocessimprovem

entw

orkshop;

Y,yes.

Table 3 Relevant themes from realist review

What works Accurately account for the time it takes to deliverQI education in the clinical setting due tocompeting demands, existing work load oftrainees, and work-hour rulesIdentifying educational and clinically relevantproject topics is challengingConsider having trainees choose their own projectChoose topics of clinical importanceUse near misses as a way to identify system errors

For whom Medical students can, and should be expected, tocontribute to quality of care in the clinical settingResidents are front-line providers and have deepinsights into the clinical processes and theknowledge for improvement within the system

Under whatcircumstances

Successful QI teaching in the clinical settingrequires support from both educational and caredelivery leaders and the work of the traineesData are critical. The availability of data, especiallythrough health information technologies, has adirect positive impact on learner satisfaction andengagementOpportunities for interprofessional engagementand education can be found in teaching about QIwithin the clinical settingPrograms can be successful by either engaging allfaculty around QI or by having dedicated QIfaculty for teaching QI within the clinical setting

To achieve whatoutcomes

There is lack of clarity around whether educationaland clinical outcomes are of equal or relativehierarchical importanceSustainability is important for the clinical settingand the trainee. Sustainable projects can impactthe culture of the clinical setting, butunsustainable projects may leave the trainee andother participants disheartened aboutimprovement work

QI, quality improvement.

Systematic review




j.com/

BM

J Qual S


eptember 2014. D

ownloaded from


around education and acquisition of improvementknowledge and skills. As Batalden and Davidoff58

wrote, ‘Learning how to do quality improvement andactually carrying out quality improvement are essen-tially one and the same; both are special forms ofexperiential learning.’ In fact, medical education is cur-rently embracing this culture in teaching clinical skillsusing Adult Learning Theory, which reminds us thatprofessionals learn best when they see a need to acquirethe knowledge and skills for fulfilment of goals.59 Iftrainees see that faculty are asking questions of theprocess and needing to learn more to improve thesystem, then they have the opportunity to engage withthem.60 Interestingly, Asch et al61 have demonstratedthat within obstetrical residencies, it is possible and,perhaps beneficial, to rank obstetrical programmesbased on overall performance on clinical rather thaneducational outcomes, representing yet another innov-ation for the future of the medical profession. TheACGME has acknowledged the importance of the clin-ical learning environment as an essential component ofresident education and, accordingly, adopted theClinical Learning Environment Review, ‘to generatenational data on programme and institutional attri-butes that have a salutary effect on quality and safety insettings where residents learn and on the quality ofcare rendered after graduation.’62 Through the knowl-edge, skills and attitudes that trainees achieve duringtheir educational programme while taking part in theimprovement of the clinical care, we expect to nurturelifelong learners and improvers who will advance clin-ical improvements in patient care and systemperformance.This realist review has limitations that begin with

the known publication bias in this field. This biaswas corroborated in our review, as none of the pub-lished studies described a clinical process that wasworsened. Not sharing failed pilots and curriculum

limits the learning that can occur across programmes.Also, although our search strategy allowed us toanalyse any circumstance of trainees being involvedin QI in the clinical setting, this may have caused usto falsely criticise studies with primarily clinicalinterventions, because they did not aim to prove thatinvolving trainees made a difference in their clinicaloutcomes. The realist review process, however, helpsto differentiate these studies and their importantqualitative information about ‘what works,’ ‘forwhom,’ ‘under what circumstances’ and ‘to achievewhat outcomes’.Because a methodological quality tool does not

exist for assessment of the QI education literature, wecreated a tool by combining elements of existing vali-dated tools. Although we did not use a validatedinstrument, the tool we developed contains the speci-ficity for the QI education literature and thus madethe quality assessment more rigorous. We did not,however, identify any ‘excellent’ quality studies (seeonline supplement 5). Although we found that all thelowest-quality studies did not demonstrate significantresults, there were no strong studies to show improve-ment in both clinical and educational outcomes. Weidentified many studies with minor weaknesses, andthe realist review process helps to glean the notablecharacteristics from these data. More high-qualitystudies would take steps to minimise bias in the studypopulation, clearly describe the intervention and edu-cational objectives, minimise other exposures orsecular trends that could have accounted for theresults, analyse results with enumerative or analyticstatistics, explain all biases and confounders andreport funding sources. Among the studies in thisreview, many would have been improved by particularattention to minimising bias and confounding in thestudy population, and a clear articulation of educa-tional objectives.

Figure 2 A conceptual framework describing the relationships between the contexts, mechanisms and outcomes for qualityimprovement (QI) in medical education.

Systematic review




j.com/

BM

J Qual S


eptember 2014. D

ownloaded from


CONCLUSIONThe studies in this review reported on many moreclinical outcomes than had been described in previousreviews, in large part due to the development of clin-ically oriented QI programmes since the priorreviews. Using the realist approach allowed us theadvantage of synthesising these data to not justupdate, but reconceptualise (figure 2) the currentlandscape of QI teaching. Advances in teaching anddoing QI has made tremendous strides in the pastdecade, but further work is needed to determine thefactors that reliably facilitate the development of phy-sicians who will believe and are capable of doing theirwork and improving their work—ultimately, physi-cians who are lifelong learners and improvers.

Acknowledgements The authors thank Tom Mead, MLS,Reference Librarian, Biomedical Libraries, Dartmouth College,for assistance with development of the search strategy; AuroraLeute Matzkin, PhD and Martha Reagan-Smith, MD, EdD(Professor Emerita, Geisel School of Medicine) for theirassistance in reading drafts of the work and preparing theconceptual framework.

Contributors The conception or design of the work andinterpretation of data was performed by ACJ, SAS and GO. Theacquisition and analysis of data was performed by ACJ and GO.The manuscript was drafted by ACJ, and revised critically forimportant intellectual content by ACJ, SAS and GO. All authorsapproved the final version of the manuscript. ACJ, SAS and GOagree to be accountable for all aspects of this work and willensure that questions related to the accuracy or integrity of anypart of the work are appropriately investigated and resolved.

Funding This work was supported by the VA National QualityScholars Fellowship Program, the VA Office of AcademicAffiliations, and the Geisel School of Medicine at DartmouthOffice of Health Systems and Clinical Improvement with theuse of facilities and materials from the White River Junction VAin White River Junction, VT.

Competing interests ACJ and SAS have no competing intereststo report. GSO is an Associate Editor of BMJ Quality andSafety but otherwise has no competing interests to report.

Provenance and peer review Not commissioned; externallypeer reviewed.

Data sharing statement ACJ had full access to all the data in thestudy, and takes responsibility for the integrity of the data andthe accuracy of the data analysis.

REFERENCES1 Kohn LT, Corrigan JM, Donaldson MS, eds. To err is human:

building a safer health system. Washington, DC: NationalAcademy Press, 2000.

2 ACGME Common Program Requirements. 2012. http://www.acgme.org/acgmeweb/ProgramandInstitutionalGuidelines.aspx(accessed 27 Dec 2012).

3 Patterson BR, Kimball KJ, Walsh-Covarrubias JB, et al.Effecting the sixth core competency: a project-basedcurriculum. Am J Obstet Gynecol 2008;199:561.e561–566.

4 Ogrinc G, Nierenberg DW, Batalden PB. Building experientiallearning about quality improvement into a medical schoolcurriculum: the Dartmouth experience. Health Aff2011;30:716–22.

5 Josephson SA, Engstrom JW. Residency Training: developing aprogram of quality and safety to train resident neurologists forthe future. Neurology 2012;78:602–5.

6 Boonyasai R. Effectiveness of teaching quality improvement toclinicians: a systematic review. JAMA 2007;298:1023–37.

7 Ogrinc G, Headrick LA, Mutha S, et al. A framework forteaching medical students and residents about practice-basedlearning and improvement, synthesized from a literature review.Acad Med 2003;78:748–56.

8 Patow CA, Karpovich K, Riesenberg LA, et al. Residents’engagement in quality improvement: a systematic review of theliterature. Acad Med 2009;84:1757–64.

9 Wong BM, Etchells EE, Kuper A, et al. Teaching qualityimprovement and patient safety to trainees: a systematic review.Acad Med 2010;85:1425–39.

10 Wong BM, Levinson W, Shojania KG. Quality improvement inmedical education: current state and future directions. MedEduc 2012;46:107–19.

11 Pawson R, Greenhalgh T, Harvey G, et al. Realist review—anew method of systematic review designed for complex policyinterventions. J Health Serv Res Policy 2005;10:21–34.

12 Shepperd S, Lewin L, Straus S, et al. Can we systematicallyreview studies that evaluate complex interventions? PLoS Med2009;6:e1000086.

13 Boonyasai RT, Windish DM, Chakraborti C, et al. Effectivenessof teaching quality improvement to clinicians—a systematicreview. JAMA 2007;298:1023–U1039.

14 Scott Shipman M, MPH; Course Director, Capstone Series atThe Dartmouth Institute for Health Policy and ClinicalPractice; Director for Primary Care Affairs and WorkforceAnalysis, American Association for Medical Colleges.

15 Benneyan JC, Lloyd RC, Plsek PE. Statistical process control asa tool for research and healthcare improvement. Qual SafHealth Care 2003;12:458–64.

16 Reed DA, Cook DA, Beckman TJ, et al. Association betweenfunding and quality of published medical education research.JAMA 2007;298:1002–9.

17 Higgins JPT, Altman DG. Assessing risk of bias in includedstudies. Cochrane Handbook for Systematic Reviews ofInterventions: John Wiley & Sons, Ltd, 2008:187–241.

18 Ogrinc G, Mooney SE, Estrada C, et al. The SQUIRE(Standards for QUality Improvement Reporting Excellence)guidelines for quality improvement reporting: explanation andelaboration. Qual Saf Health Care 2008;17:13–32.

19 Singh MK, Ogrinc G, Cox KR, et al. The QualityImprovement Knowledge Application Tool—Revised(QIKAT-R). Acad Med 2014. In press.

20 Diaz VA, Carek PJ, Dickerson LM, et al. Teaching qualityimprovement in a primary care residency. Jt Comm J QualPatient Saf 2010;36:454–60.

21 Dysinger WS, Pappas JM. A fourth-year medical schoolrotation in quality, patient safety, and population medicine.Am J Prev Med 2011;41(4 Suppl 3):S200–5.

22 Laiteerapong N, Keh CE, Naylor KB, et al. A resident-ledquality improvement initiative to improve obesity screening.Am J Med Qual 2011;26:315–22.

23 Oyler J, Vinci L, Arora V, et al. Teaching Internal Medicineresidents quality improvement techniques using the ABIM’spractice improvement modules. J Gen Intern Med2008;23:927–30.

24 Oyler J, Vinci L, Johnson JK, et al. Teaching internal medicineresidents to sustain their improvement through the qualityassessment and improvement curriculum. J Gen Intern Med2011;26:221–5.

25 Shiner B, Green RL, Homa K, et al. Improving depressioncare in a psychiatry resident psychopharmacology clinic:

Systematic review




j.com/

BM

J Qual S


eptember 2014. D

ownloaded from

http://www.acgme.org/acgmeweb/ProgramandInstitutionalGuidelines.aspx




measurement, monitoring, feedback and education. Qual SafHealth Care 2010;19:234–8.

26 Varkey P, Karlapudi SP. Lessons learned from a 5-yearexperience with a 4-week experiential quality improvementcurriculum in a preventive medicine fellowship. J Grad MedEduc 2009;1:93–9.

27 Arbuckle MR, Weinberg M, Cabaniss DL, et al. Trainingpsychiatry residents in quality improvement: an integrated,year-long curriculum. Acad Psych 2013;37:42–5.

28 Clark CJ, Sindell SL, Koehler RP. Template for success: using aresident-designed sign-out template in the handover of patientcare. J Surg Educ 2011;68:52–7.

29 Tomolo AM, Lawrence RH, Aron DC. A case study oftranslating ACGME practice-based learning and improvementrequirements into reality: systems quality improvement projectsas the key component to a comprehensive curriculum. Qual SafHealth Care 2009;18:217–24.

30 Varkey P, Karlapudi SP, Bennet KE. Teaching qualityimprovement: a collaboration project between medicine andengineering. Am J Med Qual 2008;23:296–301.

31 Varkey P, Reller MK, Smith A, et al. An experientialinterdisciplinary quality improvement education initiative. Am JMed Qual 2006;21:317–22.

32 Buckley JD, Joyce B, Garcia AJ, et al. Linking residency trainingeffectiveness to clinical outcomes: a quality improvementapproach. Jt Comm J Qual Patient Saf 2010;36:203–8.

33 Carey WA, Colby CE. Educating fellows in practice-basedlearning and improvement and systems-based practice: thevalue of quality improvement in clinical practice. J Crit Care2013;28:112.e1–5.

34 Coleman MT, Nasraty S, Ostapchuk M, et al. Introducingpractice-based learning and improvement ACGME corecompetencies into a family medicine residency curriculum.Jt Comm J Qual Saf 2003;29:238–47.

35 Fischman D. Applying Lean Six Sigma methodologies toimprove efficiency, timeliness of care, and quality of care in aninternal medicine residency clinic. Qual Manag Health Care2010;19:201–10.

36 Halverson LW, Sontheimer D, Duvall S. A residency clinicchronic condition management quality improvement project.Fam Med 2007;39:103–11.

37 Landis SE, Schwarz M, Curran DR. North Carolina familymedicine residency programs’ diabetes learning collaborative.Fam Med 2006;38:190–5.

38 Mohr JJ, Randolph GD, Laughon MM, et al. Integratingimprovement competencies into residency education: a pilotproject from a pediatric continuity clinic. Ambul Pediatr2003;3:131–6.

39 Stevens DP, Bowen JL, Johnson JK, et al. A multi-institutionalquality improvement initiative to transform education forchronic illness care in resident continuity practices. J GenIntern Med 2010;25(Suppl 4):S574–80.

40 Stueven J, Sklar DP, Kaloostian P, et al. A resident-ledinstitutional patient safety and quality improvement process.Am J Med Qual 2012;27:369–76.

41 Yu GC, Beresford R. Implementation of a chronic illness modelfor diabetes care in a family medicine residency program.J Gen Intern Med 2010;25(Suppl 4):S615–19.

42 Stapleton FB, Hendricks J, Hagan P, et al. Modifying theToyota Production System for continuous performanceimprovement in an academic children’s hospital. Pediatr ClinNorth Am 2009;56:799–813.

43 Vidyarthi AR, Baron RB. Financial incentives for residents andfellows: a disruptive innovation to drive quality improvement.Acad Med 2011;86:1338.

44 Gould BE, Grey MR, Huntington CG, et al. Improving patientcare outcomes by teaching quality improvement to medicalstudents in community-based practices. Acad Med 2002;77:1011–18.

45 Holmboe ES, Prince L, Green M. Teaching and improvingquality of care in a primary care internal medicine residencyclinic. Acad Med 2005;80:571–7.

46 Kaddan W, Poznansky O, Amir L, et al. Medical education andquality of care in the pediatric emergency department setting:a combined model. Eur J Emerg Med 2006;13:139–43.

47 Kirschenbaum L, Kurtz S, Astiz M. Improved clinical outcomescombining house staff self-assessment with an audit-based qualityimprovement program. J Gen Intern Med 2010;25:1078–82.

48 Paukert JL, Chumley-Jones HS, Littlefield JH. Do peer chartaudits improve residents’ performance in providing preventivecare? Acad Med 2003;78(Suppl 10):S39–41.

49 Asao K, Mansi IA, Banks D. Improving quality in an internalmedicine residency program through a peer medical recordaudit. Acad Med 2009;84:1796–802.

50 Carek PJ, Dickerson LM, Boggan H, et al. A limited effect onperformance indicators from resident-initiated chart audits andclinical guideline education. Fam Med 2009;41:249–54.

51 Krajewski K, Siewert B, Yam S, et al. A quality assuranceelective for radiology residents. Acad Radiol 2007;14:239–45.

52 Smith KL, Ashburn S, Rule E, et al. Residents contributing toinpatient quality: blending learning and improvement. J HospMed 2012;7:148–53.

53 Canal DF, Torbeck L, Djuricich AM. Practice-based learning andimprovement: a curriculum in continuous quality improvementfor surgery residents. Arch Surg 2007;142:479–82; discussion482–473.

54 Oujiri J, Hakeem A, Pack Q, et al. Resident-initiatedinterventions to improve inpatient heart-failure management.[Reprint in Postgrad Med J 2011;87:700–5; PMID: 21954033].BMJ Qual Saf 2011;20:181–6.

55 Reardon CL, Ogrinc G, Walaszek A. A didactic andexperiential quality improvement curriculum for psychiatryresidents. J Grad Med Educ 2011;3:562–5.

56 Sockalingham S, Stergiopoulos V, Maggi J, et al. Qualityeducation: a pilot quality improvement curriculum forpsychiatry residents. Med Teach 2010;32:e221–6.

57 Weingart SN, Tess A, Driver J, et al. Creating a qualityimprovement elective for medical house officers. J Gen InternMed 2004;19:861–7.

58 Batalden P, Davidoff F. Teaching quality improvement—thedevil is in the details. JAMA 2007;298:1059–61.

59 Adult Learning Theory. http://www.qotfc.edu.au/resource/?page=65375

60 Cooke M, Ironside PM, Ogrinc GS. Mainstreaming quality andsafety: a reformulation of quality and safety education forhealth professions students. BMJ Qual Saf 2011;20(Suppl 1):i79–82.

61 Asch DA, Nicholson S, Srinivas S, et al. EValuating obstetricalresidency programs using patient outcomes. JAMA 2009;302:1277–83.

62 ACGME. Clinical Learning Environment Review. 2014. http://www.acgme.org/acgmeweb/tabid/436/ProgramandInstitutionalAccreditation/NextAccreditationSystem/ClinicalLearningEnvironmentReviewProgram.aspx

Systematic review




j.com/

BM

J Qual S


eptember 2014. D

ownloaded from

http://www.qotfc.edu.au/resource/?page=65375



http://www.acgme.org/acgmeweb/tabid/436/ProgramandInstitutionalAccreditation/NextAccreditationSystem/ClinicalLearningEnvironmentReviewProgram.aspx






Documents

SYSTEMATIC REVIEW Key characteristics of successful quality improvement … · patient care and system performance outcomes. The realist review approach offers one approach to decon-struct