Upload
dokien
View
235
Download
6
Embed Size (px)
Citation preview
FINAL REPORT
Healthcare Improvement and Rapid PDSA Cycles of Cha nge: A
Realist Synthesis of the Literature
Esther Curnock, MPH MRCGP Julie Ferguson, PhD John McKay, MD FRCGP Paul Bowie, PhD FRCP (Edin) NHS Education for Scotland Department of Postgraduate General Practice Education 2 Central Quay GLASGOW G3 8BW
Further information: [email protected] The study was commissioned and funded by the Patient Safety Multi-Professional Steering Group of NHS Education for Scotland.
Abstract Background
Plan-Do-Study-Act (PDSA) is a quality improvement (QI) tool used in healthcare. It
pursues effective changes in healthcare processes to enhance health outcomes, using
rapid small-step change cycles. There is limited research into the effectiveness of QI
programme components. This systematic ‘realist synthesis’ aimed to identify and
summarise the implementation and contextual factors that may influence the
effectiveness of PDSA.
Methods
We established outline theories relating to whether, how and why PDSA works by
reviewing recent papers. We systematically searched for articles in Medline, Embase,
British Nursing Index, PsycINFO, HMIC, MIDIRS and EBM (2005-2009). Articles
referencing or assumed to use PDSA (e.g. having employed the Institute of Healthcare
Improvement (IHI) collaborative methodology) were included. Any outcome measure
reported as leading to QI was included, whether or not it was clinical.
A data extraction form, informed by PRISMA guidelines was employed. Using agreed
inclusion and exclusion criteria we selected papers which would add to, or refine, the
theories. Articles were added through snowballing from citation lists. The
data/findings were extracted and synthesised according to the contextual and
implementation issues which facilitated or limited successful PDSA outcomes.
Results
94 articles were reviewed as relevant to our research question with a total of 44
contributing to our evaluative framework and full theory development. PDSA has led
to changes in practice and health outcomes in a variety of settings, geographies and
patient groups. Several factors were identified as influencing effectiveness. These
included: organisational size, context and stability; setting clear and measurable
improvement goals underpinned by evidence; staff engagement and ownership; speed
and size of cycles; actions taken to spread and sustain changes; and resources.
Conclusions
PDSA may be more successful in smaller, defined settings with stable staff and patient
groups. Culture seems more important than size. Success is more likely where:
evidence linking process goals to intended outcomes exists; is understood and visible to
staff; the workforce is actively engaged; and, staff have prior QI experience. Although
difficult to achieve, quicker and shorter PDSA cycles may lead to faster/more sustained
practice change. Improvements are spread and sustained more when progress and
associated data are shared and visible. Existing support structures and additional
resources lead to greater success.
Registration:
Not applicable
Keywords:
PDSA, Quality Improvement, Patient Safety, Realist Synthesis, Evidence Review,
Healthcare
Background
PDSA (Plan-Do-Study-Act) is a tool that is utilised in a diverse range of quality
improvement (QI) programmes and projects in healthcare systems worldwide. The aim
of PDSA is to pursue, sustain and rollout effective changes in care processes that
favourably affect outcomes, using rapid small-step change cycles. The model involves
four cyclical stages: hypothesis formation (Plan); implement the new process with data
collection (Do); interpreting the results (Study); and a decision as to what to do next
(Act) [1]
PDSA (also known as Plan-Do-Check-Act, PDCA) was developed as a method for
achieving efficiency in Japanese car manufacturing [2] and was influenced by earlier
work on industrial statistical quality control [3]. It was one of the key tools developed
as part of the ‘Total Quality Management’ movement [4]. Langley et al [5] used PDSA
as the central component of their ‘Model for Improvement’, and were arguably the first
to propose the use of PDSA cycles in health care QI.
The Model for Improvement is popularised by the Institute of Healthcare Improvement
(IHI) through their ‘Breakthrough Series’ collaborative approach [6]. IHI methods are
strongly influenced by ‘industrial’ continuous QI methods that originated in
manufacturing industries and organisational change theories that are perceived to be
complementary [7]. In the United Kingdom (UK) in recent years several large-scale QI
programmes such as the Scottish Patient Safety Programme [8] and the Health
Foundation’s Pilot ‘Safer Patients Initiative’ [9] have been implemented in partnership
with the IHI. However, empirical evidence of the effectiveness of QI programmes, as
well as evaluation about their impact on healthcare quality is limited [10]. Few
controlled studies of QI collaboratives have been undertaken, and the results are
equivocal. For example, Landon et al [11] evaluated the effectiveness of a QI
collaborative to improve HIV patient care using a prospective controlled study design,
involving 9986 patients in 69 clinics. No significant differences in quality of care
measures were found between control and intervention clinics).
A systematic review of QI collaboratives from 1995 to 2006 had limited scope. In it a
small number of controlled trials showed limited positive effect and two showed no
significant benefit [12]. Their objective was specifically to assess for empirical
evidence of effectiveness, but they were unable to draw any conclusions from the vast
majority of reports due to uncontrolled study design. A systematic review of the
effectiveness of teaching QI methods to clinicians also had mixed results [13].
A common criticism of QI initiatives has been their tendency to favour action over
evidence, with the rapid dissemination of innovative but unproven initiatives, with
potential unintended consequences of harm and opportunity cost [14-16].
Although evidence of the effectiveness of many QI approaches is lacking [17] several
factors that appear to be critical to the success or failure of QI collaborative projects
have been identified. A review [13] found that using incremental tests of change (i.e.
the PDSA approach), and having access to performance data appeared to be important
factors, but were themselves insufficient for ensuring beneficial outcomes. The review
was unable to analyse the determining factors any further or make specific
recommendations regarding curricula.
Using randomised control trial (RCT) designs to evaluate interventions where the
context, content, and implementation have a high degree of heterogeneity can be
problematic. Such methods when applied to less stable and complex interventions, and
when unaccompanied by a detailed process evaluation, may fail to identify that
interventions work in some instances rather than others, or to explain why they work
(i.e. the potential casual mechanisms). This reduces the generalisability of both
research and subsequent review conclusions.
In addition, strict eligibility criteria mean that much useful contextual information has
been lost in the existing traditional systematic review literature, including information
required by healthcare managers and policy makers to help them decide whether to
implement a specific QI approach in their local setting [18].
The need for an alternative approach to evaluation and the systematic review of QI
programmes to elicit this contextual and generalisable learning is recognised by many
authors [1, 16, 18-22]. It has been argued that because programmes are embedded in a
range of attitudinal, individual, institutional, and societal processes they are not things
that can be claimed to ‘work’ or ‘not work’: instead they contain certain ideas that work
for certain subjects in certain situations [23]. Theory driven approaches to both
evaluation and evidence review that try to understand the links between context,
content, implementation and outcomes are being developed and tested to see if they can
reduce such problems [15, 18].
At the point of embarking on this review in 2010 there was to our knowledge no
systematic theory informed review of the evidence available which identifies the key
mechanisms within many QI programme initiatives - PDSA - that are likely to lead to
practice and health outcome change and improvement.
Review purpose
Given the above limitations in current knowledge and by using elements of a realist
synthesis approach, as espoused by Pawson et al [24], this review attempts to address
the following question:
“What are the implementation and contextual factors that may enhance or
reduce the effectiveness of PDSA as a QI tool in healthcare settings?”
The review was commissioned and funded by NHS Education for Scotland, a special
health board with responsibility for the education, training and life-long learning of the
healthcare workforce in Scotland. Training in QI concepts and methods - including
PDSA – is a learning need that can be identified across the health care workforce. If
such interventions are to be rolled out it is vital to understand how best to implement
them and in what circumstances and contexts they could work. We present the
discussion and learning in relation to the key theories uncovered via the review
processes. This learning then informs our recommendations regarding the training of
health care professionals in PDSA methods, as well as our guidance on the effective
implementation of the technique as part of ongoing improvement initiative.
Methods
Key theories to be explored
An initial background search was conducted examining 37 papers on PDSA published
over the previous year (2009) from study commencement, and browsing references
identified from these papers. This allowed mapping of the diverse perspectives and
theories about PDSA. Inclusion criteria for the review were refined in light of emerging
learning from this process. Informal networking with known experts in the field also
informed this process. The initial resultant theories derived from this initial process are
shown in Table 2 and more refined versions in the findings section. These informed our
data search and extraction processes.
Determining the search strategy
We subsequently conducted a systematic search of papers using the terms in Figure 1
See Figure 1: Formal Search Criteria (additional file 1)
Only English language papers were included from over a five year period from 2005 to
the time of the search in September 2009. These time limits were selected due to
resource limitations and pragmatism. Either the setting or the participants had to be
directly involved in healthcare. Any outcome measure in the papers that led to QI was
included, regardless of whether or not it was clinical.
Study Quality Assessment and data extraction
An initial process was carried out to screen abstracts in order to exclude papers of low
relevance or low worth. Two questions were applied:
1. Do the authors refer to either a PDSA or PDCA process, or IHI collaborative
methodology for use in QI in healthcare?
2. Does the paper go beyond superficial description or commentary, i.e. is it a broadly
competent attempt at research, enquiry, investigation or study?
Papers were rejected at this stage if the answer to either of these questions was ‘no’ and
the rationale for this was documented. Papers for which the abstract did not offer
enough information to determine eligibility were retrieved for full text review. The
same initial questions were asked of all full texts obtained for review, and papers were
subsequently rejected if either answer was ‘no’. Papers that met the criteria of
relevance and worth underwent full data extraction.
A data extraction form shown in Figure 2 was developed.
See Figure 2: Data Extract Form (additional file 2)
This data extraction form was informed by PRISMA (Preferred Reporting Items for
Systematic reviews and Meta-Analyses) guidelines [25], and the data extraction
framework used in a systematic review with a similar approach [26]. The SQUIRE
(Standards for QI Reporting Excellence) guidelines [27] were used to establish a guide
to quality criteria however no formal quality scoring system was employed, in keeping
with similar systematic realist reviews [28]. Excluding a large number of relevant
studies on grounds of rigour would reduce the validity and generalisability of the
learning from realist synthesis review findings [29]. The form went through an iterative
cycle of improvement in a piloting phase involving the three reviewers until a final
format was agreed upon.
The data extraction variables are shown in Table 1
Table 1: Data Extraction variables
Data Extract Variables • Year of Publication • Paradigm (theoretical lens used) • Type of Paper • Perspective (unit of analysis) • Aim & Objectives • Design
o Study design o QI methods used o Method of evaluating outcomes
• Funding • Key actors
o who supporting – how, why o who delivering – roles, training o who receiving o who comparing to
• Nature of the Improvement • Context factors
o Country / region o Urban / rural o Primary / secondary healthcare o Other local / structural factors of influence
• Implementation process o Costs of implementation o Plans for ongoing monitoring
• Data Collection o Timeline
• Data Analysis o Handling of disconfirming observations
• Results o Summary of main results o Facilitators / strengths / successes of implementation o Barriers / weaknesses / difficulties of implementation o Strength of relationship between intervention & changes
• Dissemination Process • Conclusions • Reflexivity (consideration of potential biases & conflicts of interest)
• Ethical implications
The principal summary measures were factors influencing and critical to the
implementation of the PDSA process (see Table 2) and the associated theories in the
findings section below.
A final summary page asked the reviewer the following questions:
1. Does the paper have an important message for our review question?
2. Does the paper fulfil the established quality criteria in its domain?
3. What factors does the paper identify as being critical to the use of PDSA or to the
delivery of education on PDSA?
Papers were randomly allocated to three reviewers (EC, JM, and JF). Twenty percent of
papers were extracted independently by two reviewers (pairing between reviewers
varied to ensure consistency across the group). Reviewers met in pairs after each had
completed extraction of a cycle of 10 papers. Any disagreements on the data in the
papers extracted in duplicate were then resolved by consensus. The meeting also
provided the opportunity for discussion of uncertainties arising with any of the other
papers completed by each reviewer. The approach taken in a realist review makes such
dialogue between researchers essential for the iterative process of classification.
For each article, efforts were made to uncover any sources of bias from either the
authors or the main support team of the initiative. Each article was examined for
evidence of who or what had triggered the decision to initiate the QI project; why it had
occurred at that time; funding sources; and whether the authors had commented on
potential conflicts of interest. A judgement was made on the transparency of data
presentation and the degree to which the reader could analyse results objectively.
Finally we used a snowballing technique to scan the references of the most relevant
articles identified in the systematic search. This technique was used to identify key
papers prior to 2005, and any further papers not identified by the formal search process
that could contribute to populate the synthesis framework.
Following this a final search for additional studies from the grey literature was
conducted to strengthen specific aspects of the evaluative framework (Table 2) and
theories. Additional studies considered included sources such as Government Reports
and national and international PDSA/Collaborative group papers (e.g. National Primary
Care Collaborative [English], Australian Primary Care Collaborative [part of
Improvement Foundation], and Health Foundation Reports).
Evidence/data synthesis
To facilitate the synthesis process an evaluative framework that informed the
development of our initial six theories was developed and populated using the
evidence/learning. This framework identified crucial dimensions of the theories that
related to mechanisms (core elements of the PDSA cycle, or how it was applied) and
contexts (staff motivation, stability, etc). We were seeking data and learning on these
areas to allow us to test and further refine our initially identified theories. The
framework is illustrated in Table 2.
Data and learning were therefore synthesised in order to determine what implementation
and contextual factors (circumstances and constraints) that may enhance or reduce the
effectiveness of PDSA as a QI tool in healthcare settings. ‘Contradictory’ evidence was
used to generate insights about the influence of the PDSA mechanisms or contexts.
To ensure transparency within the subsequent discussion and findings sections we have
referenced the authors which support our assertions. In addition we have used codes
which can be used to identify the actual key finding/learning from within each paper
that has been used to refine our theories and inform our conclusions. The T codes
identify which of our contexts/mechanisms the text relates to (e.g. T1 – T9 shown in
Table 2 and the C code relates to the precise contextual issues (or constraining factors)
identified by the author that has influenced the success of the PDSA intervention
studied/reviewed. The author, paper and constraint or contextual factor relating to each
code can be seen by referring to Figure 5.
Conclusions were drawn by synthesising the data in relation to the evaluative
framework (Table 2) and are presented as a narrative account. Draft recommendations
and conclusions were tested with key stakeholders, and the review team worked
alongside practitioners and policy-makers to apply recommendations in particular
contexts.
Results
Review statistics/ initial theories and synthesis framework.
Recent relevant papers (n=37) from the year preceding the review start date were
identified and reviewed and from this process several draft implementation and
contextual mechanisms were identified which might influence the effectiveness of
PDSA. These are shown in Table 2:
Table 2: Data synthesis framework based on initial draft identified mechanisms and contexts
MECHANISM enhancing outcomes –practice changed/sustained
T1. T2. T3 T4
Measurable process goals that are linked to outcomes via evidence
Changes are more likely to be sustained by ensuring the impact of change (data/progress) is visible
Changes broken down into small-step cycles completed in rapid succession lead to faster sustained change
Local spread to surrounding settings/context can be achieved via key actions
CONTEXT (Constraining and facilitating Factors)
T5 T6 T7 T8 T9
Frontline staff may have relevant knowledge of changes & implementation skills that can influence success
Engagement of frontline staff can influence participation and impact
Existing resources can sustain PDSA/ changes
The tool is can achieve practice change/outcomes in multiple disciplines & organisational contexts
The context/stability/ size of the organisation can impact on PDSA cycles/impact
These mechanism and contexts informed the data extraction process. They were
subsequently integrated and reordered into six key theories about what influences the
effectiveness of PDSA. We have kept the initial codes above however to allow
reference back to the data extraction process. The more collapsed /integrated theories
are:
• Theory One: PDSA is can achieve practice change/outcomes across different
health care disciplines, organisational contexts and settings (see T8/9 in Table
2).
• Theory Two: PDSA interventions are more likely to lead to practice change
when the intended changes are measurable in the short term and when
practitioners can see the changes are evidence based (see T1 & T2 in Table 2)
• Theory Three: PDSA leads to more substantial and sustained practice change
when frontline staff are fully engaged with the process (see T5 & T6 in Table 2)
• Theory Four: Rapid PDSA cycles lead to faster improvement, longer-term
change and sustained momentum for change (see T3 above Table 2 in Table).
• Theory Five: Achieved changes can spread into other settings and contexts (see
T4 above in Table 2).
• Theory Six: Improved practice in PDSA can be achieved and sustained through
existing resources (see T7 in Table 2)
Figure 3 shows the results from the data base search (from 2005 -2009) and the reasons
for exclusion of papers.
See Figure 3 Data base search results and exclusions (additional file 3)
The search identified a total of 762 papers of which 94 were eligible after the
application of our review criteria. These 94 papers were identified as relevant to our
research question and one or more of our subsidiary theories and contained some
worthwhile information. Of these, a significant proportion described ‘before and after’
studies, using baseline data as the comparison group. The quality of these papers varied
significantly, but nevertheless many were able to contribute useful information to refute
or verify our postulated theories and add new knowledge to existing evidence [7]. Of
these 94 papers, 32 were identified as holding relevant information to populate the
evaluation synthesis framework that informed our initial theories (Table 2). The
remaining 62 papers provided information on the general effectiveness of PDSA (e.g.
Theory one only) and detailed information on the settings it was applied within, but did
not provide sufficient detail on any individual PDSA components (e.g. cycle speed) or
contexts (e.g. previous QI culture) to add substantially to our synthesis framework.
The snowballing technique used to identify key papers prior to 2005 involved scanning
the references of the most relevant 32 articles identified in the systematic search that
populated the synthesis framework. This technique identified a further 12 papers.
Therefore, a total of 44 (32 plus 12) papers were drawn on to populate the evaluation
synthesis framework and refine our theories. The discussion and findings in this paper
are based mainly on these 44 papers.
Review Limitations
This review has focussed on how PDSA can be used to best effect in healthcare settings.
However PDSA is often used in conjunction with other components of QI. In particular
it has been used most widely within the ‘collaborative’ approach, which can involve
multiple sites sharing knowledge, learning days, and external support from ‘experts’. It
is therefore difficult to disentangle the dimensions which influence the effectiveness of
PDSA from the dimensions that effect the wider approach utilised.
We have been unable to produce definite statements about ‘what works’; instead, we
have added further detail to our initial theories and produced advice on the likely
implications of different contexts on the success of PDSA. This outcome is a natural
consequence of the type of question being asked, rather than a failing of the scope of the
review or its methodology. The contextual details within individual studies were
limited in many cases. It is unclear whether we have omitted other important contextual
factors that either aren’t identified by study authors, or are considered not influential
enough to report. However by collecting the data available on a large number of papers
we hope we have been able to identify most of the significant elements that influence
the effectiveness of PDSA.
Although we were systematic in our data extraction processes and the application of our
criteria, it is feasible that we missed some relevant articles. This will be particularly
true for articles published prior to 1995, as prior to this date we used a snowballing
process for papers cited in our most relevant articles identified in the systematic search.
Publication bias is a significant concern and one that is difficult to quantify. Successful
organisations have a vested interest in promoting their successes [14]. Once a QI
initiative has been undertaken there is little incentive to devote resources to
investigating its efforts, particularly where private resources have been invested.
Financial interests may be an influence in the popularity of specific QI methods [18].
Mittman (2004) suggests objective evaluations of QI methods are lacking due to
‘demand-led bias’ and points out that much of the published information is located in
management and practitioner-orientated journals such as the ‘The Joint Commission
Journal on Quality in Healthcare’, which tend to emphasise only successful accounts
[16]. Our findings should be considered in light of the likelihood of significant
publication bias. However due to the nature of our question, we would argue that it is
less of a concern than if we had attempted to answer the question ‘does PDSA work?’
Some common criticisms of realist synthesis are that: the iterative approach can
introduce bias; there can be a tendency to treat all forms of evidence as equally
authoritative; and this can lack transparency in the choice of evidence used. Several
steps were taken to minimize these weaknesses. A significant period of time was spent
refining the research question at the scoping stage, giving stability to the question being
asked at the point data was extracted, and therefore reducing risk of bias introduced.
SQUIRE guidelines were used to determine the authoritativeness of different sources of
evidence. In the first phase of data extraction the reviewers examined all eligible
studies within a specified time frame, rather than take a purposive sample, in order to
ensure the transparency of study selection. In taking these steps the realist approach
was modified to suit our specific needs and resources. This is in part a reflection of the
complexity of synthesising the evidence on organisational interventions. If complex
interventions are by nature ‘leaky’ and susceptible to local adaptation, it is perhaps
inevitable that the review methodology applied to them will also need local adaptation.
Discussion
Theory One: PDSA can achieve practice change/outcomes across different
disciplines and organisational contexts/settings.
Figure 4; Illustrates the numbers and proportion of studies used for data extraction that
detailed specific factors of interest to the review question, theories and our discussion.
Figure 4 –Number and proportions of the 94 studies used for data extraction
studies detailing specific factors of interest to the review question (additional file 4)
Application of PDSA across settings and organisations and patient groups
PDSA is a mechanism for QI that has been used to change practice in a wide variety of
disciplines and organisational contexts. The examination of 94 papers included found
that in most cases the context was sufficiently well described that the findings could be
related to other settings (Figure 4 Table 26). 70% of papers were based in either the
USA or Canada, and a further 12% were UK based (see Figure 4 Table 27). Both urban
and rural contexts were described (see Figure 4 Table 28). The projects were based in
secondary care in 63% of cases, and primary care in 18% of cases (see Figure 4 Table
29). Of the 29 hospitals that gave additional descriptive information 12 were described
as a tertiary centre, and a further 12 were described as academic or teaching hospitals.
The patient group receiving the improvement were sometimes defined by clinical
diagnosis (e.g. diabetes, asthma, depression, stroke), some by a procedure or risk factor
(e.g. requiring ventilation, at risk of falling), and some by location of setting (e.g. ITU,
radiology, A&E, haemodialysis, oncology ward). Overall there was a very wide
spectrum of clinical problems and scenarios in which PDSA was used as a tool for
healthcare improvement in some capacity (see notes at the end of Figure 4).
Figure 5 details the references and the associated codes for the information contained in
the subsequent discussion. These codes can be used to identify the key findings from
the review used in our evaluation synthesis (Table 2) to support our discussion and
findings. All of the codes in the subsequent discussion therefore relate back to Figure 5
See Figure 5: – Codes for evidence used in the evaluation synthesis (additional file
5)
Organisational culture, characteristics and stability
The evaluation of the UK patient safety project found no specific organisational
characteristics associated with greater perceived value of PDSA [30, T8/C17], and no
association was found between site characteristics and the achievement of sustaining
and spreading change in a US mental health project [31].
Whilst there is no clear characterisation of the organisation that predicts effective use of
PDSA there are some organisational contexts and cultures in which PDSA appears to be
more or less effective.
A small community hospital A&E department found using PDSA cycles to be a vey
useful tool for achieving improvements [32, T8/C5]. Projects that involve changes in a
well defined geographical area such as an ICU or a single clinic [33, T2/C3] appear
ideal. This may relate at least in part to the visibility of practice change as staff
feedback can be centralised and focused.
Although larger settings may be more likely to contain internal QI expertise, it can be
harder to sustain and spread changes in these contexts possibly due to difficulty
obtaining a ‘critical mass’ that makes changes routine. The impact of any small changes
in larger settings can become diluted, and the changes less visible [31, T8/C3].
A lack of on-site QI expertise may lead to teams struggling [34, T8/C4]. Contexts
which contain QI expertise, most likely larger settings, can be supportive of change
cycles, especially if PDSA is integrated into existing QI programmes. Leadership and
management may also be more supportive where QI is routine. However as identified
above larger settings can have drawbacks.
Organisations with a ‘developmental’ rather than hierarchical culture were associated
with greater staff motivation to conduct PDSA cycles [35, T8/C6]. In contexts where
staff are less familiar with discussing care processes the tool may be less applicable.
A strong culture of clinician autonomy can also be a barrier [30, T8/C13]. A Dutch
mental health project using PDSA found the model did not fit in to the existing culture
of primary care professionals, who found it difficult to apply [36, T8/C7]. A lack of
staff with a prior interest in QI was found to be an important factor in an initiative with
clinical trainees where the PDSA model did not translate into rapid cycles of change
[37, T8/C8].
Organisations able to combine the use of PDSA with other features of QI projects where
information and reporting is shared between other organisations may find it more
effective [38, T8/C16].
A primary care diabetes QI project also found that some changes were implemented
simply as a result of discussion and agreement. There was no requirement in this small
setting to test the changes and roll them out gradually [39, T3/C18]. In these situations
PDSA as a QI tool is made redundant.
Staff, disciplines and management support
When several disciplines were involved poor communication between groups was
identified as a barrier [40, T8/C14]. Offering a ‘virtual’ environment to facilitate
communication does not appear to have been helpful if the staff were used to such
technology [36, T8/C15].
Leadership distraction with other priorities was a significant barrier [41, T8/C2]. Poor
leadership support had a significant impact on the effectiveness of a US mental health
QI project [31, T8/C11]. Senior support is often assessed as crucial [42, T8/C1], with
the possible exception of units which are self-contained or operate fairly independently
such as a neonatal ICU [43, T8/C12].
Organisational and staff stability
As PDSA cycles start small and gradually expand they may be more resilient to
instability and more suitable for changing situations than other QI tools. However
large-scale organisational changes will adversely affect staff motivation to implement
and sustain cycles [34, T9/C1; 42, T9/C2].
Staff turnover was a key barrier to implementation of cycles [44, T9/C3; 45, T9/C4],
particularly if there was reliance on a small number of ‘champions’ [34, T9/C5; 46,
T9/C6]. Staff turnover and retention were often a barrier in contexts such as teaching
hospital [47, T9/C7] and a US mental health collaborative [31, T9/C9] and settings with
a high proportion of part-time staff [34, T9/C8].
Goal definition
PDSA is more effective if goals have clear operational definitions [48, T1/C13]. This
distinction can be described as the difference between control and learning QI projects
[4, T1/C11]. Goals that are applicable in all situations (rather than discretionary)
appeared to be more successful in terms of achieving and sustaining changes when
using PDSA [4, T1/C11]. Such goals are more common and appropriate in settings with
high control, for instance ICU [49, T1/C12].
Stability of patient population
The impact of changes were harder to measure (and so visibility of success reduced)
when the patient population was very transient [44, T2/C9], or when an acute clinical
problem left little time for staff to document an implemented change [46, T2/C10; 40,
T2/C11]. Measuring and documenting the impact of changes may be easier in contexts
where patient populations are more stable, less transient and where patient turnover is
generally lower.
Staff skills, capacity and training
Implementation of the PDSA cycle is difficult without some degree of process-thinking
skill [34, T5/C2]. Some form of training was given to staff in 68% of papers used for
data extraction (see Figure 4, Table 21), and in one third the training covered both the
intervention itself and some type of input on the use of PDSA. In a further third the
training focussed only on the intervention (see Figure 4, Table 22). The concept of
identifying appropriate measures for PDSA can be difficult to grasp [50, T5/C13; 41,
T5/C14]. Skills were also needed to collect and present performance data and these
could also be difficult to learn [50, T5C15]. Training on basic concepts of the process
may need to be consolidated once put into practice [51, T5/C16; 41, T5/C17].
A UK Accident & Emergency-based project found the medical consultants grasped the
concept more easily than senior managers [52, T6/C13]. Ideas regarding
implementation strategies and appropriate measures did not always reside within staff
knowledge bases [53, T5/C5; 50, [T5/C6], although frontline staff took ownership of
sourcing implementation ideas within an ICU based project [49, T5/C4].
External input regarding knowledge of scientific literature and clinical organisational
features from a support team were sometimes required [54, T5/C7]. External support
may also be required for goal setting, for making recommendations about which tools
might be used, and for supplying template tools to implement [31, T5/C10; 43, T5/C11;
50, T5/C9; 55, T5/C12]. Top-down instruction was favoured in a primary care mental
health collaborative [48, T5/C1]), but reduced frontline ownership. Use of external
expertise was cited as key in helping motivate staff in a US secondary care project over
58 sites [42, T6/C24], and staff learning sessions were rated as essential to staff
motivation in a UK patient safety collaborative [30, T6/C25].
Financial incentives for staff could help overcome initial resistance, and were critical to
frontline staff engagement in a Canadian primary care collaborative project [56,
T6/C26].
A lack of dedicated time to conduct a QI project using PDSA can hinder progress [36,
T7/C2]. If staff time is already saturated building in additional processes to implement
change can lead to staff having to work in their own time [34, T7/C3]. A project based
on an in-patient cardiology ward found that the equivalent of a 1.5 full-time nurse’s post
was required for the time spent on daily monitoring, reviewing results, and giving staff
feedback [47, T7/C4].
The additional staff time required was found to be a limiting factor in several projects
[51, T7/C7; 57, T7/C5; 58T7/C6;]. A project based in nursing homes found that other
demands on staff time led to difficulty in persuading staff of the importance of the QI
project [59, T7/C8]. However as the extra work associated with a change became part
of ‘normal routine’ one project observed that workload was reduced as staff worked
more efficiently as a consequence of the improvements [51, T7/C7]. Staff turnover is a
key barrier, particularly if there has been an over–reliance on local ‘champions’. As
such, the engagement of a range of committed staff may help sustain and spread change
to other suitable settings.
Theory Two: PDSA interventions are more likely to lead to practice change when
the changes are measurable, visible and evidence based/plausible.
Theory two postulates that PDSA interventions are more likely to lead to practice
change amongst staff when the changes anticipated (practice changes or changes in
patient outcomes) are measurable in the short term (e.g. within each/early PDSA cycles)
and when practitioners can visibly see that the anticipated changes are taking place, are
based on evidence and plausibly linked to longer term changes in patient/health care
outcomes.
Measuring short term practice outcomes or longer-term health care outcomes
QI projects using PDSA cycles often have an overall aim with a patient or clinical care
focus. However initiatives are often analysed in terms of processes-of-care measures
rather than clinical outcomes. One advantage of process-based measures is that it is
much more likely that an improvement can be demonstrated within a short-time frame:
goals can be defined tightly and specific targets set. A major disadvantage of focussing
on processes-of-care is that the link with clinical benefit can often be unclear, which has
a significant impact on staff motivation. Additionally, although the evidence base for
the overall clinical aim may be robust, the evidence relating the achievement of the
practice change measure to this can be lacking.
The review learning suggests that the most successful projects are those in which the
relationship between process goals and clinical benefit was evident, and in which
clinical benefit for patients could be observed by staff. Translation of process
improvements into clinical improvements may be difficult in the short timeframe of a
specific project however [36, T1/C15; 50, T1/C14], and a cause and effect relationship
between the change made and clinical benefit can be difficult to establish [30, T1/C17;
48, T1/C17].
Both a paediatric asthma QI programme and a project to reduce hospital drug errors
found that outcomes relating to processes of care were more successful than patient
level outcomes [50, T1/C4; 60, T1/C3). Identifying and implementing appropriate
measures could be a difficult task however [30, T1/C5].
It can also be unclear as to what constitutes ‘enough’ evidence when clinical aims are
being set. In a critical care QI project evidence emerged suggesting they might actually
be causing harm instead of benefit [46, T1/C7]. In reality clinical goals were chosen on
the basis of both evidence base and stakeholder priorities [50, T2/C8]. When the risk of
harm was low, for instance improving patient knowledge or reducing patient waiting
times, an evidence base was less important as these non-clinical aims were deemed
beneficial in themselves [38, T1/C9; 61, T1/C10.
Providing feedback on goals and keeping progress visible
Ensuring visibility of data relating to the goals set was also important [48, T2/C4]. A
smaller setting/ sub unit also enables the display of data in a central location that is
visible to all relevant staff, for example a data wall displaying progress towards the
goals set [49, T4/C5].
Visually displaying baseline measures can demonstrate the need for change to staff and
build motivation [48, T6/C1], as can clear presentation of the evidence-base for the
improvement goals [40, T6/C2].
Initial staff resistance can be overcome by ensuring positive changes are highly visible
[62, T6/C30]. Celebrating successes even when small and creating a sense of
competition between departments were found to be effective [46, T6/C34]. However
high visibility acted as a discouragement to staff where little progress was made towards
the goals despite staff effort [32, T6/C32]. A neonatal pain management project
highlighted the need for perseverance even when early results may have been perceived
as a failure, as follow-up data demonstrated success [40, T6/C33]. The difficultly with
interpreting early results has been viewed as ‘demoralising and confusing’ in other
projects [53, T6/C31].
One project using PDSA found the focus on individual patients was very beneficial for
gaining staff enthusiasm as the benefits were directly seen, however without process-
thinking skills the motivation gained did not translate into action with a wider group of
patients [34, T6/C27]. Focussing on small scale changes can lower expectation of the
eventual clinical benefit, and staff may be less motivated to be involved [61]. Other
projects found small changes fuelled enthusiasm that led on to motivation to see larger
changes [50, T6/C28].
Sustaining the changes
Regular feedback is associated with greater ability to sustain change [31, T2/C7], and in
some studies there were periods where the frequency needed to be particularly high [57,
T2/C8]. Even when changes had become part of accepted routine care, ongoing
monitoring and display of measures were necessary to sustain the change [57, T2/C12;
48, T2/C13].
Direct comparison between patients that were included in a PDSA cycle and those that
had not was helpful for sustaining momentum, especially when the link with clinical
benefit was clear, such as a care bundle for ventilated patients [48, T2/C1]).
Improvements are of most value when they are sustainable, and when secondary spread
to other relevant areas is achieved. Sustainability appears to be aided by ensuring
feedback to staff is given regularly, and progress towards goals is kept visible. This is
much more likely to be achieved in situations when the improvements are being carried
out within a defined team in a defined setting.
Theory Three: PDSA leads to greater and more sustained practice change when
frontline staff take ownership of and are engaged with the process
Frontline clinical staff were responsible for delivering the improvement process in 78%
of papers used for data extraction (see Figure 4, Table 23). Active engagement of
frontline clinical staff seemed to be important for the success of projects [63, T6/C6],
and participation itself may have been perceived to create a sense of ownership amongst
staff [50, T6/C9].
Achieving engagement and buy in
Although staff ‘buy-in’ to participating in QI in their area of work is crucial to its
success it is not always easy to achieve. It is unlikely that this difficulty is particular to
PDSA cycles. The small-step cycles concept suggests that it achieves a greater level of
staff empowerment than more radical ‘top-down’ changes. However the reality appears
to be that this is often not the case.
Staff engagement and motivation can be essential in ‘championing’ the spread of
changes to other sites. Capturing the interests of different staff groups may require a
variety of techniques. For example, busy leadership personnel will need to be updated
using a time-effective method, such that their attention is not lost. However, clinicians
may be more engaged if outcomes of clinical interest and relevance are emphasised.
Front line staff have been found to be more engaged when they have a sense of
ownership over the change outcome decisions.
Factors associated with staff resistance
Staff can also demonstrate resistance due to concerns regarding loss of clinical
judgement and control, particularly when changes were protocol and target driven [64,
T6/C4]. Resistance from clinical staff in particular was an issue [50, T6/C13; 31,
T6/C14]. An evaluation of a UK based orthopaedic collaborative using PDSA cycles
found ‘clinicians did not appear to wish to lead’ [53, T6/C10]. The evaluation of the
orthopaedic project found that of 17 components of the collaborative initiative, PDSA
was rated the lowest, with many unconvinced of its value for driving change [53].
However the PDSA process was rated as highly important by both clinical staff and
board-level staff in a UK based patient safety project [30]. A different picture emerged
from a UK Accident & Emergency project in which staff motivation and empowerment
amongst frontline staff was high [52, T6/C5].
Motivation can be difficult to achieve, particularly if more than one hospital department
or team is involved [61, T6/C17; 37, [T6/C18]. Process measures were often judged on
documentation and if there was a perception these tasks were already being done (but
not documented) motivation was poor as there were low expectations of clinical
improvement [57, T6/C4].
Allowing the staff team to decide on change ideas was effective in building ownership,
but was also inefficient as lessons learned elsewhere are re-learned [41, T6/C8]. A
further consequence of a strong sense of ownership was slowing down of the
improvement process as ‘everyone has their say’ [58, T6/C7]. It seems that there needs
to be a balance between frontline ownership and stalling progress as everyone has their
say [58, T3/C22]. Although staff motivation or ‘buy-in’ was critical to achieving
improvement goals, it varies widely between contexts and was not easily predicted [54,
T6/C15].
Participation on a voluntary basis was found to increase motivation [65, T6/C16].
Ensuring a broad range of staff were on the improvement team was important if the
initiative involved multiple sites and disciplines, for example using PDSA cycles to
improve patient transportation [66, T6/C19]. Focussing on ‘early adopters’ or those
staff most favourable disposed to PDSA at the start promoted the involvement of other
staff [48, T6/C20].
Supervisory and management support
Where supervisor support is perceived by staff it facilitates empowerment [35, T6/C21],
but as empowerment is associated with increased scrutiny and responsibility if things go
wrong, staff can be ambivalent about taking greater ownership, and managers may be
reluctant to release staff if they feel it could threaten the security of their role [4,
T6/C22]. The UK A&E project met resistance from the senior management who
appeared threatened by the empowerment of frontline staff.
Theory Four: Rapid PDSA cycles lead to faster improvements and sustained
momentum for change
There are two opposing philosophies in QI, that of ‘gradualism’ or ‘incrementalism’ (to
which PDSA belongs), and that of radical transformational change. Bate et al (2002)
suggest ‘much hangs on which is correct’[53]. Plsek (1999) suggests that ‘pace is
crucial’ in the utilisation of PDSA cycles [67], however the reality captured in our
review seems to be that momentum can be difficult to both achieve and sustain.
The likely speed of PDSA cycles and of change
There was reasonable evidence of the relationship between changes observed and use of
PDSA cycles in 46% of papers used for data extraction (see Figure 4, Table 19). There
were examples of projects having been able to conduct cycles rapidly, for instance a US
drug error QI project conducted 739 cycles over 15 months, of which 63% were
described as ‘real tests of change’, the remainder being ‘educative’ [50, T3/C2].
There were, however, numerous examples in the literature of single PDSA cycles taking
place over a number of weeks or months [37, T3/C6; 53, T3/C7; 68, T3/C8; 69, T3/C8;
45, T3/C9; 57, T3/C10; 70, T3/C11; 71, T3/C13]. Along with other enthusiasts of the
PDSA method Kilo (1998) states ‘dramatic, rapid changes in outcomes can (and do)
occur within months’ [72]. Our review suggests that this expectation is not often met,
and it is often more realistic to prepare for slower and less radical change. This
somewhat contradicts the ‘rapid-cycle’ terminology used to describe PDSA.
Reasons for the relatively slow change
There seems to be a variety of reasons for slower cycle speeds such as difficulties
translating the goals of a project into small steps that can be tested within cycles
conducted in rapid succession. Concerns exist that as cycles involve increasing
numbers of patients, inadequate time for learning may be given if cycles are conducted
too rapidly, especially when new skills amongst staff need to be developed [33,
T3/C14]. Process changes that involve a small number of patients (for instance a
clinical problem which presents less than once a week) means that changes are harder to
break down into small steps and cycles are slower [61, T3/C19]. Spending too long
collecting baseline data was thought to be a common cause of loss of momentum in a
US drug error initiative [50, T3/C20]. Projects that were required to generate ideas
internally were found to conduct cycles more slowly [43, T3/C1]. In addition
inclusivity and engagement of a wide range of staff all slow cycles down.
Some changes necessitated large-scale disruption to supporting services, even when
applied to small number of patients. In these cases the disruption to the system was
greater if tested in small steps than if changes were made in one large cycle 61,
T3/C15]. A UK orthopaedic QI initiative found that specifying changes must be
achieved in small steps limited the changes that were achievable [53, T5/C16]. There
were also some changes which were implemented immediately on a wide scale with
little difficulty, for instance making a decision to remove bags of potassium chloride
from ward supplies [50, T3/C17]. A primary care diabetes QI project also found that
for some changes it was sufficient to discuss and agree on the changes that were needed
in order to see them implemented.
There seems also to be a relationship between this and Theory Two concerning the
importance of the visibility of changes and cycle speed. Visibility of the changes was
co-dependent on the speed & size of the cycles – a large change would go through the
cycle at a lower speed, reducing the visibility of the change due to too little timely
feedback [61, T2/C6]. Given this slower cycle speeds may impact on both effectiveness
and sustainability of resultant changes.
Critics of the incremental approach suggest that energy and enthusiasm is worn away
over time as everyday problems continue to present themselves, without the time,
support or capability to overcome them. Our learning on cycle speed and resultant
change may add weight to this view.
Theory Five: Changes can be spread into other settings and contexts
Descriptions of spread of the improvements within the local context (or plans to do so in
the future) were given in only 41% of studies used for data extraction (see Figure 4,
Table 20). There is little evidence of diffusion of improvements locally without
intentional action or the setting intentional targets [36, T4/C8; 31, T4/C9; 34, T4/C7].
Barriers to sharing good practice
Implementation of ideas to achieve goals were often context specific, so spread could be
difficult where systems differed in the new setting, even if the overall aim was
unchanged [61, T7/C10]. Differing IT resources for example acted as a barrier to
exchange between settings [31, T4/C11].
Even within settings some studies report the need to start again with small cycles in a
new setting to encourage the spread of good practice even if only small changes were
made to implementation [48, T4/C3]. For example a chronic pain QI project based in
nursing homes found that there was no uptake of changes in other homes [59, T4/C4].
There was also difficulty when staff involved in the first site moved on before spread
was achieved [41, T4/C6].
What encourages the spread of good practice?
Senior management and administrative support is likely to be required if spread is to
occur in terms of both facilitation and finances, but the evidence of benefit provided by
PDSA cycles may not be deemed robust enough to justify this [46, T4/C1; 41, T4/C2].
It seems valuable to use staff from the initial context to help start momentum in a new
setting [48, T4/C5]. Presentation of results to wider audience, for example at hospital
teaching meetings, stimulated interest [37, T4/C12] although this was less of an issue on
smaller sites [31, T4/C13].
A psychiatric in-patient QI project recognised that widespread change was unlikely
without achieving systematic change regardless of the use of PDSA cycles [65,
T4/C14]. An evaluation of a UK orthopaedic collaborative also found that more radical
system design was required (alongside small step changes), to achieve broader
achievement of QI [53, T4/C15]. With regard to this theory the literature seems to
suggest that spread can often be slow and needs active input.
Theory Six: Improved practice in PDSA can be achieved and sustained as aresult
of existing (rather than new) resources
Few of the papers used for data extraction gave information on their costs or resource
requirements. Costs were stated clearly in only 11% of papers, and only further 11%
gave any information at all (see Figure 4, Table 23). The oversight or support for
projects was described in 80% of papers, and often this came in part from an external
team, although the actual nature of the relationship was often unclear (see Figure 4,
Table 24).
Are existing resource sufficient for successful implementation of PDSA?
The concept that PDSA cycles make use of existing resources and can therefore be
conducted without additional investment does not often appear to hold true. Financial
investment was often needed for staff training, staff time, implementation costs, and
monitoring costs. This was sometimes considerable [73, T7/C14; 31, T7/C15; 50,
T7/C16].
A local co-ordinator may be employed to support teams, for which financial resources
may need to come from external sources [36, T7/C17]. Training costs can be reduced
by telephone or ‘virtual’ provision, but may not be favoured by staff compared to face-
to-face training [41, T7/C18]. Changes in a large highly systematised organisation may
be more costly to achieve than in smaller independent practices [54, T7/C19]. In
centrally coordinated projects the resource demands can be too prohibitive to enable
participation by all [50, T7/C20].
Cost related to monitoring process and outcome data
If monitoring involved data collection not already gathered routinely on existing IT
systems it consumed a significant amount of staff time [61, T7/C9]). If a specific time
limited team coordinated this, the monitoring may have been subsequently been
integrated into normal staff roles [54, T2/C14]. Plans for ongoing monitoring and
maintenance of improvements were stated clearly in only 13% of the papers used for
data extraction; a further 28% contained some information, and 55% had no information
(see Figure 4, Table 18).
Information technology infrastructure was critical to successful integration of changes
into routine care [74, T7/C10], but was sometimes newly created and supported [44,
T7/C11; 30, T7/C12] and could be expensive [54, T7/C13]. In view of the financial,
staff, IT, and training resources that may be required it is important that an assessment
of existing capacity in each of these areas is conducted before the initiation of a project
[30, T7/C1].
Findings and recommendations
This review has identified a number of factors that seem to enable PDSA to be used
more or less effectively. We summarise this learning below.
PDSA has been used successfully across a variety of: geographies; settings;
organisations; clinical populations; and, patient groups. There seem to be both
advantages and disadvantages relating to the size of the organisation. The culture of an
organisation is perhaps more important than its size.
The effectiveness of PDSA in a particular setting cannot be fully predicted as it will be
impacted on by a variety of implementation and contextual factors. It appears however
that it may stand a greater chance of success in smaller and well defined settings. A
‘developmental’ rather than ‘hierarchical’ organisational/team culture is associated with
a greater motivation to conduct PDSA cycles.
Success is also likely to be greater in settings where staff and /or patient groups are
stable and less transient where more time is available and it is easier to document and
measure change. Figure 7 shows a template developed by us in response to the learning
above that may aid organisations decide whether they have setting/context in which
PDSA would work well.
See Figure 6 – Template: Assessing the applicability of using PDSA to achieve
quality improvement in your context & objectives (additional file 6)
PDSA projects are more likely to improve practice when process goals are linked via
evidence/logic to the intended outcomes (changes in practice and patient health) and
where these links are evident to, and observable by, staff.
Similarly effectiveness will be greater where the practice and clinical outcomes sought
are clearly identified and measurable within reasonably short time frames.
Focussing on small scale change may lower expectation of eventual clinical benefits but
significant evidence suggests that small changes can fuel enthusiasm and in turn, a
motivation to see greater changes.
Framing PDSA as ‘a scientific method’, ‘real-time science’ or the ‘science of change’
[66, 72, 75]. rather than as an improvement tool may credit it with too much ‘scientific
rigour’, establish unrealistic expectations, and lead to disappointment and diminishing
commitment from staff. A better (heuristic) approach might be to describe PDSA as a
model for ‘daily work application’ [76] or to build it into existing knowledge and skill
in audit methodology, rather than as a new approach. PDSA cycles should be integrated
into current QI programmes of work where they exist.
Approaching QI by small-scale changes does not appear to be possible, appropriate or
necessary in all situations and in some cases a different improvement tool may be
needed. PDSA projects are more likely to be effective if staff are actively engaged in
the process (identification and measurement of the changes) and if they have prior
exposure to QI processes or support from others who have QI skills. A broad range of
staff should be on the improvement team if the intervention is seeking change in
multiple sites or disciplines. Early adopters can be used to motivate engagement
amongst other staff. Training in process thinking skills is likely to enhance
understanding and participation.
Whilst improved practice is likely to happen faster and be sustained longer when PDSA
cycles are quicker and shorter such speed is difficult to achieve especially in contexts
with high or increasing patient numbers or addressing rare conditions. These contexts
provide more limited time for implementation and measurement. Ensuring staff
engagement and participation can also slow the change process.
It seems to be harder to sustain and spread good practice within larger settings however
such settings may have a stronger QI culture, greater QI expertise and leadership and
management support available which are likely to enhance outcomes.
Enhanced practice is sustained and likely to be spread more widely when evidence of
progress/feedback is highly visible to staff, frequent and provided in central locations.
This is particularly important within longer PDSA cycles
In order to implement PDSA cycles effectively staff required knowledge of the
evidence base, knowledge of appropriate measures to achieve a particular aim, and
knowledge of their organisational dynamics. Technical skills relating to the monitoring
progress are also required. External input to train and support frontline staff in these
aspects is often required. Resourcing capability and support internally is also important
for reducing potential organisational structural barriers to change and spreading change.
Before embarking on a PDSA project it might be useful to conduct a needs assessment
of capacity and capability and (where necessary) follow this up with a realistic business
plan. Depending on the scale and aims of the project, organisations participating in
PDSA QI initiatives need to provide sufficient additional finances, staff time, and IT
infrastructure. Careful thought as to how current IT capability in healthcare
organisations may change the dynamics of the PDSA
Achieving spread of improvements to other areas once they have been tested and
implemented in one area involves intentional and planned input.
Conclusions
This review and synthesis of the literature reporting evidence of PDSA implementation
in healthcare QI initiatives has made a small contribution to advancing our knowledge
and understanding of how and why application of the tool by clinicians and others may
or may not lead to successful improvements in patient care, particularly with regard to
the influential role of the contextual and socio-cultural factors at play.
List of abbreviations
A & E Accident and Emergency
IHI Institute of Healthcare Improvement
NHS National Health Service
PDCA Plan-Do-Check-Act
PDSA Plan-Do-Study-Act
PRISMA Preferred Reporting Items for Systematic reviews and Meta-Analyses
QI Quality Improvement
RCT Randomised Controlled Study
SQUIRE Standards for QI Reporting Excellence
Competing interests The authors state that there are no competing interests..
Authors' contributions
EC co-led the study design, conducted the literature searches, coordinated data
collection, analysis, synthesis and interpretation, and wrote the first draft. AB assisted
with study design, theory refinement, data synthesis and interpretation and contributed
to the writing of the manuscript. JF, HH and JM assisted with literature searches, data
analysis, synthesis and interpretation and contributed to the writing of the manuscript.
PB conceived the study idea, acquired funding, co-led the study design, assisted with
theory refinement, data synthesis and interpretation and contributed to the writing of the
manuscript.
Acknowledgements
We thank Dr Alison Powell, University of St Andrews, Dr Michelle Beattie, University
of Stirling, and Dr Avril Blamey, Planning & Evaluation Consultant (Avril Blamey and
Associates), for critically reviewing and commenting on earlier drafts of this report. We
are also grateful to Drs Claire Tochel and Helen Allbutt, NHS Education for Scotland,
for providing advice on systematic literature review methods.
The study was funded by the Patient Safety Multi-Professional Steering Group of NHS
Education for Scotland, a special health authority responsible for the education &
training and life-long development of the healthcare workforce in Scotland.
References
1. Speroff T, O’Connor GT. Study designs for PDSA quality improvement research.
Quality Management in Healthcare, 2004;13:17-32.
2. Deming, W. Edwards. Out of the Crisis. MIT Press (1986).
3. Shewhart, Walter A. Statistical Method from the Viewpoint of Quality Control.
(1939) Dover Publications December 1, 1986. ISBN 0-486-65232-7. OCLC 13822053.
4. Bloor G. Organisational culture, organisational learning and total quality
management: A literature review and synthesis. Australian Health Review,
1999;22:162-179.
5. Langley GJ, Moen RD, Nolan KM, Nolan TW, Norman CL, Provost LP. The
Improvement Guide 2nd Ed. Josey-Bass (2009).
6. Institute for Healthcare Improvement. The Breakthrough Series. Boston (2003).
(www.IHI.org Accessed 10th July 2012).
7. Boaden R, Harvey G, Moxham C, and Proudlove N. Quality improvement: theory
and practice in healthcare. NHS Institute for Innovation and Improvement,
University of Warwick, Coventry UK (2008).
8. Scottish Patient Safety Programme. Welcome to the Scottish Patient Safety
Programme. http://www.scottishpatientsafetyprogramme.scot.nhs.uk/programme
[Accessed 10th July 2012]
9. Safer Patients Initiative. Learning Report: Safer Patients Initiatives. The Health
Foundation, February 20011. http://www.health.org.uk/publications/safer-patients-
initiative/ Accessed 10th July 2012
10. Grol R, Berwick D, Wensing M. On the trail of quality and safety in health
care. BMJ, 2008;336:74-76.
11. Landon BE, Wilson IB, McInnes K, Landrum MB, Hirschhorn L, Marsden PV,
Gustafson D, Cleary PD. Effects of quality improvement collaborative on the
outcomes of care of patients with HIV infection: the EQHIV study. Annals of
Internal Medicine, 2004;140:887-896.
12. Schouten LMT, Hulscher MELJ, van Everdingen JJE, Huijsman R, Grol RPTM.
Evidence for the impact of quality improvement collaboratives: a systematic
review. BMJ, 2008;336:1491-4.
13. Boonyasai RT, Windish DM, Chakroborti C, Feldman LS, Rubin HR, Bass EB.
Effectiveness of teaching Quality improvement to clinicians – A systematic review.
Journal of the American Medical Association, 2007;298: 1023-1036.
14. Auerbach AD, Landefeld CS and Shojania KG. The tension between needing to
improve care and knowing how to do it. New England Journal of Medicine.
2007;357: 608-613.
15. Walshe K. Understanding what works - and why – in quality improvement: the
need for theory driven evaluation. International Journal for Quality in Health Care,
2007;19:57-59.
16. Mittman BS. Creating the evidence base for quality improvement
collaboratives. Annals of Internal Medicine, 2004;140:897-900.
17. Ovretveit J, Bate P, Cleary P, Cretin S, Gustafson D, McInnes K, McLeod H,
Molfenter T, Plsek P, Robert G, Shortell S, Wilson T. Quality collaboratives: Lessons
from research. Quality and Safety in Healthcare, 2002;11:345-351.
18. Walshe K. Pseudoinnovation: the development and spread of healthcare
quality improvement methodologies. International Journal for Quality in Health
Care, 2009; 21:153-159.
19. Lindenauer PK. Effects of quality improvement collaboratives. BMJ,
2008;336:1448-1449.
20. Soper B, Buxton M, Hanney S, Oortwijn W, Scoggins A, Steel N, Ling T.
Developing the protocol for the evaluation of the health foundation’s ‘engaging
with quality initiative’ - an emergent approach. Implementation Science, 2008;3:46.
21. Lavis JN, Davies HTO, Gruen RL, Walse K, Farquhar C. Working within and
beyond the Cochrane collaboration to make systematic reviews more useful to
healthcare managers and policy makers. Healthcare policy, 2006;1:21-33.
22. Pawson R.(2002). Evidence-based policy: the promise of ‘realist synthesis’.
Evaluation, 8:340-358.
23. Pawson R, Tilley N. (1997). Realistic Evaluation. Sage Publications.
24. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist synthesis: an
introduction. ESRC research methods programme working paper series. ESRC:
August 2004.
25. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche P, et al. and the PRISMA
Group. The PRISMA statement for reporting systematic reviews and meta-
analyses of studies that evaluate health care interventions: Explanation and
elaboration. PLoS Med 6: e1000100. 2009; doi:10.1371/journal.pmed.1000100
26. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O, Peacock R.
Storylines of research in diffusion of innovation: a meta-narrative approach to
systematic review, Social Science & Medicine, 2005;61:417-430.
27. Davidoff F, Bataldan P, Stevens D, Ogrinc G and Mooney SE. Publication
guidelines for quality improvement studies in health care: evolution of the
SQUIRE project. BMJ, 2009;338:a3152.
28. Greenhalgh T, Potts HWW, Wong G, Bark P, Swinglehurst D. Tensions and
paradoxes in electronic patient record research: a systematic literature review
using the meta-narrative method. The Millbank Quarterly, 2009;87:4.
29. Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review – a new method of
systematic review designed for complex policy interventions. Journal of Health
Services Research 2005’10:S1 21-34.
30. Benn J, Burnett S, Parand A, Pinto A, Iskander S and Vincent C. Perceptions of the
impact of a large-scale collaborative improvement programme: experience in the
UK safer patients initiative. Journal of Evaluation in Clinical Practice, 2009;5:524-
540.
31. Meredith LS, Mendel P, Pearson M, Wu S, Joyce G, Straus JB, Ryan G, Keeler E,
Unutzer J. Implementation and maintenance of quality improvement for treating
depression in primary care. Psychiatric Services, 2006;57:48-55.
32. Warburton RN. Preliminary outcomes and cost-benefit analysis of a community
hospital emergency department screening and referral program for patients aged
75 or more. International Journal of Health Care, 2005;18:474-484.
33. Varkey P, Sathananthan A, Scheifer A, Bhagra S, Fujiyoshi A, Tom A, Murad MH.
Using quality-improvement techniques to enhance patient education and
counselling of diagnosis and management. Quality in Primary Care, 2009;17:205-13.
34. Macintosh-Murray A. Challenges of collaborative improvement in continuing
care. Healthcare Quarterly, 2007;10:49-57.
35. Lin MK, Marsteller JA, Shortell SM, Mendel P, Pearson M, Rosen M, Wu S.
Motivation to change chronic illness care: results from a national evaluation of
quality improvement collaboratives. Health Care Management Review, 2005;30:139-
156.
36. Franx G, Meeuwissen JA, Sinnema H, Spijker J, Huyser J, Wensing M, de Lange J.
Quality improvement in depression care in the Netherlands: the depression
breakthrough collaborative. International Journal of Integrated Care, 2009;9:1-11.
37. Tomolo AM, Lawrence RH, Aron DC. A case study of translating ACGME
practice-based learning and improvement requirements into reality: systems
quality improvement projects as the key component to a comprehensive
curriculum. Quality & Safety in Health Care, 2009;18:217-224.
38. Nembhard IM. Learning and improving in quality improvement collaboratives:
which collaborative features do participants value most? Health Services Research,
2009;44:359-378.
39. Geboers H, Grol R, van den Bosch W, van den Hoogen H, Mokkink H, van
Montfort P, Oltheten H. A model for continuous quality improvement in small scale
practices. Quality in Health Care, 1999;8:43-48.
40. Dubar III AA, Sharek PJ, Mickas NA, Coker, KL, Duncan J, McLendon D,
Pagain C, Puthoff TD, et al. Implementation and case study results of potentially
better pratcies to improve pain management of neonates. Pediatrics. 2006;118:
Supplement 2. 587-595
41. Gould DA, Lynn J, Halper D, Myers SK, Simon L. The New York city palliative
care quality improvement collaborative. The Joint Commission Journal on Quality an
Patient Safety, 2007;33: 307-316.
42. Leape LL, Rogers G, Hanna D, Griswold P, Federico F, Fenn CA, Bates DW, Kirle
L, Clarridge BR. Developing and implementing new safe practices: voluntary
adoption through statewide collaboratives. Quality & Safety in Health Care,
2006;15:289-295.
43. Wilson T, Berwick DM, Cleary PD. What do collaborative improvement projects
do? Experience from seven countries. Joint Commission Journal on Quality and
Safety, 2003;29:85-93.
44. Wagner EH, Glasgow RE, Davis C, Bonomi AE, Provost L, McCulloch D, Carver
P, Sixta C. Quality improvement in chronic illness care: a collaborative approach.
Journal on Quality Improvement, 2001;27:63-80.
45. Carboneau CE. Achieving faster quality improvement through the 24 hour team.
Journal for Quality in Healthcare, 1999;21: 4-10.
46. Lipshutz AK, Fee C, Schell H, Campbell L, Taylor J, Sharpe BA, Nguyen J,
Gropper MA. Strategies for success: a PDSA analysis of three QI initiatives in
critical care. The Joint Commission Journal on Quality and Patient Safety,
2008;34:435-444.
47. Nolan E, VanRiper S, Talsma A, Mageno L, Richter A, Kearly G, Kendrick C,
Leggett S, Crissey J, et al. Rapid-cycle improvement in quality of care for patients
hospitalized with acute myocardial infarction or heart failure: moving from a
culture of missed opportunity to a system of accountability. The Journal of
Cardiovascular Management, 2005;Jan:14-19.
48. Pulcini C, Crofts S, Campbell D, Davey P. Design, measurement, and evaluation
of an education strategy in the hospital setting to contact antimicrobial resistance:
theoretical considerations and a practical example. Disease Management & Health
Outcomes, 2007;15:151-163.
49. Krimsky WS, Mroz IB, McIlwaine JK, Surgenor SD, Christian D, Corwin HL,
Houston D, Robison C, Malayaman N. A model for increasing patient safety in the
intensive care unit: increasing the implementation rates of proven safety measures.
Quality & Safety in Health Care, 2009;18:74-80.
50. Leape LL Kabcenell AI, Gandhi TK, Carver P, Nolan TW, Berwick DM. Reducing
adverse drug effects: lessons from a breakthrough series collaborative. Journal on
Quality Improvement, 2000;26:321-331.
51. Fremont AM, Joyce G, Anaya HD, Bowman CC, Halloran JP, Chang SW, Bozzette
SA, Asch SM. An HIV collaborative in the VHA: do advanced HIT and one-day
sessions change the collaborative experience? Joint Commission Journal on Quality
and Patient Safety, 2006;32:324-336.
52. Walley P, Gowland P. Completing the circle: from PD to PDSA. International
Journal of Heatlh Care Quality Assurance, 2004;17:349-358.
53. Bate P, Robert G and McLeod H. Report on the ‘breakthrough’ collaborative
approach to quality and service improvement within four regions of the NHS.
Health Services Management Centre Research report No. 42. February 2002.
54. Liu CF, Rubenstein LV, Kirchner JE, Fortney JC, Perkins MW, Ober SK, Pyne JM,
Chaney EF. Organizational cost of quality improvement for depression care. Health
Services Research, 2009;44:225-244.
55. Moeschler JB, Amato, RS, Brewster, T, Burke, L, Dinulous, MB, Smith, R, Smith
W, and Miller, P. Improving genetic healthcare: A Northern New England pilot
project addressing the genetic evaluation of the child with developmental delays or
intellectual disability. American Journal of Meidcal Genetics. 2009;151c: 3; 241-254
56. Green CJ, Fortin P, Maclure M, Macgregor A, Robinson S. Information system
support as critical success factor for chronic disease management: necessary but
not sufficient. International Journal of Medical Informatics, 2006;75:818-828.
57. Gordon DB, Rees SM, McCausland MP, Pellino TA, Sanford-Ring S, Smith-
Helmenstine J, Danis DM. Improving reassessment and documentation of pain
management. The Joint Commission Journal on Quality and Patient Safety,
2008;34:509-517.
58. Bolch D, Johnston JB, Giles LC, Whitehead C, Phillips P and Crotty M. Hospital to
home: an integrated approach to discharge planning in a rural south Australian
town. Australian Journal of Rural Health, 2005;13:91-96.
59. Buhr GT, White HK. Quality improvement initiative for chronic pain
assessment and management in the nursing home: a pilot study. Journal of the
American Medical Directors Association, 2006;7:246-253.
60. Mangione-Smith R, Schonlau M, Chan KS, Keesey J, Rosen M, Louis TA, Keeler
E. (2005). Measuring the effectiveness of a collaborative for quality improvement
in pediatric asthma care: does implementing the chronic care model improve
processes and outcomes of care? Ambulatory Pediatrics, 5:75-82.
61. Vos L, Duckers MLA, Wagner C, van Merode GG. Applying the quality
improvement collaborative method to process design: a multiple case study.
Implementation Science, 2010;5:19.
62. Hampton DC, Griffith D, Howard A. Evidence-based clinical improvement for
mechanically ventilated patients. Rehabilitation Nursing, 2005;30:160-165.
63. Gitomer RS. Improving access, quality of care, and patient satisfaction in a
general internal medicine practice. Journal of Clinical Outcomes Management 2005
May;12(5):245-249.
64. Nicholls, S, Cullen R, and Halligan A. Clinical Governance ...after the review-
What next? Agreement and implementation. British Journal of Clinical Governance.
2001;6:2:129-135
65. Barry S, Hughes G, LawtonSmith S. Small is beautiful. Mental Health Today.
2006 Jun.pp.30-3 2006.
66. Esmail R, Banack D, Cummings C, Duffet-Martin J, Rimmer J, Shultz J, Thurber
T, Hulme T. Is your patient ready for transport? Developing an ICU patient
transport decision scorecard. Healthcare Quarterly, 2006;9:80-86.
67. Plsek PE. Quality improvement methods in clinical medicine. Paediatrics,
1999;103: 203-214.
68. Henry D, Muriel FR, Hirway P. Sustaining improvement in surgical infection
prevention measures for hysterectomy. Journal for Healthcare Quality, 2007;29:50-
56.
69. Bittle MJ, Charache P and Wassilchalk DM. Registration-associated patient
misidentification in an academic medical center: causes and corrections. Joint
Commission Journal on Quality and Patient Safety, 2007;33: 25-33.
70. Marang-van de Mheen PJ, Stadlander M, Kievit J. Adverse outcomes in surgical
patients: implementation of a nationwide reporting system. Quality & Safety in
Health Care, 2006;15:320-324.
71. Gray J, Eden G, and Williams M. Developing the public health role of a front
line clinical service: integrating stop smoking advice into routine podiatry services.
Journal of Public Health. 2007;29:2:118-122
72. Kilo CM. A framework for collaborative improvement: lessons from the
institute for healthcare improvement’s breakthrough series. Quality management in
health care, 1998;8:1-13.
73. Mercier CE, Barry SE, Paul K, Delaney TV, Horbar JD, Wasserman RC, Berry P,
Shaw JS. Improving newborn preventative services at the birth hospitalization: a
collaborative, hospital-based quality-improvement project. Paediatrics,
2007;120:481-488.
74. Green CJ, Fortin P, Maclure M, Macgregor A, Robinson S. Information system
support as critical success factor for chronic disease management: necessary but
not sufficient. International Journal of Medical Informatics, 2006;75:818-828.
75. Berwick, DM. Harvesting knowledge from improvement. The Journal of the
American Medical Association 1996;275 877-878.
76. Batalden PB and Stoltz PK. A framework for the continual improvement of
health care: building and applying professional and improvement knowledge to
test changes in daily work. Joint Commission Journal on Quality Improvement.
1993;19: 424-452.
Figure 1: Formal Search Criteria
a) Search Terms:
1. PDSA
2. “plan do study act”
3. PDCA
4. “plan do check act”
5. SDSA
6. “standardi?e do study act”
7. “review agree* implement* demonst*”
8. “rapid cycle change”
9. “rapid cycle improvement$”
10. “rapid improvement cycle$”
11. “Deming cycle”
12. (Deming and cycle)
13. “Deming wheel”
14. (Deming and wheel)
15. “Shewhart cycle”
16. (Shewhart and cycle)
17. “FOCUS-PDSA”
18. “small step change method*”
19. “breakthrough method*”
20. “breakthrough series”
21. “breakthrough collaborative$”
22. “model for improvement”
23. “Rapid Improvement Event$”
b) Databases:
Medline 1996+
Embase 1996+
British Nursing Index 1994+
Pychinfo 2002+
HMIC: Health Management Improvement Consortium
MIDIRS: Maternity & Infant Care
All EBM Review
Figure 2: – Data extraction form (Developed by EC, drawing on TG
(2005) data extraction form, PRISMA & SQUIRE guidelines, and discussions with team
members)
DATA EXTRACTION FORM FOR A PAPER BEING CONSIDERED FOR NES REVIEW OF RAPID
CYCLE IMPROVEMENT PROCESSES IN HEALTH CARE
AUTHOR/TITLE OF PAPER
YEAR OF PUBLICATION
NAME OF REVIEWER
A. [FIRST SIFT] Is the paper relevant to our resear ch question and worthy of further consideration?
Relevance. Do the authors refer to either a PDSA or PDCA process, or IHI
collaborative methodology for use in quality improvement in healthcare?
Worth. Does the paper go beyond superficial description or commentary – i.e. is
it a broadly competent attempt at research, enquiry, investigation or study? [If a
confident ‘no’ to either of these, reject now]
B. How does the paper fit into larger framework?
1. Paradigm. What is
the predominant
theoretical ‘lens’ used?
[if more than one, put
double circle round the
dominant one]
1. Business
management theory
2. Manufacturing
theory
3. Education / learning
theory
4. Leadership theory
5. Translational
medicine theory
(implementing EBM)
6. Complexity /
general systems theory
7. Health economics
theory
8. Organisational
theory
9. Other (specify)
NOTES
2. Type of paper.
How does the paper
fit into our taxonomy
[classify as the MAIN
pitch of the paper]
1. Theory or
conceptual framework
2. Editorial review,
commentary or
opinion
3. Systematic review 4. RCT
5. Non-RCT
experimental or quasi-
experimental study
6. Questionnaire
survey
7. Qualitative interview
study (incl focus
group)
8. Ethnographic study
(‘anthropological’ case
study)
9. Mixed methodology
case study
10. Action research 11. Tool/ checklist /
component of model
12. Guideline/pathway
implementation
13. Comparative case
study
14. Network analysis 15. Other (specify)
NOTES
3. Perspective. What is
the paper’s main unit
of analysis?
Individual Group or
team
Department Organisation Inter-
organisational
Regional
/
national
Multi-level
NOTES
Figure 3 Algorithm of data base search results and exclusions
762 papers identified on electronic
database search
231 duplicates identified by OVID
531 papers abstracts screened for exclusion criteria:
1. Relevance. Does the paper refer to any type of rapid cycle process used in QI
in healthcare?
Total 385 papers excluded
172 rejected as topic not relevant to
question
33 rejected as no data of worth
134 papers excluded as review limited to
last 5 yrs
146 full text articles assessed for
eligibility 52 further papers excluded
25 rejected as topic not relevant to
question
Data extraction from 94 papers (PART B of
review); 44 in total populated our evaluation
Figure 4 –Number and proportions of studies detaili ng specific factors of interest to the review question
Table No.
Description Frequency Percentage
1 B2. Type of paper
1. Conceptual framework 1 1%
2. Editorial review 0 0%
3. Systematic review 0 0%
4. RCT 1 1%
5. Quasi-experimental 47 50%
6. Questionnaire 4 4%
7. Qualitative interview study 5 5%
8. Ethnographic study 0 0%
9. Mixed methods case study 8 9%
10. Action research 0 0%
11. tool / checklist / component of model 3 3%
12. Guideline/pathway implementation 21 22%
13. Comparative case study 1 1%
14. Network analysis 0 0%
15. Other 3 3%
descriptive cost analysis (1) 94 100%
review, not systematic (2)
2 B3. Perspective
1. Individual 23 24%
2. Group / team 23 24%
3. Department 24 26%
4. Organisation 11 12%
5. Inter-organisation 9 10%
6. Regional/national 2 2%
7. Multi-level 2 2%
94 100%
3 C2. Objective
1. PDSA itself 31 33%
2. Outcomes only 63 67%
94 100%
4 3D. Evaluation of outcomes
1. Records 30 32%
2. Other 53 56%
3. Unclear 11 12%
94 100%
5 12A. Data collection
1. Clearly explained 61 65%
2. Not clear 23 24%
3. Partial information 10 11%
94 100%
6 13A Sufficient data presented
1. Yes 47 50%
2. No 35 37%
3. Partial 12 13%
94 100%
7 3C. Comparison group
1. None 34 36%
2. Baseline only 49 52%
3. Yes 6 6%
4. N/a 5 5%
94 100%
8 8. Control
1. stated 6 6%
2. No info 0 0%
3. N/a 88 94%
94 100%
9 13B. Disconfirming observations
1. limited info 69 73%
2. partial info 19 20%
3. clear account 6 6%
94 100%
10 16. Conclusions appropriate
1. Yes 57 61%
2. No 22 23%
3. Unclear 15 16%
94 100%
11 4. Funding acknowledged
1. No 53 56%
2. Yes 41 44%
94 100%
12 17. Reflexivity - potential conflicts of interest
1. No statement 68 72%
2. State none 18 19%
3. Some info 8 9%
94 100%
13 C1. Clear aim
1. Yes 91 97%
2. No 3 3%
94 100%
14 5C / 5D. Trigger
1. Stated 57 61%
2. Unclear 34 36%
3. N/a 3 3%
94 100%
15 7. Receiving the improvement
1. Patient 79 84%
2. Other 12 13%
3. Mixed 3 3%
94 100%
16 9. Improvement aim
1. Safety 16 17%
2. Clinical effectiveness 41 44%
3. Patient centredness 19 20%
4. Timeliness 0 0%
5. Efficiency 8 9%
6. Equity 0 0%
7. Cost Effectiveness 0 0%
8. Other 10 11%
94 100%
17 B1. Paradigm (theoretical lens used)
1. Business management theory 3 3%
2. Manufacturing theory 1 1%
3. Education / learning theory 11 12%
4. Leadership theory 1 1%
5. Translational medicine theory 50 53%
6. Complexity /general systems theory 4 4%
7. Health economics theory 1 1%
8. Organisational theory 22 23%
9. Other - Behavioural theory 1 1%
94 100%
18 11B. Ongoing monitoring plans
1. None stated 52 55%
2. Some info 26 28%
3. Clear plans 12 13%
4. N/a 4 4%
94 100%
19 14D. PDSA evidence
1. Good 17 18%
2. Some 26 28%
3. Poor 38 40%
4. None 13 14%
94 100%
20 15. Local dissemination process
1. Stated 39 41%
2. Not stated 51 54%
3. N/a 4 4%
94 100%
21 6C. Training provided
1. Yes 64 68%
2. No 4 4%
3. Unclear 21 22%
4. N/a 5 5%
94 100%
22 6D. Focus of training
1. PDSA 7 7%
2. Intervention 28 30%
3. Both 27 29%
4. Unclear 24 26%
5. N/a 8 9%
94 100%
23 6A. Delivering changes
1. Clinical staff 73 78%
2. Other 7 7%
3. N/a 5 5%
4. Unclear 9 10%
94 100%
24 11A. Costs
1. Stated 10 11%
2. Not stated 71 76%
3. Limited info 10 11%
4. N/a 3 3%
94 100%
25 5A. Support
1. Stated 75 80%
2. Not stated 3 3%
3. Unclear 14 15%
4. N/a 2 2%
94 100%
26 10E. Sufficient contextual information
1. Yes 69 73%
2. No 24 26%
3. N/a 1 1%
94 100%
27 10A. Country
1. USA 57 61%
2. UK 11 12%
3. Other Europe 5 5%
4. Australia 5 5%
5. Other world 5 5%
6. Mixed 3 3%
7. Canada 8 9%
94 100%
28 10B. Geography
1. Rural 9 10%
2. Urban 19 20%
3. Mixed 17 18%
4. Unclear 49 52%
94 100%
29 10C. Setting
1. Primary 17 18%
2. Secondary 59 63%
3. Mixed 11 12%
4. Unclear 3 3%
5. Other 4 4%
94 100%
Other methods of evaluating outcomes (total greater than 53 as some using >1 method):
Questionnaire / survey 24 Interviews 12 Focus group 7 Specific measure tool 8 Direct observation 6 Incident reports 4 Text analysis 2 Appointment books 2 Others: cost analysis, participation rates, central register, theoretical framework Patient receiving improvement In community/primary care setting: Diabetics 3 Asthma 4 Patients with depression 5 Nursing/residential care 3 Osteoarthritis 1 Screening – women 1
Awaiting appointment 1 Podiatry patients 1 At risk of falls 1 At risk of DVT 1 Palliative 1 Non-specific primary care pt 1 In hospital/secondary care setting: Group defined by location:
ITU/HDU 10 Radiology 2 ‘Hospitals’, not-specified 4 Psychiatry in-patient 2 Others (1 in each): A&E, haemodialysis unit, postnatal ward, ‘bone clinic’, general medical wards, rehabilitation, oncology
Group defined by problem / procedure: Stroke 3 Having transplant 3
Psychiatry 4 Requiring ventilator 2 Having hysterectomy 2 At risk falls 2 Orthopaedic – joint 4 Others (1 in each):
MI, chronic heart failure, COPD, asthma, diabetes, requiring cardiothoracic surgery, incontinence, requiring analgesia, hyperglycaemia, sepsis, learning difficulties, depression, at risk of DVT, labour, colonic surgery, taking atypical anti-psychotics, requiring hospital discharge, palliative, counselling, requiring nasogastric tube, requiring genetic appointment, at risk of pressure ulcers
Further info given on hospitals involved:
Tertiary centre 12 Academic / teaching hospital 12 General hospital 2 Rural / community hospital 3 In-house QI expertise 7 No previous QI experience 1
- 62 -
Figure 5: – Codes for evidence used in the evaluati on synthesis
MECHANISM
Co
nst
rain
t
T1
Measurable process goals that are linked to
outcomes via evidence
T2
Changes are more likely to be sustained
by ensuring the impact of change
(data/progress) is visible
T3
Changes broken down into small-step cycles
completed in rapid succession lead to faster
sustained change
T4
Local spread to surrounding settings/context can
be achieved via key actions
1 C. Start with an easy task & easy to measure
goals (Pulcini et al, 2007)
C. Momentum sustained when cycles run
in setting patients observed in real time
in same location - ventilated pt in ICU
(Pulcini et al, 2007)
C. Collaborative that generated improvement
change ideas internally perceived as less rapid
than those with external advice (telephone
interview 15 collabarative leaders) (Wilson et
al 2003)
C. Buy-in also needed from administrators who
control resources allocation, providing cost-
effectiveness data crucial for ongoing support of
critical care project (Lipshutz et al, 2008)
2
C. Changes relating to healthcare processes
& management easier to implement than
ones directly relating to pt care (Meredith et
al, 2006)
C. Neonatal pain management project
found visibility key to maintaining
momentum (Dunbar et al, 2006)
C. US 40 hospital drug error collabarative
conducted total of 739 tests of change over
15 months over 209 'ramps', real tests of
change in 63% - rest were consensus-building
or educative (Leape 2000)
C. Lack of rigid measures for success & no. of pt
small senior leaders not sufficient evaluation
evidence of benefits for organisational leaders to
facilitate spread of improvements (Gould et al,
2007)
- 63 -
3
C. Processes of care outcomes showed more
consistent impact of paediatrics asthma
collaborative than patient-level outcomes
(Mangione-Smith et al 2005)
C. Method deemed acceptable, feasible &
efficient in creating optimal pt education
tool over 3 week using 5 PDSA cycles in
single endocrinology out-pt clinic
(Varkey et al. 2009)
C. Continuous retesting of score cared for
ICU pt transport was successful in
developing effective tool ( Esmail et al, 2006)
C. Start small again when next ward involved,
even if not changing what you're doing (Pulcini
et al, 2007)
4
C. Changes that were most successful
involved processes not people in US hosp
drug error collaborative (Leape 2000)
C. Ensure process measure data is visible
to staff, patients & relatives to generate
interest (Pulcini et al, 2007)
C. Canadian primary chronic care project
found model was both an instigating &
sustaining factor in making changes, but
other factors also critical (Green et al 2006)
C. Attempt to spread chronic pain project
improvements to other NH facilities - positive to
resources but none implemented PDSA cycles
(Buhr and White 2006)
5
C. Process measurement highly important
but difficult to implement initially (Benn et
al, 2009).
C. Real time 'data wall' ensured visibility
in ICU QI project (Krimsky et al, 2009)
C. 58 site collaborative in secondary care
patient safety US - didn't meet regularly,
approx 20% didn’t use PDSA at all. But when
did do rapid cycle tests had better outcome
(Leape, 2006)
C. Involve people in first team to spread to the
next ward (Pulcini et al, 2007)
6
C. Purpose of PDSA could be confused with
providing a research base for making
changes if evidence not provided -
inappropriate use (Bate et al, 2002)
C. if cycles are slow or no breakdown
into smaller steps then no feedback is
given and there is no stimulation to
further change (Vos et al, 2010)
C. Application of cycles took place over
several months in trainee QI project - 5
partial & overlapping cycles over 10mth
period (Tomolo et al, 2009).
C. US palliative care project found 'emerging
champions' move ahead before hard-won gains
are consolidated & spread (Gould et al, 2007)
- 64 -
7
C. Difficulty knowing when is 'enough'
evidence to warrant a change - critical care
project 2 recommendations later evidence
against emerged (Lipshutz et al, 2008)
C. regular feedback of changes to staff
associated with sustaining change
(Meredith et al, 2006)
C. On average only 3 cycles achieved over (12
month) period of UK orthopaedic
collaborative (Bate et al, 2002)
C. changes can remain stuck at small scale
implementation unless targets are set for it to
spread to a greater number of patients
(Macintosh- Murray 2007)
8
C. US state-wide patient safety initiatives
agreed on goals by consensus process with
stakeholders (Leape 2000)
C. US secondary care pain assessment
project - monthly audit too slow -
needed intense input of 2 hourly rounds
24/7 for 2 weeks to check
implementation, then to daily audits
(Gordon et al, 2008)
C. 3 cycles over 28 month period conducted
in hysterectomy infection prevention project
in US (Henry, Muriel & Thirway 2007
C. Spread beyond the QI project did not occur in
a large collaborative with Dutch mental health
care teams (Franx et al, 2009)
9
C. Aim of outpatient based clinic was to
improve pt understanding at end of visit.
Presumed to result in errors & worse
outcomes but no clear EB (Varkey et al
2009)
C. Difficulty encountered in org with
transient patient populations (Wagner et
al 2001)
C. Cycle 1. 4 months, cycle 2. 4 months, cycle
3. 2 months over 3 yr period in project to
reduce pt ID errors in US hosp, delay due to
mass retraining effort in cycle 1 & 6 month
data check after (Bittle, Charache and
Wassilchalk 2007).
C. Sites that successful in implementing &
sustaining changes were less likely to spread
changes in US mental health collaborative
(Meredith et al, 2006)
- 65 -
10
C. Process improvements eg. Reduced
waiting times/patient journey times not
considered to need evidence base (Vos et al,
2010)
C. Data collection can be difficult when
dealing with very acute conditions - does
not allow time for completion of data
tool in critical care project, but will be
done if part of standard care (Lipshutz et
al, 2008)
C. US hosp goal was to achieve a PDSA cycle
in 6 month period (previous QI took 1-2 yrs)
(Carboneau 1999)
C. Solutions to process outcomes are specific to
the local context so there will be lower levels of
exchange (Vos et al, 2010)
11
C. There can be a distinction between QI
control projects & QI learning projects.
Control goal appropriate when 'fully
understood' (Bloor 1999).
C. Patient volume, acuity at time of
implementation, & poor communication
between disciplines found to be barriers
in neonatal pain management project
(Dunbar et al, 2006)
C. US secondary care pain assessment QI
project implemented single large-scale PDSA
cycle - improvement only reached with
intense input (Gordon et al, 2008)
C. Difficulty with computerised registries limited
spread (Meredith et al, 2006)
12
C. In ICU QI project aims not only EB but
also classed as 'disciplinary' rather than
'discretionary' - aim for 100% (Krimsky et al,
2009)
C. Support & visibility of improvements
needed for at least 6-9 months to sustain
the improvement. Use shared database
with unit specific data to show trends on
units (Gordon et al, 2008).
C. Dutch project to implement database for
reporting of surgical adverse outcomes - used
one large-scale PDSA, poor uptake of new
database, ?due to IT problems & lack clarity
re public availability of data (Marang Van de
Mheen et al 2006)
C. Presentations on QI projects at hospital
teaching meetings led to approaches from other
departments with new project ideas (Tomolo et
al, 2009)
13 C. Goals should have reliable & valid
operational definitions (Pulcini et al 2007)
C. continuous data monitoring required
even when considered that changes are
embedded into normal culture (Pulcini et
C. Smoking cessation advice in podiatry care -
conducted 1 large PDSA cycle 'tackled
everything at once' over a 6 mth period,
C. Less need to spread changes in smaller sites as
most of site included already in initial
implementation (Meredith et al, 2006)
- 66 -
al, 2007) acknowledged after would be better to break
down - unclear why didn't (Gray, Eden and
Williams 2007)
14
C. Collaborative time frames often too short
for changes to processes of care to translate
into measurable outcome results (Leape
2000)
C. Once changes become routine care
need to continue monitoring & retraining
staff - must transition away from QI
experts delivering this (Liu et al, 2009)
C. Anticipated problems in applying PDSA to
larger numbers and allowing adequate time
for learning in endocrinology outpatient QI
project (Varkey et al, 2009 )
C. Acute psychiatric in-patient project stress that
widespread & enduring changes not be achieved
without broader, systemic level changes (Barry)
15
C. Short project time frame (12 month)
perceived as too short for real change in
Dutch mental health collaborative (Franx et
al, 2009)
C. Not applicable when a small change would
necessitate large-scale disruption to
supporting processes - testing bit by bit
therefore causes more disruption (Vos et al,
2010)
C. Incremental small-step change found to be
limited in scope in UK orthopaedic collaborative
- evaluation suggest combine with radical
redesign approach also (Bate et al, 2004)
16
C. Focus should be on process outcomes (eg
compliance with guideline) not clinical
outcomes as these harder to interpret - eg.
Won't account for secular trends (Pulcini et
al 2007)
C. Scale of changes achieved in UK
orthopaedic collaborative perceived to have
been limited by method used, no
'breakthrough' (Bate et al, 2004)
- 67 -
17
C. Difficulty in establishing cause & effect
relationship in UK pt safety collaborative
(Benn et al, 2009).
C. US drug error collaborative removing KCl
from wards 'unlikely to be reversed', (Leape
2000)
18
C. Small GP practice diabetes QI project -
discussing changes enough to implement -
was no need for specific implement plans /
testing to be made (Geboers et al, 1999)
19
C. Rapid cycles are not possible when focus
on processes involving a small number of
patients - will be spread out over time (Vos et
al, 2010)
20
C. Common cause of failure in US hosp drug
error collaborative was spending too long
collecting baseline data - lost momentum
(Leape 2000)
- 68 -
21
C. Short feedback cycles deemed key to staff
compliance with changes as asked to explain
failures while pt still remembered - within few
days (Nolan et al, 2005)
22
C. Discharge planning project in rural
Australia - high level of ownership slowed
development of interventions - ?implication
that were too many iterations of each stage
(Bolch et al, 2005)
23
C. "Real improvement work is often messier
than the logical sequence of the steps, rigid
conformance to steps in the model delays
testing change" (Batalden & Sholtz 1993)
24
C. Incomplete cycles can inform further
cycles, the path to improvement is rarely
linear, some cycles are less important than
others. Strict adherence to steps in sequence
inhibited natural progression of responses to
challenges /opportunities. More complex
- 69 -
modified model needed to be developed
(Tomolo et al, 2009)
- 70 -
CONTEXT
Constraint/
facilitator
T5
Frontline staff may have relevant
knowledge of changes &
implementation skills that can
influence success
T6
Engagement of frontline staff can
influence participation and impact
T7
Existing resources can
sustain PDSA/
changes
T8
The tool is can achieve practice
change/outcomes in multiple
disciplines & organisational
contexts
T9
The context/stability/
size of the organisation
can impact on PDSA
cycles/impact
1 C. Top down goal setting and
detailed intervention toolkit &
instructions to achieve favoured
by GPs in mental health
collaborative (Franx et al 2009)
C. Provide staff with visual
stimulus of room for improvement
based on baseline data (Pulcini et
al, 2007)
C. Assessment of existing
capability (support,
experience, resources, pre-
existing routine monitoring)
may be appropriate before
initiating QI project (Benn et
al, 2009).
C. US secondary care pt safety
project found engagement of
senior administrator important
for success (Leape et al 2006)
C. Large scale organisational
changes will affect staff
motivation to implement &
sustain cycles (Macintosh-
Murray, 2006)
2 C. Blending improvement work
into daily work is a skill to be
learnt - process thinking skills,
not just education on the
intervention (Macintosh-Murray
C. Neonatal pain management
project found staff buy-in difficult
to achieve - seeking staff feedback,
high visibility, present evidence
base helps (Dunbar et al, 2006)
C. Dutch mental health QI
project - lack of dedicated
professional time hindered
project (Franx et al, 2009)
C. Leaders often distracted by
other 'worthwhile goals' & fail to
lead - US palliative care project
(Gould et al 2007)
C. Downsizing & mergers
made US drug error
collaborative low priority &
slowed progress in some hosp
(Leape 2000)
- 71 -
2007)
3 C. UK A&E projects found
consultants more freq accept
PDSA approach than senior
managers (Walley and Gowland
2004)
C. Process goals measured by
documentation - staff can perceive
already doing but not
documenting, so no change to pt
outcome - low motivation
(Gordon et al, 2008)
C. Building change processes
into normal working time
does not work if this is
already saturated -
precipitation elsewhere ie
work by staff in own time
(Macintosh-Murray 2007)
C. Smaller clinics more likely to
sustain change as project more
visible & easier to manage by a
few committed people. Larger
clinics less likely to sustain
change - harder to reach critical
mass via training, dilution of
efforts, less visible changes
(Meredith et al, 2006)
C. Rapid turnover in leadership
and staff identified as barrier in
23 org diabetes collaborative
(Wagner et al, 2001)
4 C. Changes in implementation
were sourced from frontline staff
in ICU QI project (Krimsky et al
2009)
C. Resistance to implementation
from staff initially - loss of control
independent judgement by
following protocol introduced -
improved as saw clinical benefit
(Nicholls , Cullen and Halligan
2001)
C. Average 1.5 fulltime
equivalent input nursing time
spent on daily monitoring,
reviewing results, giving staff
feedback - input secondary
care cardiology ward (Nolan
et al, 2005)
C. A team from an organisation
without QI structures on site
may not be able to learn all it
needs to in whilst doing the
improvement work (Macintosh–
Murray 2007)
C. Large proportion of new
employees may lead to lack of
understanding of inter-
department dependence on
each other (Carboneau 1999)
- 72 -
5 C. Method found not to address
the 'how' of implementation -
difficulty in this area, overall
poor uptake of concept (Bate et
al, 2002)
C. UK A&E project - small
nunmber. sites kept senior control
- changes proposed large scale.
Most frontline staff took control -
these most successful. (Walley and
Gowland 2004)
C. Audit documentation of
pain reassessment QI project
US secondary care -
insufficient staff (Gordon et
al, 2008)
C. Found to be practical & useful
tool for use in emergency dept of
small community hospital with
limited resources (Warburton
2005)
C. Reliance on a few
champions to sustain changes
means changes are fragile when
instability of context occurs
(Macintosh-Murray 2007)
6 C. Specific strategies & templates
of tools needed to implement
recommendations in US
secondary care project - but staff
still found difficult & was huge
task itself (Leape 2000)
C. Active physician involvement
crucial in general medicine access
& quality project (Gitomer, 2005)
C. Discharge planning
project rural Australia limited
by poor IT support, poor
admin support, staff
shortages (Bolch et al 2005)
C. Dominant hierarchical culture
negatively & development
culture positively associated with
motivation to implement PDSA
(Lin et al, 2005)
C. Departure of champion
from critical care project -
realised should have acted
quickly to replace - this with
staff turnover affected
motivation (Lipshutz et al,
2008)
7 C. Intensive technical support
team in US depression QI
project - in depth knowledge of
scientific literature & clinical
organisation structure deemed to
be required (Liuet al, 2009)
C. High level of ownership &
interest in discharge planning
project rural Australia - but slowed
down development as 'everyone
had their say' (Bolch et al, 2005)
C. Limited staff resources
assessed as factor explaining
clinics difficulty in
implementing PDSA in HIV
community clinic, but Extra
work ass with change
became 'part of normal
C. PDSA cycles hard to apply &
did not fit into existing culture of
primary care profession in Dutch
mental health collaborative - not
used to discussing care processes
(Franx et al, 2009)
C. context of teaching hospital
more difficult as house officers
& med students always rotating
- needed monthly input for US
secondary care project on MI
(Nolan et al, 2005)
- 73 -
routine' & improved
efficiency reduced their
workload (Fremont et
al,2006)
8 C. Kilo suggests BTS method
use external ideas for change
which are then adapted locally
(Kilo 1998)
C. US palliative care project chose
not to 'dictate changes' - effective
in building ownership (downside -
multiplicity of effort) (Gould et al,
2007)
C. Difficulty persuading
some staff of importance of
chronic pain assessment
project due to other
demands on their time (Buhr
and White 2006)
C. Trainee educational initiatives
struggle to take off if the clinical
setting staff don not have an
interest in QI - needed
'champions in the department -
PDSA model did not translate to
the application of rapid cycle
change in the clinical context in a
medical trainee QI project
(Tomolo et al, 2009).
C. Contexts with high
proportion of part-time staff
will be more difficult to
implement (Macintosh- Murray
2006)
9 C. Teams in US hosp drug error
collaborative used established
changes - did not dev own
changes from human factors
concepts provided as starting
points for doing so (Leape 2000)
C. Involvement with UK patient
safety collaborative found to create
sense of ownership (Benn et al,
2009)
C. Measures that are not
gathered on existing hospital
IT systems require significant
staff time to collect data
(Vos et al, 2010)
C. The PDSA model was found
to shift the organisational culture
in UK pt safety collaborative to
that more favourable to QI
(Benn et al, 2009).
C. Difficulty in retaining staff
commonly stated (65% sites) as
barrier to change in US mental
health collaborative (Meredith
et al, 2006)
- 74 -
10 C. Mental health collaborative in
US relied on implementation of
change guidance from external
support team (Meredith et al
2006)
C. UK orthopaedic collaborative
'clinicians did not appear to wish to
lead' - resulted in project manager
led collaborative, local ownership
not achieved at scale anticipated
(Bate et al, 2002)
C. ICT component viewed as
critical to success in
Canadian chronic disease
management primary care
project (Green et al, 2006)
C. May have been some evidence
of implicit culture change as
result of UK orthopaedic
collaborative but not conclusive
(Bate et al, 2009)
11 C. Most collaboratives used
outside expert groups to suggest
improvements rather than
participants themselves
(telephone interview with 15
collaborative leaders worldwide)
(Wilson et al 2003)
C. PDSA was lowest rated
component (of 17) of UK
orthopaedic collaborative - not
seen as the main driver for change.
Views were mixed but most not
convinced of their value (Bate et al,
2002)
C. Administrative & IT
support needed for US
diabetes collaboratrive
(Wagner et al, 2001)
C. Poor leadership support
significantly affected projects in
US mental health collaborative
(Meredith et al, 2006)
12 C. Genetic evaluation project felt
that lack of quality improvement
expertise contributed to slower
implementation (Moeschler et al
2009)
C. PDSA methodology rated as
highly important component of
UK hosp patient safety
collaborative - no sig diff in rating
between frontline staff & board
level staff (Benn et al, 2009)
C. Data collection systems
often newly created for UK
patient safety collaborative
(Benn et al, 2009)
C. Senior leadership support
perceived as crucial, except in
contexts with teams in self-
contained units where already
working more independently eg.
Neonatal ICU (Wilson et al,
2003)
- 75 -
13 C. Concept of 'small tests of
change' difficult for most
healthcare workers to learn, lack
knowledge as to how to collect
performance data, 1/2 day
instruction insufficient (Leape
2000)
C. Physician / nurse resistance
could be 'formidable barrier' in US
hosp drug error collaborative –
possibly due to anxiety / including
workload (Leape 2000)
C. Integrating process
changes into electronic
medical record is high cost,
but support sustainability &
spread once implemented
(Liu et al, 2009)
C. Culture of clinician autonomy
need constrained in UK patient
safety collaborative (Benn et al,
2009)
14 C. improvement targets set by
individual teams in US palliative
care project - inc ownership but
difficulty identifying appropriate
measures (Gould et al, 2007)
C. primary care provider resistance
to change encountered in 76% of
sites involved in US mental health
collaborative, but associated with
more change activity (Meredith et
al, 2006)
C. Estimated cost of
newborn preventative service
project was $29000 per
hospital (excl staff time in
doing QI work) (Mercier et
al, 2007)
C. Patient volume, acuity at time
of implementation, & poor
communication between
disciplines found to be barriers
in neonatal pain management
project (Dunbar et al, 2006)
15 C. Teams in US hosp drug error
collaborative found defining
measures & collecting data
difficult (Leape 2000)
C. Motivation is complex & not
easily predicted by contextual
factors (Lin et al, 2005)
C. cost of mental health
collaborative ranged $30,000-
252,000 in US mental health
collaborative - mainly staff
time costs (Meredith et al, et
al, 2006)
C. Virtual project environment
created by website as main
source of information not
helpful to professions unused to
this format of communication
(Franx et al, 2009)
- 76 -
16 C. Learning sessions shortened
in HIV collaborative - found
that more than half of teams
struggled with basic concepts at
end of session, needed to
practice in further LS (Fremont)
C. Acute psychiatric inpatient
project UK found teams joining
later did so out of choice & were
'best performers', less success with
those 'volunteered' by trust (Barry
et al 2006)
C. Complex organisational
change required dedicated
staff time to test the changes,
measure the impact, refine
the approach, & spread the
changes - sig funding (Leape
2000)
C. Collaboratives work better in
organisations combining intra-
org features (such as PDSA) with
inter-org features (such as
external reporting) (Nembhard
2009)
17 C, US mixed primary &
secondary palliative care project -
found between learning session
coaching needed re PDSA cycles.
Many teams came to report
waves to consolidate (Gould et al
2007)
C. Process outcomes that affect
several departments - resistance
more likely as motivation focussed
on their area of work. Political
solution may be needed (Vo et al,
2010s)
C. Strong local team co-
ordinator needed in Dutch
mental health QI project -
employed by governmental
health department - external
financial support needed
(Franx et al, 2009)
C. All trusts valued PDSA, but
was sig diff variation in how
much in UK patient safety
collaborative - may be variability
in appropriateness for diff org
(no info on trust characteristics)
(Benn et al. 2009)
18 C. If cross-department
collaboration needed the
supporting group might need to
make initial networking contacts
(Tomolo et al, 2009)
C. Face-to-face training
preferred to long distance
coaching on telephone /
internet in US palliative care
project (Gould et al, 2007)
C. Greater number. of changes
observed in communities with
higher incomes, but no
association between site
characteristic & sustaining &
spreading change (Meredith et
- 77 -
al, 2006)
19 C. Team members from all sites &
disciplines needed for Canadian
ICU transport project to achieve
'early buy-in' (Esmail et al, 2006)
C. QI in a large highly
systemetised organisation
may be more costly than in
small independent practice
(Liu et al, 2009)
20 C. Ownership & interest grew after
initiation by an 'early adopter'
respected by staff (Pulcin et al,
2007i)
C. State-wide pt safety
project in US - 8 hospitals
didn't participate - unable to
gather resources & personnel
for the project (Leape 2000)
21 C. Improving chronic illness care
collaborative in US 43
organisations - high staff
motivation re PDSA , lower with
clinical staff especially depression -
perception of supervisor support
needed for staff to perceive it as a
- 78 -
useful tool (Lin et al 2005)
22 C. Empowerment associated with
increased scrutiny. Staff ambivalent
as have to accept greater
responsibility & 'blame'. Managers
lose security (Bloor 1999 - review)
23 C. motivation high in UK A&E
teams where staff empowered to
make changes - can became threat
to senior management & resistance
from this group instead. Need to
be consistent with overall aims of
organisation - clear at start on
boundaries of this if control given
to frontline staff (Walley and
Gowland 2004)
- 79 -
24 C. Use of national experts helped
motivate staff in US secondary care
project on 58 sites (Leape 2006)
25 C. Learning sessions rated as
essential to staff motivation in UK
pt safety collaborative (Benn et al,
2009)
26 C. Frontline doctor financial
incentives critical to their
participation in Canadian primary
care collaborative (Green et al
2006)
27 C. Focus on individual patients can
increase motivation due to strong
identification of frontline staff with
individual patient stories
(Macintosh- Murray 2007)
28 C. Small changes fuelled
excitement & enthusiasm for larger
changes in US secondary care
- 80 -
patient safety initiatives (though
still only partial implementation)
(Leape 2000)
29 C. Smaller changes can lower
expectations of benefits of a
change and reduce motivation for
involvement (Vos et al, 2010)
30 C. staff compliance biggest barrier
to ventilated pt initiative -
overcome by demonstrating
positive pt outcomes (visibility)
(Hampton et al, 2005)
31 C. Reporting could be seen as
burdonsome, demoralising,
confusing, and of little meaning
when results shown 'too early', and
therefore may not be fully
implemented (Bate et al, 2002)
- 81 -
32 C. Staff awareness of continued
lack of improvement in aims
despite efforts led to
discouragement - decreased
motivation (Warburton 2005)
33 C. Perceived early failure of an
initiative should not be allowed to
stop - neonatal pain management
project found initiative viewed as
failure actually successful on
follow-up data (Dunbar et al 2006)
34 C. Creating competition between
departments & celebrating even
small successes can create
enthusiasm in critical care project
(Lipshutz et al ,2008)
- 82 -
Figure 6 – Template: Assessing the applicability o f using PDSA to
achieve quality improvement in your context & objec tives
Key
Dimension Facilitating factors
Tick if
applies Constraining factors
Tick if
applies
1 Overall aim translates easily into
measurable goals
Overall aim does not translate
appropriately into measurable
goals
2
There is a strong and well-
established evidence base for the
overall aims and measured goals
The evidence-base is weak or
frequently being updated
3
The choice of goals is informed
by previous experience of their
elsewhere
Measured goals are not based on
experience of their use elsewhere
to meet the aim
4 Measured goals can have a 100%
target ‘every patient, every time’
Goals are discretionary – there is
a need to allow clinical judgement
5 Measured goals can be tightly
defined
The measured goals are open to
interpretation
6
There is a clear relationship
between cause and effect in the
achievement of measured goals
Confounders & secular trends are
expected to have an impact on
the progress towards measured
goals
1.1
Imp
rove
men
t go
als
& e
vid
ence
bas
e
7
Clinical benefit is expected to be
demonstrated within the time
frame of the project
Impact on clinical benefit unlikely
to be demonstrated in time frame
of project
1 The cycles are being carried out in
a defined geographical area
The cycles are being carried out
over dispersed locations
1.2.
1 Sust
ain
ing
-
visi
bili
ty
2 A data-wall to present progress
towards measured goals can be
Staff are dispersed and so a
central visual display of data is
- 83 -
located somewhere that is visible
to all relevant staff
difficult to achieve
3
Cycles are planned to be
conducted rapidly (within working
day of a staff member)
Cycles are expected to take place
over weeks or months
4
Staff are given feedback on
progress towards measured goals
in realtime or a very short
timeframe
There is a delay before data is
feedback is given to staff
5
Cycles take place in a location
with a relatively stable patient
population
Cycles take place in a setting with
a fast transition of patients
6
The target population have
chronic problems allowing time
for documentation and collection
of progress towards measured
goals
The target patient population are
those with acute problems
needing immediate action
meaning there is little time to
document achievement of goals
7
Ongoing monitoring of goals is
planned to continue and remain
visible beyond the testing phase
There are no plans for ongoing
monitoring of goals when the
changes become part of routine
care
1
External input is available to
advise on change ideas that might
achieve goals, and so cycles of
tests can start quickly
Staff need to generate the change
ideas themselves to achieve the
goals
2 Goals can be broken into small
steps
It is difficult to break the goals
down into small steps
1.2.
2 Sust
ain
ing
- si
ze &
sp
eed
of
cycl
es
3
Opportunities for conducting
cycles are expected to arise
regularly (at least daily)
The aim applies to a patient
group or clinical situation seen
only occasionally
- 84 -
4
The measures remain relatively
stable unless an intentional
change to the system is initiated
There is significant daily or
seasonal variation in the
measured goals and so baseline
trends must be established over a
prolonged time frame
5
The measurement of the goal
(study) can be carried out at the
time the change is made (act)
There is a delay between when
the action is taken and measuring
the impact of that change
6
There is an ability to take decisive
action to continue cycles when
there is a spectrum of opinion
There is an expectation that all
the staff a change may affect will
be consulted at each stage of the
cycle
7
A flexible approach is taken to the
completion and expansion of
cycles
There is an expectation that all
cycles must be completed fully
and expanded sequentially
1
Senior management are agreeable
to promote the spread the
changes and provide the
resources to do so
Senior management are not
convinced of the benefit of the
changes and so are resistant to
spreading the changes
2
Staff are motivated to restart with
small tests of change and adapt
the change ideas to the new
setting
There is an expectation that the
improvements will spread directly
without any need to start small
tests again in the new setting
3
Staff from the first area of
implementation can help start off
cycles and give advice in the new
setting
Work roles are not flexible, so
staff are unable to learn directly
from those involved at the first
place of implementation
1.3
Sp
read
ing
the
chan
ges
4 Targets to achieve spread of
improvements to an incrementally
increasing number of patients can
No intentional targets to spread
changes are set. Spread is
anticipated to occur without
- 85 -
be set additional input once successful
changes have been implemented
in one area
5
Systems used to help implement
changes or monitor goals are
similar in the area you wish to
spread the improvement to
There are different systems (such
as IT infrastructure, working
patterns, communication systems)
in the area you wish to spread the
improvement to
6
Places of interaction between staff
from different disciplines or
locations already take place on a
regular basis e.g. educational
meetings
The are no pre-established
connections with the with area or
group you wish to spread the
improvements to
7
Spread to a wider area can be
achieved without the need for
broader systematic changes to the
organisation
Systematic change in the
organisation (that is not amenable
to PDSA) is needed to spread the
changes to other areas
1
Staff are able to decide on the
overall aim and set measurable
targets based on their knowledge
of the evidence base
Staff are unfamiliar with the
scientific literature surrounding
their overall aim, and are unaware
of goals which may be used to
measure improvements
2
Staff involved are aware of the
processes & systems that support
their clinical role
Staff have little awareness of the
processes & systems that support
their clinical role
2.1.
1 Sta
ff -
Ski
lls &
kn
ow
led
ge
3
Staff are able to translate their
knowledge of the scientific
literature and processes of care
into specific ideas to implement
changes to achieve the goals set
Staff are likely to struggle to
identify change ideas that they
can implement in their context
without specific training
- 86 -
4
Staff have the skills & experience
needed to collect, collate, present,
and interpret both baseline data
and ongoing monitoring data
Staff have little or no experience
of data collection, collation,
presentation & interpretation
1
The evidence base for the goals
set and the deficiencies at baseline
are clearly presented
The evidence base is not
communicated clearly to staff and
there is no perception of any
need to improve
2
Staff perceive a clear link between
the measured goal and clinical
benefit, and are confident in the
evidence-base
Staff perceive that meeting the
goals will only achieve better
documentation rather than better
clinical care, and are concerned
they will lose the ability to
exercise clinical judgement
3
Frontline staff are involved in all
stages of the deciding the aims &
measurable targets set, and all
stages of the PDSA cycle
The overall aim, measured goals,
targets, and the change ideas are
given to staff by an external
group or a management team
4 Staff are able to take part in the
initiative on a voluntary basis
Staff involvement is coerced or
compulsory
5
The team involved in conducting
cycles come from the same site or
discipline
The cycles involve several
departments and disciplines
making and testing changes
2.1.
2 Sta
ff -
en
gage
men
t
6
There is representation from each
area (if the goal involves several
sites or disciplines) on the team
deciding on changes and
monitoring goals
Some of the sites or teams
involved in implementing
changes are not represented in
the process of setting goals and
monitoring goals
- 87 -
7
‘Early adopters’ can be identified
who are respected by other
frontline staff & can help drive
the cycle forward
There are no obvious frontline
staff members to champion the
changes who have the respect of
their colleagues and who are
enthusiastic about using PDSA
8
Staff are confident of senior
support and are given clear
guidance on the responsibility
they have for making changes
Staff are concerned they will be
held accountable by their seniors
if the changes they introduce are
unsuccessful
9
Managers are willing to allow
frontline staff to take ownership
of changing processes of care
provision and agree clear
boundaries with staff
Managers perceive frontline staff
shaping the organisation of care
delivery as a threat to their role,
resist allowing staff more
responsibility in this area and fail
to make boundaries explicit
10
External expertise is available and
is viewed as a positive support by
staff
External expertise is unavailable,
or is perceived as having a
negative impact
11 Financial incentives can be
provided when goals are achieved
There is no ability to provide
financial incentives when goals
are met
12
Staff can see clear benefit for
individual patients which
encourages them to expand the
changes to a larger group
The changes made in each cycle
are perceived by to be too small
to result in significant clinical
benefit and motivation is lost
13
The cycles can be completed
rapidly and awareness of positive
outcomes helps maintain staff
momentum
The cycles are completed slowly
so staff are less aware of the
outcomes of the changes and lose
enthusiasm
- 88 -
14
The progress towards achieving
the measured goals can be clearly
displayed to all frontline staff
implementing the changes
It is difficult to display the
progress towards the goal visually
on a regular basis or in a location
that is seen by all frontline staff
involved
15
Successes can be acknowledged
and celebrated, and a sense of
competition between teams
implementing the changes can be
created
There is no acknowledgement of
success achieving goals and staff
are unaware of the performance
of other teams
1
A full assessment of both the
current capacity and the financial
and resource implications can be
carried out before the start of the
project
The capacity within the system
currently is unknown and there is
no information or ability to carry
out an assessment of the resource
implications
2
There is capacity to reallocate
staff time to allow involvement in
planning and testing changes
Staff time is fully saturated
already, with no ability to drop
any activities to create space
3 Monitoring of goals makes use of
data gathered routinely already
Monitoring of goals requires a
new system of data collection
4
There is capacity to invest in IT
systems that may be needed to
monitor the progress towards the
measured goals
There are no resources available
to update or create IT systems to
assist in data collection and
monitoring
2.2.
1 O
rgan
isat
ion
– r
eso
urc
es
5
Finances for co-ordination, staff
training, additional staff time,
implementation costs, and
monitoring costs are available
There is no provision for
additional finances to cover the
costs of planning, implementing,
sustaining, and spreading the
changes
- 89 -
1
Senior management make the
improvement process a key
priority for the organisation
The organisation has other
priorities that are unrelated to
quality improvement
2
The organisation is large enough
to provide some internal expertise
and experience, but small enough
to allow improvements in one
area to be visible to others and
allow spread
The organisation is either too
small to provide internal support
to staff, or is so large that the
impact of changes are diluted
within the larger system and are
difficult to perceive
3
There is already a strong
developmental culture in the
organisation, that emphasises
continuous improvement and the
involvement of all staff in that
process
There is a strong hierarchical
culture in the organisation with
changes initiated in a top-down
approach and a history of slow
adaptation to change
4
There is a strong perception of
senior management support for
frontline staff
There is a lack of senior
management support, or senior
management support is not
perceived by the staff
5
There is a strong emphasis on
evidence based medicine and the
need to follow best practice
guidelines within the organisation
There is a strong culture of
clinician autonomy with the
organisation, with a greater
emphasis on clinical judgement
than the evidence base
2.2.
2 O
rgan
isat
ion
-
clim
ate
6
There are clear lines of
communication established
already between the sites and
disciplines involved in the project
Communication between the
groups involved in the
improvement project is poor or
non-existent
- 90 -
7
The organisation has a culture of
sharing information and learning
with other organisations, and has
established relationships and
systems to support this
There is little experience of
sharing learning with other
organisations, and there is no
relational or structural basis on
which to do this
1
There has been recent
organisational stability and no
significant changes are anticipated
There have been significant
structural changes to the
organisation, either recently or
ongoing
2.2.
3 O
rgan
isat
ion
- s
tab
ility
2
There is a high proportion of staff
who have worked in the site for a
long time and are unlikely to
move on, the staff have regular
and predictable working patterns
There is high staff turnover either
due to a high proportion of junior
clinical staff, high proportion of
part-time or shift workers, or
difficulty retaining staff
- 91 -
- 92 -