Embed Size (px)
Citation preview
1
Using information about personal
outcomes: Examples from the
Meaningful and Measurable Project
Emma Miller and Karen
Barrie
June 2015
2
Using information about personal outcomes
Introduction
Embedding outcomes in practice has been a goal of public services in Scotland for several
years (Scottish Government 2010, 2011). As illustrated in Figure 1, the three components of
an outcomes approach have been identified as engagement, recording and using information
(Cook and Miller 2012). While considerable effort has been invested in supporting outcomes
focused engagement or conversations at the frontline, and more recently recording outcomes,
less progress is evident with using outcomes information for decision-making at the wider
level. The quest to advance the use of collated personal outcomes information in service
settings underpinned the Meaningful and Measurable project, which informs this paper.
Figure 1: The Components of a Personal Outcomes Approach
Background to the Meaningful and Measurable project Meaningful and Measurable was an ESRC funded research project that ran from November
2013 until March 2015 with the following aims:
1. To develop and test out in practice approaches to the qualitative and quantitative
analysis of personal outcomes data and use of this information for decision making
within organisations.
2. To capture emergent good practice in the analysis and use of personal outcomes
information and disseminate this widely to practice, policy and academic audiences.
3. To explore the practical, epistemological and political tensions inherent in this work
and capture evidence as to the benefits and limitations of different approaches.
The project facilitated eight organisations (practice partners) to participate in local research
activity, each supported by an academic mentor. Drawing on the principles of both Action
Research (Sharp, 2005) and dialogical approaches to Knowledge Exchange (Nutley et al, 2007)
the project adopted a collaborative action research approach to progressing analysis and use
of personal outcomes information in practice. Collaboration was primarily supported by four
data retreats, enabling the partners to share and reflect on learning and to work together to
build an evidence base as to what works, when and how.
Cook and Miller 2012
3
Project learning and previous project papers Broadly, the project tapped into the aspiration to use both qualitative and quantitative data
to inform decision-making. However it has long been recognised that data must be of
sufficient quality if they are to inform decisions. Early in the project, reviews of their
records by practice partners revealed a range of issues with recording and measuring
outcomes that required responses if the project was to achieve its objectives. In particular:
1) As it became clear that narrative recording of outcomes required attention in its own
right and drawing on examples from the project, a paper comprising a collection of
narrative recordings was published to support this (Miller and Barrie 2015).
2) It also became apparent that the tendency to concentrate on statistical data in
services meant that understandings of qualitative data remain limited. A series of
project briefings were produced as a result, covering the differences between using
individual stories and qualitative analysis, issues around sampling and generalisability,
and different approaches to qualitative analysis, now available as a single paper
(Barrie and Miller 2015a).
3) As we grew to understand how different conceptualisations of outcomes impacted on
measurement, a separate paper explored the limits to establishing causality with
outcomes, conceptualisations of outcomes measurement, and approaches to
categorisation (Barrie and Miller 2015b).
The papers, together with further information about the project approach and methods, are
available on the project website.
About this paper Despite the range of emerging challenges identified by the practice partners and the need to
channel efforts accordingly, there were also examples of personal outcomes information
being used in various ways within and across their organisations. This paper provides a broad
overview of the main uses of collated personal outcomes information identified and
facilitated through the project, notably in the following areas of longstanding interest:
Practice and service development
Service planning and commissioning
Using more personalised methods to measure outcomes to gauge performance
Returning to the project aims, this paper engages directly with the first and second aims,
while touching on the third (which will be covered in more detail in a subsequent paper). In
so doing, it demonstrates how the views of people using services can be applied to decision-
making, while also including practitioner perspectives.
Some uses of information, or changed understandings about the use of information, are
linked to individual partners and this paper draws on the final reports of six practice partners,
with reference to the other two partners. Alongside this, many insights and changes emerged
as a result of project interactions, with partners regularly feeding into the collaborative
process and then applying the learning to a new phase of development in their own settings,
and so on. The paper therefore also considers how project information fed into wider
learning, and use of information at the collective level. In addition to considering areas of
progress, the paper highlights areas requiring further attention in the use of information. The
next section of this paper sets out how each partner used collated outcomes information in
their organisation, followed by information uses supported through collective project activity
and learning.
4
Using personal outcomes information overview
Table 1 below was produced in consultation with the eight practice partners:
Table 1 - Project partner aims, methods and uses of information
Partner Aims Methods Uses of information Type of use
Angus To better understand if, and how,
an outcomes focused approach can
support working practice and be
more effective in supporting the
wellbeing and independence of
children and their families
Further aims identified to develop
consistency of approach, collate
and analyse data for evaluation
and commissioning purposes
5 practitioner
interviews followed
by case file audit
(20 cases)
4 informal staff
interviews
1 focus group of 9
practitioners
(jointly with
Penumbra)
Following discussion with colleagues further
development to support linking of wellbeing
scores to personal outcomes plans is being
built into training
Tool amended to include practitioner analysis
to paint a more complete picture
Inconsistencies across teams demonstrated a
clear need for multi-agency training and
support, under discussion by GIRFEC
evaluation group
Practice/service
development
Practice/service
development
Practice/service
development
Bridgend Phase 1 – to explore how we
currently record and evidence an
outcome focused approach to
practice, with a view to
identifying improvement
opportunities and people’s roles in
taking the necessary change and
development forward
Phase 2– to identify and agree
what development and change is
Workshop with
senior and frontline
managers
3 focus groups with
frontline managers
3 focus groups with
practitioners
Making of 2 linked
Revised assessment, planning and review
framework to shift recording away from
deficits and outputs
Identification of an integrated multi-disciplinary community network team to test out new approaches and ways of recording outcomes for people they work with Use of written and digital stories to support
shared understanding of personal outcomes
Practice/service
development
Practice/service
development
Practice/service
development
5
Partner Aims Methods Uses of information Type of use
needed to influence and support
decision making and have better
information which can inform
practice development,
commissioning and performance
management
In parallel to the local pilot work,
the project linked directly to
participation in early stage pilots
(continuing) to develop outcomes
measurement for the Welsh
National Outcomes Framework.
digital stories
Welsh NOF methods
adapted in response
to project learning
e.g. learning event,
collaborative
communication
sessions
Commissioned an integrated IT system to
support integrated recording by different
professionals
A more appreciative and collaborative
approach to workforce development
Following the project, work underway to
develop a new supervision policy across the
service, linking to a revised appraisal and
quality assessment approach
Linking in the findings of the Meaningful and
Measurable project to the National Outcomes
work resulted in a shift to focus on
meaningful conversations at the frontline
rather than top-down imposed 'tick boxes'.
Practice/service
development
Practice/service
development
Practice/service
development
Performance
management
Practice
development
Performance
management
Edinburgh Our aim has been to better
understand the practice of
recording of outcomes in
assessment prior to and after the
implementation of the Social Care
(Self-directed Support) (Scotland)
2013 Act.
Audit of 5 case files
each for 5
information team
members
Plan to engage with
practitioners to be
completed following
the project
The intention of this is to work with
practitioners to share the learning from this
analysis and will focus on developing shared
purpose and approach to the recording,
categorisation and use of outcomes, to
support practice and service planning.
Develop recommendations for organisational
learning in relation to practice, management
and reporting purposes
Practice/service
development
(future)
Practice/service
development
(future)
Performance
Management
(future)
East
Renfrew
shire
Exploring frontline practitioners
understanding of personal
outcome data and how that
Detailed search
through 50 case files
for outcomes data
Case file audit data and overall project
findings on recording linked back to service
improvement work around improving
Practice/service
development
6
Partner Aims Methods Uses of information Type of use
influences practitioners’ service
improvement activity.
Sick leave limited
progress to phase 2
to consider
performance
management
recording practice via a specific post with
continuing remit for embedding outcomes
Launching support planning tool based on
outcomes focused conversation – based on
pick list of talking points themes with free
text to give more detail
Practice/service
development
Moray The focus for the Moray project is
to explore the use of personal
outcomes data in relation to
telecare and home care visits of
15 minutes of duration.
This focus will aim to consider the
extent which information on
personal outcomes can support
service improvement,
performance management and the
commissioning of services.
Analysis of case file
quantitative data
Audit of 5 case files
(qualitative)
5 interviews with
social workers
Focus group with 2
senior managers
Follow up meetings arranged with Penumbra
and VOCAL to help develop recording practice
In the context of integration, the report will
contribute to a more bottom up approach to
engaging with qualitative personal outcomes
data with front line members of staff
All of the project reports have been reviewed
by the Community Care Performance
Management Group, and were used to frame a
discussion about local outcomes data trends
Consideration is being given to how personal
outcomes data can support a SDS micro
commissioning approach.
Practice/service
development
Practice/service
development
(future)
Performance
management
Commissioning
(future)
Penumbra To explore to what extent
outcomes focused conversations
are actually happening
Based on this knowledge, to then
explore to what extent these
conversations are reflected in the
6 interviews with
staff
9 case file audits
2 informal staff
interviews
Have now included recording within our I.ROC
and Planning4Hope training days.
We are using information from this project to
help design new technology-based tools to
help the recording process, for example an
I.ROC app and the use of tablets by staff. This
is to help issues of time for good quality
Practice/service
development
Practice/service
development
7
Partner Aims Methods Uses of information Type of use
associated reporting practices.
1 focus group of 9
practitioners
(jointly with Angus
Council)
recording, and reduce duplication of efforts.
Improved understanding of challenges with
recording practice are informing changes in
the guidance, materials and training for good
personal outcomes recording.
Inclusion of far more detailed questions
regarding I.ROC use and recording of
outcomes in internal audit structure.
Greater clarity about the need for qualitative
and contextual data to make sense of scores
has strengthened resolve to avoid use of
quantitative data alone to inform
commissioning.
Practice/service
development
Performance
management
Commissioning
Stirling Improve identification of personal
outcomes with people using the
reablement service
Develop a common understanding
and approach to recording
outcomes, including quality of life
outcomes across the service
Improve links between practice
and performance by involving
reablement practitioners in
developing performance indicators
Improve effective communication
of outcomes between assessors
Audit of 4 case files
of individuals who
had used
reablement more
than once
3 meetings held
with representatives
across the council to
share perspectives
and share and
embed learning
Further detailed
audit of 4
reablement case
Clearer recording of progress against
reablement goals supportive of better
decision making about support required to
enable people to live independently
Clearer understanding of what good recording
looks like in the context of reablement
established through audit, engagement with
team and through MM participation
Understanding of good recording criteria built
into internal audit processes
Different measures considered by reablement
staff who agreed to testing them
Practice/service
development
Practice/service
development
Performance
management
Performance
management
8
Partner Aims Methods Uses of information Type of use
and providers.
Use learning about documenting
personal outcomes in improving
services and performance
files
2 meetings with the
reablement team
Following the project, Stirling linked in to a
Benchmarking Network pilot to develop
measures for reablement
Performance
management
VOCAL The initial focus of VOCAL’s action
research project was how
information on personal outcomes
is being recorded on our electronic
recording system by staff and
volunteers and how that
information could support service
improvement, planning and
performance management within
and beyond VOCAL. However as
we have analysed the information
we hold on personal outcomes our
focus has shifted to getting a
better understanding of what
supports good outcome focused
recording practice.
Detailed analysis of
5 sets of records
Focus group with 12
practitioners
Small group
discussions involving
15 staff
2 individual
interviews with
focus group
members
Further review of 4
sets of casenotes
Developing understanding of the skills
involved by supporting discussion and debate
within and between teams on good recording
practice, including using the project report to
promote team discussion
Support staff/volunteers skill development via
training, supervision and team meetings
Elevating the status of recording within the
organisation so that it is seen as a crucial part
of the support we offer to the carer.
Supporting staff to see recording as an
opportunity for reflection on what the carer
has said and what they have understood from
this, informing both the action taken and the
next conversation with the carer
Using information recorded by staff and
volunteers to inform service development
(improving) and performance management
(proving) to inform recording practice
Practice/service
development
Practice/service
development
Practice/service
development
Practice/service
development
Performance
management
9
Partner Aims Methods Uses of information Type of use
Collective To develop a better understanding
of different approaches to
outcomes focused working and
ensure that project findings are
grounded in diverse perspectives
Data retreat discussion
and debate
Modelling outcomes
approach
Understanding of similarities and
differences between the various partner
approaches at both practical and
conceptual levels enabled more
inclusive and productive dialogue
Enabling change
Practice
development
To support good recording
practice across organisations by
agreeing what good recording
looks like
Data retreat discussion
including review of
sample documentation by
practice partners
Authoring and distribution of Recording
Guidance
Practice/service
development
To support good recording
practice and appropriate use of
information across organisations
by understanding the limits and
possibilities of quantitative data
about outcomes, and more broadly
Data retreat discussion
Literature review
Analysis and review of
data retreat discussions
Authoring and distribution of Measuring
Outcomes in Service Settings paper
Practice/service
development
To develop capacity in the use of
qualitative data in service settings
Data retreat discussion
Literature review
Analysis and review of
data retreat discussions
Authoring and distribution of Supporting
Use of Qualitative Data in Service
Settings paper
Practice/service
development
To use more personalised methods
to measure outcomes to gauge
performance
Data retreat discussion
Review of project reports
Analysis and review of
data retreat discussions
Authoring and distribution of this report Performance
management
10
Before discussing developments within each of the three broad areas of information use,
namely practice/service development, performance management and service
planning/commissioning, it is essential to emphasise two points:
First, there is a considerable degree of interplay between the three areas. e.g. use of
information for commissioning/service planning depends on practitioner interest in and
recognition of the ‘contribution’ of different influences on outcomes, including the
contribution of the person, family, social supports and community resources alongside
service inputs, and then recording this. What practitioners prioritise through recording is in
turn shaped by prevailing performance management regimes.
Secondly, and critically, the primary aim of gathering outcomes information is to engage with,
understand and identify the person’s priorities, and those of their family as appropriate, and
to track progress. By using information at the individual planning level, the organisation
already ensures that improvements are driven by the priorities of its users. That is why it has
been a longstanding concern to ensure that the predominant emphasis on, and drive to
measure outcomes do not undermine the quality of interaction (Cook and Miller 2012).
Additionally, many of the practice and service developments identified in this paper directly
impact on individual planning and it is important that these uses of information are
recognised as fundamental, and prerequisite to sustainable culture change, rather than
perhaps deemed secondary to developments in the use of outcomes information effected at
service, organisational or wider levels.
Practice and service development
Overview As shown in table 1, practice and service development represented the main area of collated
outcomes information use by the practice partners, and is discussed in this section as
follows:
Two key areas of practice and service development relate to the first two
components of an outcomes approach: engagement and recording, and these are
discussed in the next two sub-sections.
Consideration is then given to how information from the project was used to improve
understanding about measurement of outcomes.
Following from that are examples of scale measure data being used to inform service
developments.
Finally, examples of the use of qualitative data by several project partners are
highlighted, including consideration of how improved capacity to use qualitative data
in itself represents progress in terms of using information.
Supporting good conversations It is important to note that in parallel to the concepts of engagement and recording, most
practice partners undertook both interviews/focus groups with practitioners and a review of
their records in carrying out their local action research projects. Ultimately, it is not just
how information was used in the organisations which is of interest here, but also how
information was gathered.
11
All eight practice partners intended to involve practitioners, and despite many challenges,
six managed to do so, with the other two planning to do so after the end date. There were
several rationales for doing so.
First, there was debate during the data retreats about the extent to which good quality of
recording reflects the quality of conversation underpinning the record. While views on this
varied, there was agreement that recording which meets core criteria was a reasonable
indicator of good underlying practice, but that other information was required to obtain a
full picture. Direct engagement with practitioners was viewed as critical to avoid mistaken
assumptions about the records:
Early engagement with practitioners was essential, highlighting assumptions that
would have been made by looking at the data alone (Angus report P11)
The Penumbra and Angus partners shared the explicit aim of exploring the extent to which
outcomes focused conversations were taking place, as well as the extent to which these
conversations were being recorded. A joint study was conducted, and stepping through a
recently completed case record as part of each practitioner interview proved insightful,
anchoring the discussion in concrete terms. This work also highlighted that, in some settings
and for some people, the use of structured tools with a strong visual component, can actually
aid conversations, both by helping to manage complexity in the context of somewhat chaotic
lives and by helping people to look at the broader picture rather than getting bogged down in
the latest issue. Sharing this finding with other practice partners was critical.
In addition, the focus on practitioner engagement within the project also served to support
shared learning through dialogue and information exchange. This is significant because it has
been argued that modelling an outcomes approach within organisations supports
implementation at the front line, including conversations which help to clarify purpose and
shared understandings (Cook and Miller 2012). Some partners also engaged different
departments to develop a shared understanding across the service and further learning here
proved invaluable in indicating where improvements could be made. This commitment in the
longer term to collaborative, relational and responsive learning is integral to understandings
of practice development associated with transformational change (Patterson et al 2011).
Overall there was strong consensus around the benefits of promoting good conversations
within the organisation as a means of encouraging good conversations between practitioners
and people using services. Engaging with different teams was highlighted as helpful in
developing shared learning and consistency:
Developing an understanding of the skills involved by supporting and encouraging
discussion and debate within and between teams (VOCAL report p10)
Engaging with teams also clarified how outcomes might be identified in different settings:
Direct engagement with reablement staff was valuable in providing opportunities to
exchange ideas about the limits and possibilities of personal outcomes in a
reablement context (Stirling report p10)
Through reviewing outcomes data and engaging with staff over the course of the project,
several partners identified improvements to their organisational approach to engaging with
staff (Bridgend, Edinburgh, VOCAL). For instance, in an email conversation with the project
lead for VOCAL after the project ended, she informed:
12
[Two team leaders] have encouraged all staff to read the project report and then
used it as a basis for further discussion and reflection. The feedback I have had is
that it has been very positively received as staff feel it reflects their experiences,
and that it has helped to focus and clarify people’s thinking around recording... This
feels like the right way to do it now, as before we would probably have attempted
an organisation wide session, which might have brought more structure but less
individual reflection.
Using outcomes information to improve recording practice An audit or review of records was the most consistent approach to accessing outcomes
information in the project as a whole, undertaken by all practice partners at the start of the
project. The main focus here is review or audit of narrative recording about outcomes,
which proved to be a wake-up call in many cases:
Audit of existing recording – provided clear evidence of practice and systemic issues, and provided a realistic baseline for improvement work, and dismissed ‘wishful thinking’ (Bridgend report p10)
While initial feedback from practice partners at the first data retreat suggested some
concerns about the quality of recording, these concerns solidified following discussion of
findings at the subsequent retreat, with the idea emerging that the audit had ‘lifted a rock’
on the records, with lots of system bugs emerging as barriers to good recording practice
(Miller and Barrie 2015). While quality of narrative recording was a common concern,
Penumbra also included a quantitative component to its audit, counting the number of fields
in completed tools which included comments and also the number of words in the comments,
finding a steady increase in the four years since the tool was introduced.
With regard to practice and service developments implemented by partners in response to
the review of records, as identified in table 1 these include IT adaptations, tool
developments and staff support and development initiatives. However the shared
experience of conducting the review also resulted in converged efforts to find a common
approach to recording. Progress made on defining what good recording looks like, and the
development of methods to support this represent an area of significant collective practice
and service improvement (Miller and Barrie 2015). ‘Elevating the status of recording’ signals
progress for the following reasons:
Recording shapes the quality of interaction; good recording principles support good conversations
Recording is an analytical tool
As records are increasingly shared with and owned by people using services, records such as support plans impact on the person’s perception of themselves, their outcomes, and the roles that they and others play in working towards those outcomes
Agreeing principles across different agencies is important in the context of integration
Effective recording of narrative data is essential to understanding whether and how outcomes are being achieved
The quality of recording influences the potential for effective decision making both at
the individual and service level.
In the Stirling reablement team, considerable effort was invested in developing recording
practice. While reablement has a focus on goals, this tends to concentrate on change
outcomes or functional goals, with less attention paid to quality of life. The concentrated
efforts to include quality of life outcomes for people using reablement in Stirling, supported
by a new tool and guidance, training and supervision for staff, are important for the
13
individuals using the service, but also for the sustainability of the service. Social isolation for
example, is strongly associated with morbidity among older adults living in the community
(Nicholson 2012). The work on including quality of life considerations in the service was
linked to increasing knowledge amongst the staff of local resources to support this, and
positive feedback from individuals and their families as well as staff (Stirling report).
While audits/reviews of the records proved valuable, caution was urged by partners about
relying on the records alone, with the feedback loop with practitioners emphasised by many.
Measuring outcomes – improved understanding of limits and possibilities Measurement was not a core focus of any of the action research projects, and therefore does
not appear against the aims for any of the practice partners in table 1, but it did feature
heavily in discussions and debate at the data retreats.
Five of the eight practice partners included the use of scales measures in their outcomes
approaches prior to the project starting. Some of these partners investigated how the scale
measures were used by practitioners, with two partners identifying that they wished to
review their approach to measurement, including reconsidering their choice of scale, as a
result of the learning (East Renfrewshire and Moray).
However, on the whole less effort was invested in trying to improve scale measure data
quality than to understanding its limits and possibilities. For example, one of the two
organisations which had developed a wellbeing measure, Penumbra, used information from
practitioner focus groups to highlight concerns about the isolated use of scores by external
organisations, because of mistaken assumption that increases in scores necessarily reflect
improved outcomes, or the converse of this. This contributed to a shared view of the need to
consider narrative data alongside the measures to sense check the statistics (Barrie and
Miller 2015b). We return to use of narrative data in the section on using qualitative data.
Another key area of learning to emerge from the collaborative process involved an increased
understanding of key differences between measurement of wellbeing and measurement of
personal outcomes and how the conceptual differences play into the complexities of both
scale measures and categorisation, and in turn how this impacts on the conversation. As
identified in table 1, the two services using wellbeing measurement approaches identified
improvements required to link the measures to personal planning processes and associated
amendments to staff development programmes.
In terms of the extent to which it is possible to ‘measure personal outcomes’, the shared
experiences of the different practice partners supported the conclusion that:
We can use an evidence-based framework (or tool)
Consisting of sufficiently high level categories of outcomes
To help determine, locate, organise, or map personal outcomes
In a consistent way [across the service/organisation]
And in conjunction with a context-appropriate scale measure [that does not compromise the quality of engagement]
And thereby generate outcome measurements
That can be used with caveats and not in isolation
To help inform different levels and types of decision making
While there was consensus around the need to avoid using quantitative outcomes data in
isolation, discussions between practice partners also resulted in a better shared
understanding regarding the conditions necessary for appropriate use of aggregated score
data. Alongside the need for contextualising information, these included minimum service
14
population size and consensus that working with shifts over time made more sense than
expending effort trying to tie down the meaning of each scale point. Equally, it was
recognised that other considerations, such as the frequency of reporting and number and
frequency of reviews deemed necessary before shifts in progress towards outcomes would be
meaningful in a statistical sense, were highly context-specific, and that such understanding
had to be developed organically within the organisation over time. There was also learning
about how the monitoring of patterns and trends in score data can prompt further inquiry for
understanding and improvement, as summarised below.
Monitoring outcomes scores for understanding and improvement At the time of writing, analysis of project data about measuring outcomes is continuing, with
some briefings to support understanding available in an interim guide (Barrie and Miller
2015b). However, it is possible to highlight themes identified by practice partners with
regard to monitoring their quantitative data. Most partners using scale measures identified
benefits in being able to monitor patterns and trends, mainly with a view to understanding
what is working for whom, where and how. Within VOCAL, simply being able to see the issues
being identified most by carers was found to be helpful:
So since we started this whole approach consistently, almost without fail… The
carer’s own health and wellbeing has been the issue that’s come up more frequently.
In the conversations with the carers. Closely followed by being better informed. So
there’s a consistent pattern. So I suppose that’s where I would, sort of, use
aggregated data (VOCAL, Data Retreat 4)
Penumbra also highlighted the benefits of monitoring trends within different parts of a
service, and investigating whether there were changes or activities that might contribute to
the differences observed. However, again here there was emphasis on seeking other sources
of information before drawing conclusions, recognising that in some circumstances reduced
scores might not be indicative of a problem. For instance, within its self-harm service, it was
found that in order to improve self-esteem scores, many people often have to end
relationships having a detrimental effect on their wellbeing, such that initial decreases in
social network scores can be helpful and necessary, rather than problematic. This leads to
consideration of the role and use of qualitative data by project partners.
Using qualitative data – improved capacity as a marker of progress All practice partners worked with qualitative outcomes data to varying extents. We have
already outlined ways in which qualitative data directly impacted on understanding and
improvement around recording and engagement practices. For instance, reviewing their
narrative outcomes records resulted in practice partners collectively identifying criteria for
good recording, which in turn fed into an agreed set of criteria for the project and a
collection of recording examples to support further embedding of the approach (Miller and
Barrie 2015). Locally, the importance of using qualitative data alongside outcomes score data
has also been emphasised and further examples of the use and impact of qualitative data are
threaded through the sections that follow. However, the authors believe it is worth
considering developments in the use of in the qualitative data in a separate section for the
following reason:
A personal outcomes approach requires a significant shift in the way data are
managed, analysed and understood. Standard practice in most organisations has
focused almost exclusively on gathering, analysing and reporting quantitative
information (Barrie and Miller 2015a).
15
Just as ‘elevating the status of recording’ signals progress, improved capacity to gather,
engage with and ultimately analyse qualitative data in itself represents progress in terms of
using information and implementing a personal outcomes approach.
At a minimum, the practice partner research projects included direct engagement with pre-
existing narrative data about outcomes in the course of reviewing their records. Through
engaging with qualitative data some partners, particularly those in performance and
information roles, reported heightened awareness of the complexity of people’s lives and the
difficulties practitioners face in addressing this complexity. The effects at times were
profound (Barrie and Miller, 2015b).
One example of narrative records about outcomes being used analytically is provided by
VOCAL, where analysis of both quantitative and qualitative data is supporting decision-
making about where to concentrate resources. VOCAL has found that information captured at
review can build a picture of what carers identify as improving their health and wellbeing,
which, confirming the importance of good conversations, includes the opportunity and
support to reflect on their caring role and its impact, as well as getting breaks and
counselling.
In addition to using qualitative data about outcomes, over the course of the project, most
practice partners also gathered qualitative data through face-to-face interviews/focus groups
with staff. The process of qualitative data gathering itself resulted in some cases in
significant changes in understanding in different ways, with practice partners directly
experiencing factors shaping conversations and their distillation for reporting purposes, as
well as uncovering barriers and supports to effective outcomes focused practice from the
content of these discussions.
Discussions at data retreats and subsequent review of transcripts from these sessions also
helped to increase understanding that the various partners were at very different stages in
understanding the potential role of qualitative data, with several having no experience of
qualitative data analysis. As the academic team grew to understand that some partners
were importing assumptions about sampling, aggregation and generalisability from their
knowledge of quantitative data analysis, they worked to produce briefings to help clarify
some key differences (Barrie and Miller 2015a). This represents a further example of the use
of information at the collective level in the project.
Performance management Several partners used their experience of reviewing their records, their engagement with
staff, and the shared learning from data retreats to inform continuing quality monitoring
processes. For instance:
Penumbra identified that they included detailed questions regarding their tool use
and recording of outcomes in their internal audit structure.
Stirling had built understanding of good recording criteria into their internal audit
processes.
In Bridgend work continued after the project, as identified in table 1, “to develop a
new supervision policy across the service, linking to a revised appraisal and quality
assessment approach.”
16
Each was viewed as a way of ensuring that emerging practice development exercises were
having the desired impact, and to monitor ongoing staff support needs.
Internal quality and performance monitoring was being linked in various ways to staff
development, training opportunities and supervision. Less attention was paid to external
performance reporting requirements, although initially identified as aims by four practice
partners. There was progress in this area, although perhaps not consistent with predominant
understandings of performance management as being centrally and statistically driven.
Stirling had identified an objective of engaging reablement practitioners in defining
indicators related to personal outcomes and this progressed by the end of the project, with
the measures still be tested. Bridgend had a direct link to the testing of the Welsh national
outcomes framework and, as identified in table 1, reported that project findings significantly
influenced the pilots:
Linking in the findings of the Meaningful and Measurable project has resulted in a
shift to focus on meaningful conversations at the frontline rather than top-down
imposed 'tick boxes' (Bridgend detail from table)
As a third sector provider, Penumbra was clear that it would not be using scores to compare
services for performance management, or to set targets (Penumbra report p4). The
commissioning officer in Stirling who attended project meetings saw the national health and
wellbeing outcomes in Scotland as potentially providing a common language to inform
performance, as long as they were “seen as high-level signals, allowing for flexibility as to
how information is gathered at the local level, rather than ‘forcing people into boxes’
(Stirling report p9)
Although not consistent with more managerialist conceptualisations of performance
management, the methods emerging across different organisations point to ways of achieving
consistent and effective improvements in ways that can be evidenced for both internal and
external purposes. This topic will be reported in more detail in a subsequent paper.
Service planning and commissioning As with most other uses of data described above, qualitative and contextual information is
necessary to make sense of outcomes scores with regard to planning and commissioning as
the scores alone can be subject to misinterpretation. While quantitative data might give a
broad indication of which outcomes are being improved and which are relatively static for
example, they cannot identify contributory factors towards these patterns, or explain that in
some circumstances reduced scores might not be indicative of a problem. Penumbra has used
insights from quantitative and qualitative data to make the case for more support for the
development of self-esteem within self-harm services and for greater use of peer support
across all service areas. VOCAL also report that they increasingly mine their data to
contribute to service planning and improvement both within and beyond the organisation,
and provide the following example:
Carers of people with addictions, having noticed an increase in the number of carers
raising issues around supporting someone with addictions, the data held was analysed
to put forward a case for further resource (VOCAL report p10)
17
Interestingly, both organisations had self-initiated the introduction of personal outcomes
data into their reports to commissioners and also funding applications, and both had found
this was increasingly being asked for by commissioners, evidencing a bottom up approach. Of
course, quality of narrative recording emerges as a key consideration here. One of the
statutory partners found through examination of the records and in interviews with
practitioners, that further work was required to generate practitioner interest in how
outcomes were achieved. The need to develop interest in and understanding of contributory
factors was identified as important by a manager there:
The right thing for the right situation and the right person, you know. We have many stories of people who, you know, have worked really hard to be as independent as possible. And telecare has a crucial role in doing that – whether it be through medication prompts or families feeling much more relaxed. But equally we know there are heaps of telecare products that get issued at great expense to the department and then sit in somebody’s lobby cupboard (Service manager, focus group, Moray)
The head of community care in also participated in this focus group and added that
practitioners would ideally be gathering the ‘real qualitative data.’ Taking the example of
social isolation, this manager argued that the social work role should be getting to know and
understand the circumstances of the individual and their family and to work around that
understanding, rather than the answer necessarily being to develop a generic anti-loneliness
service in commissioning terms. Discussions about a need to broaden the focus of
commissioning emerged at the end of the project, to include the professional role,
community development and micro-commissioning, in addition to what are more traditionally
understood as services – and it was argued that outcomes data should help to shape this.
In VOCAL, where attention has turned relatively recently to recording practice, particularly
through the Meaningful and Measurable project, it was found that practitioners tended to
place more emphasis in the records on what was not working and the actions required. Based
on comments in one of the staff interviews, it was considered that this might be due to a
perceived need to justify service involvement. Although there were examples of particular
services being recorded as having contributed to improved outcomes, which was helping to
build a picture within the service of where best to invest resources, there was further work
to do to develop this and to ensure that the assets of the carer were acknowledged more too.
Penumbra identified that reports generated by their wellbeing approach had been positively
received by commissioning bodies as extra evidence of how their services are working. It had
some concerns however about being under pressure to evidence how its service or parts of
their service was increasing scores as evidence of effectiveness, again raising concerns about
the risks of using quantitative data alone:
Greater clarity about the need for qualitative and contextual data to make sense of
scores has strengthened resolve to avoid use of quantitative data alone to inform
commissioning (Penumbra report p4)
A commissioning officer in Stirling, who regularly attended meetings of the project, was
clear that its emerging approach to using logic modelling as a basis of its outcomes approach
to engaging with providers could help to ensure that preventative and low level support
remained part of the picture.
18
Conclusion
A key concern of all Meaningful and Measurable partners is that information gathered through
practice encounters requires to be of sufficiently good quality to be used for decision-making,
at both individual and collective levels. The project demonstrated that this requires
understanding of the barriers faced by practitioners, and responses to their needs and
recommendations. It was in these latter areas that most progress was made, with increased
understanding of the complexities involved, and a range of adaptions developed accordingly.
This exploratory work increased understanding on various themes, including the limitations
of measurement alone, of the criteria for good recording and use of qualitative data. This
increased understanding in turn was linked to further service adaptations to maintain the
momentum, including appraisal and supervision of staff and audit. These processes could be
described as performance management. Although internally driven, consistency between
partners was encouraged through the collaborative nature of the project.
While progress was also made with use of data for external performance reporting and
commissioning, this was more preliminary, with ideas emerging about a more bottom up
approach, consistent with, rather than diverting attention away from the improvement work
taking place. Distinctions between ‘measurement for judgement’ and ‘measurement for
improvement’ have been considered previously in relation to outcomes (Miller 2012). While
this topic is the subject of continuing analysis, it is worth mentioning here with regard to the
need to clarify ‘purpose,’ and the benefits which can be realised when the priorities of
people using the service and improvement which supports that focus, remain paramount.
References Barrie, K. and Miller, E. (2015a) Supporting the use of qualitative data in service settings, Meaningful and Measurable Meaningful and Measurable Project
Barrie, K. and Miller, E. (2015b) Measuring personal outcomes: Collected briefings from the Meaningful and Measurable Project Meaningful and Measurable Project Cook, A. and Miller, E. (2012) Talking Points personal outcomes approach: A Practical Guide, Edinburgh: JIT Miller, E. (2012) Measuring Personal Outcomes: Issues and Challenges, IRISS: Glasgow
Miller, E. and Barrie, K. (2015) Recording Personal Outcomes in Support Planning and Review: Practical Examples. Meaningful and Measurable Project Nicholson, R. Nicholson (2012) A Review of Social Isolation: An Important but Underassessed Condition
In Older Adults, The Journal of Primary Prevention 33(2-3):137-152.
Nutley, S.M., Walter, I., & Davies, H.T.O. (2007) Using Evidence: How Research Can Inform Public
Services. Bristol: The Policy Press.
Patterson, M., Nolan, M., Rick, J. Brown, J., Adams, R., and Musson, G. (2011) From Metrics to
Meaning: Culture Change and Quality of Acute Hospital Care for Older People: NHS SDO
Scottish Government (2010) A guide to implementing getting it right for every child: Messages from pathfinders and learning partners, Edinburgh: Scottish Government Scottish Government (2011) Renewing Scotland’s Public Services: Priorities for Reform in response to the Christie Commission, Edinburgh: Scottish Government Sharp, C. (2005) The Improvement of public sector delivery: supporting evidence based practice
through Action Research, Scottish Executive Health Department, Edinburgh
