Embed Size (px)
Citation preview
Department of Health
Carer supports: timeliness and outcomes measures Final report
16 July 2014
HDG Consulting Group
PO Box 442 East Melbourne VIC 8002 Contact: Ro Saxon, Director Tel: 03 9416 0447 Email: [email protected]
HDG Consulting Group i
Executive Summary Project context
In Victoria, unpaid family members and friends are the key providers of care to people who
require support in their daily lives. These unpaid care roles are essential to the wellbeing of
the population and the economy.
Services such as those provided through Home and Community Care (HACC) and the
Support for Carers Program (SCP) are designed to contribute to the capacity of carers to
continue in their care role. The ability to measure outcomes is essential to understanding
the extent to which these services make a positive difference and contribute to meeting the
support needs of carers and care recipients, and is fundamental to continual quality
improvement.
In 2012 the Victorian Auditor General’s Office (VAGO) conducted an audit of carer support
programs and concluded that greater rigour was required in relation to the monitoring of
outcomes to determine program effectiveness. In response to the report recommendations,
and to support ongoing quality improvement by service provider organisations, the
Department of Health commissioned a project in relation to HACC (Respite and Planned
Activity Groups) and the SCP. The aim of the project was to develop a monitoring
framework with carer outcomes measures to better enable the monitoring of outcomes for
carers, including the timeliness of services and support.
The project scope included four elements: a targeted literature scan to inform the project
context, selected consumer input, consultation with service providers through a survey
process, and analysis of feedback tools currently used by service providers. A Project
Reference Group provided sector based advice and input. HDG Consulting Group conducted
the project between March and June 2014.
Findings
The literature scan revealed that outcome measures can be used at the individual level,
organisational level and systemic level. There are various international examples of how this
occurs within practice. Australian studies have concluded that capturing outcomes is a
complex process and that the design and implementation of an outcomes focussed system
requires a long timeframe.
Consultation with carers and care recipients highlighted a range of domains that were
considered to underpin positive outcomes from their perspective. These included access to
information, communication and respect, a shared understanding of needs, the quality of
services, and the positive difference made to the care role.
Consultation with service providers occurred through a comprehensive survey process that
explored current practice in relation to the collection and reporting of carer outcomes. The
survey results indicated that most service providers use a range of methods and do not rely
on any one single method alone, and a high use of verbal feedback throughout service
delivery. Process related items tend to be collected in a more formal way, whereas quality of
life and care relationship items tend to be collected through verbal feedback.
HDG Consulting Group ii
The survey results supported the premise that an outcomes framework with a standard set
of questions, incorporated within a uniform carer outcomes tool, would be beneficial to
service providers and carers.
Analysis of the feedback tools currently used by service providers highlighted that they tend
to be specific to each organisation. While there is some congruence between the content of
the feedback tools the wording of items and the response scales are highly variable.
In developing an outcomes framework and uniform set of questions, key design
considerations included the necessity to be able to administer the questions by various
modes (e.g. written, verbal, online), and the capacity for such to be integrated into existing
work practices such as assessment, care planning, review and exit conversations.
Conclusion
The project found that there is clearly both the opportunity for, and a level of interest in, the
development and introduction of a carer support outcomes framework and uniform set of
questions and measures, that can be administered using a range of modes, and at various
points in the care pathway, to improve service effectiveness.
The logic model proposed in this report (refer Figure 9) provides an overarching logic and
articulates the high level desired outcomes as:
Carers and care recipients are informed and able to access services that are
responsive and effective in meeting their needs.
Carers and care recipients receive respect for, and support to maintain and sustain,
their care role and relationship.
Carers and care recipients receive services that make a positive contribution to their
quality of life, health and wellbeing.
Congruent with these three high level outcomes, an outcomes framework has been
developed that provides a uniform set of seven items (refer Figure 10) to underpin data
collection. The items reflect carer feedback, are applicable across the diversity of HACC and
SCP service provider organisations, can be implemented through written, verbal or online
modes, and can be used in everyday practice as well as more formally.
It is recommended that the items as detailed in the outcomes framework be designed in a
range of formats with associated guidelines. This recommendation and other
recommendations are listed below.
HDG Consulting Group iii
Recommendations
Recommendation 1
That the indicative questions as detailed in the proposed carer outcomes measures
be designed into a range of formats (staff administered, carer administered), and the
associated guidelines developed.
Indicative timing: July – December 2014
Recommendation 2
That further feedback is sought from the consumer groups that provided input at the
commencement of the project, regarding the proposed carer outcomes measures,
formats and guidelines.
Indicative timing: July – December 2014
Recommendation 3
That consideration is given to pilot testing the proposed carer outcomes measures,
formats and guidelines. The aim of the pilot would be to test the carer outcomes
measures and items for analysis of reliability, validity and usability purposes. The
pilot would be designed to include HACC and SCP and to test the formats with a mix
of organisation types, staff categories and implementation modes, and include
practitioner and carer evaluation.
Indicative timing: January – June 2015
Recommendation 4
That the project outcomes are used to inform the HACC Active Service Model (ASM)
project that plans to investigate and identify service user outcome measures from an
ASM perspective.
Indicative timing: To occur over 2014/15.
Recommendation 5
That the project findings are made available and communicated to project
participants and other key stakeholders.
Indicative timing: July – December 2014
These actions will progress the use of carer support outcome measures at an individual and
organisation level, and will contribute to a systemic approach in the longer term. The
proposed approach will respond to the VAGO recommendations and inform continual quality
improvement of services to support the essential role and wellbeing of carers.
HDG Consulting Group iv
Contents 1 Introduction ........................................................................................................... 1
1.1 Background ..................................................................................................... 1
1.2 Project rationale .............................................................................................. 2
1.3 Project aim and implementation ........................................................................ 3
2 Context and literature ............................................................................................. 5
2.1 Outcome measures .......................................................................................... 5
2.2 Selected literature ............................................................................................ 6
2.3 Instruments.................................................................................................... 11
3 Consumer consultation ........................................................................................... 15
3.1 Approach ....................................................................................................... 15
3.2 Consultation ................................................................................................... 15
4 Service provider consultation .................................................................................. 19
4.1 Survey development........................................................................................ 19
4.2 Sample size and organisation type .................................................................... 19
4.3 Type of support delivered ................................................................................ 20
4.4 Current practice – carer satisfaction ................................................................. 21
4.5 Current practice - process outcomes ................................................................. 22
4.6 Current practice - care relationship outcomes .................................................... 23
4.7 Current practice - quality of life outcomes ......................................................... 24
4.8 Most important items ...................................................................................... 25
4.9 Requirements of the Victorian Carers Recognition Act ........................................ 26
4.10 Barriers, enablers and suggestions ................................................................... 26
4.11 Summary of survey findings ............................................................................. 27
5 Feedback tools used by service providers ................................................................ 30
5.1 Current practice .............................................................................................. 30
5.2 Content items ................................................................................................. 30
5.3 Measurement scales ........................................................................................ 33
5.4 Summary ....................................................................................................... 34
6 Conclusion ............................................................................................................ 35
6.1 Findings ......................................................................................................... 35
6.2 Proposed logic model ...................................................................................... 36
6.3 Outcomes framework ...................................................................................... 38
6.4 Benefits and recommendations ........................................................................ 40
Appendix 1 .................................................................................................................. 42
Detailed survey results .............................................................................................. 42
Appendix 2 .................................................................................................................. 59
Sample carer outcomes survey - Guidelines for use ..................................................... 59
HDG Consulting Group v
List of tables
Table 1: Examples of instruments to measure carer variables .......................................... 13
Table 2: Respondent organisation type .......................................................................... 20
Table 3: Analysis of feedback tools forwarded by service providers .................................. 30
Table 4: Examples of items seeking views and ratings about various aspects of service provision ..................................................................................................................... 31
Table 5: Summary of findings by project element ........................................................... 35
Table 6: Benefits of proposed approach by project element ............................................. 40
Table 7: Collection of process related outcome measures ................................................ 45
Table 8: Process outcomes – collection method by organisation type ................................ 46
Table 9: Collection of care relationship outcome measures .............................................. 48
Table 10: Care relationship outcomes – collection method by organisation type ................ 49
Table 11: Collection of quality of life outcome measures .................................................. 51
Table 12: Quality of life outcomes – collection method by organisation type ..................... 51
Table 13: Most important items to collect ....................................................................... 54
List of figures
Figure 1: Project elements ............................................................................................. 3
Figure 2: Consumer consultation handout version 1 ........................................................ 16
Figure 3: Consumer consultation handout version 2 ........................................................ 18
Figure 4: Feedback methods ......................................................................................... 21
Figure 5: Feedback methods – process related items ...................................................... 22
Figure 6: Feedback methods – care relationship items ..................................................... 23
Figure 7: Feedback methods – quality of life items .......................................................... 24
Figure 8: Proposed logic model ..................................................................................... 37
Figure 9: Flow from logic model to outcomes monitoring ................................................. 37
Figure 10: Proposed carer outcomes measures ............................................................... 39
Figure 11: Feedback methods ....................................................................................... 42
Figure 12: Feedback methods – process related items ..................................................... 44
Figure 13: Feedback methods – care relationship items ................................................... 47
Figure 14: Feedback methods – quality of life items ........................................................ 50
Disclaimer
This report has been prepared by HDG Consulting Group under contract for the Department of Health. The
report relates primarily to a sector consultation process supported by HDG Consulting Group, and the content
reflects the views, opinions, ideas and suggestions made by survey respondents and subsequent analysis of
information by the consultants. The report has been prepared as one element of a broader process and is
intended for consideration alongside a number of other inputs in relation to carer supports and outcome
measures.
HDG Consulting Group 1
1 Introduction
1.1 Background
The Department of Health has a core objective to achieve the best health and wellbeing for
all Victorians. The Ageing and Aged Care Branch, located within the Department’s Mental
Health, Wellbeing and Ageing Division, has a broad policy responsibility for the older
population who are generally healthy and active, as well as those who require support
services.
In Victoria, unpaid family carers and friends are the key providers of care to people who
require support in their daily lives. These unpaid care roles are essential to the wellbeing of
the population and the economy.
The Victorian Carers Recognition Act (2012) aims to recognise, promote and value the role
of people in care relationships. It acknowledges the important contribution that people in
care relationships make to the community and the unique knowledge that carers hold of the
person in their care. The Act applies to government departments, local councils and service
organisations including providers of the Commonwealth-State jointly funded Home and
Community Care (HACC) program and the Support for Carers Program (SCP).
The HACC program is jointly funded by the Commonwealth and State governments. HACC
funded services provide basic support and maintenance to eligible frail older people, young
people and adults with a disability, and their carers whose capacity for independent living is
at risk, or who are at risk of premature or inappropriate admission to long-term residential
care. There are approximately 400 HACC service providers in Victoria. A diverse range of
organisations deliver services, including health services, community health services,
community service organisations, local government, Aboriginal Community Controlled
Organisations (ACCOs) and ethno-specific and multicultural organisations.
The Support for Carers Program (SCP) is funded by the Victorian government. The SCP
provides flexible respite and support for unpaid carers of older people, older carers, and
carers of younger people with dementia. There are approximately 49 SCP service providers
in Victoria. Similar to the HACC program, SCP services are delivered by a diverse range of
organisations.
Funded HACC and SCP service provider organisations are required to meet a range of
standards including the national Community Care Common Standards (CCCS). These
standards assess each service provider against expected outcomes. Service provider
organisations are expected to implement continual quality improvement processes to ensure
that service provision is of high quality and reflects the needs of service users. Feedback
from service users is an essential part of this process and many organisations have
developed satisfaction surveys and other feedback tools to measure consumer responses to
service provision.
HDG Consulting Group 2
1.2 Project rationale
The HACC program in Victoria has developed a range of quality initiatives at the
programmatic level, which are implemented in day-to-day service delivery practice by
service provider organisations. For example:
the Framework for Assessment in the HACC program in Victoria1 and an associated
practice guide that supports a consistent approach to high quality assessment and
care planning practice by service provider organisations
the Active Service Model (ASM), which is based on the premise that HACC services
should, wherever possible, assist people to improve their capacity and enable them
to live independently at home for as long as possible. Building on people’s strengths
and abilities and focusing on improving quality of life, social participation and
functional capacity are fundamental to this approach.
The ability and capacity to measure outcomes is essential to understanding the extent to
which individual services, and the service system, deliver quality services that make a
difference to, and meet the needs of service users.
Past practice has focussed on the collection of output data that counts the type or amount
of service provision, but not the outcome of the service provision from a service user
perspective. For example, while the HACC Minimum Data Set (MDS) collects information
about HACC service users and the type and amount of assistance being provided, it does not
currently include outcome measures from a service user perspective.
In 2012 the Victorian Auditor General’s Office (VAGO) conducted an audit of carer support
programs. The audit report concluded that the Department of Health and the Department of
Human Services consider carer support as core business, however, that greater rigour is
required in relation to specific aspects of program practice and monitoring, including the
monitoring of outcomes to determine program effectiveness. The report included specific
recommendations that these departments monitor and report on timeliness of access to
carer supports, and develop outcome measures for carer supports and monitor outcomes.
Two key recommendations arising from the audit report were:
Recommendation 4: That the Department of Health and Department of Human
Services monitor and report on timeliness of access to carer supports
Recommendation 5: That the Department of Health and Department of Human
Services develop outcome measures for carer supports and monitor outcomes.
In response to these recommendations, and to support ongoing quality improvement by
service provider organisations, the Department of Health commissioned this project to
investigate and recommend a monitoring framework for outcomes measures and timeliness
of access, for carers of HACC (Respite and Planned Activity Groups) and the SCP. It was
intended that the results of the project would:
contribute to the Department of Health’s actioning of the VAGO recommendations; and
provide a potential framework and measures to better enable service providers to
record and monitor timeliness of access and carer outcomes, to support the
continual quality improvement of services.
1 See http://www.health.vic.gov.au/hacc/assessment.htm
HDG Consulting Group 3
1.3 Project aim and implementation
The aim of the project was to develop a monitoring framework with carer outcomes
measures to better enable the monitoring of outcomes for carers, including the timeliness of
services and support.
Due to the diversity of HACC and SCP providers, the proposed monitoring framework and
outcomes measures need to:
consider the diversity of service delivery organisations in terms of their size,
geographical location, available resources and capacity platforms
consider the range of service types and supports provided to carers and care
recipients
be developed from the perspective of consumers’ experience (i.e. carers and care
recipients), so that the measures reflect the types of outcomes that are most
important to carers and care recipients.
In the context of this project a carer was defined as a person who, through family
relationship or friendship, contributes to the care and support of an older person with care
needs or frailty, or someone with a disability or chronic illness (i.e. that is not a paid carer).
A care recipient was defined as a person being cared for by a carer.
The concept of ‘outcomes’ was defined as the experience of carers and care recipients of the
service and support in the short term (not over the longer term). This was inclusive of
process related outcomes, outcomes in relation to the care role, and quality of life
outcomes.
Timeliness was defined as the extent to which the service or support was provided at a time
that suited the individual situation and needs of carers and care recipients.
The project scope included four elements. As shown below, a targeted literature scan was
undertaken to inform the project context; selected consumer input was gained through
Alzheimer’s Australia Vic; consultation with service providers occurred through an online
survey; and a sample of feedback tools as currently used by service providers were analysed
to understand current practice.
Figure 1: Project elements
•Mapping of current approach
•Key opportunities
•Focus groups
•Key outcome domains
•Current practice
• Issues, opportunities
•Carer literature
•Outcome measures
• Instruments
Literature scan
Analysis of feedback
tools
Service provider
consultation
Consumer consultation
HDG Consulting Group 4
A Project Reference Group provided sector based advice and input, and included
representation from the following organisations.
Peak body – carers: Carers Victoria
Peak body – local government: Municipal Association of Victoria
Community service organisation: Baptcare
Department of Health: Central office and Gippsland region
Health service: Bendigo Health
Local government: Boroondara City Council
Multicultural organisation: Southern Migrant and Refugee Centre Inc.
The project was implemented by HDG Consulting Group between March and June 2014.
HDG Consulting Group 5
2 Context and literature
2.1 Outcome measures
Outcome measures are used for a wide variety of purposes. These include informing
strategies or supports, policy, program and guidelines, economic and resource decisions,
and continual quality improvement in service delivery. Within the context of this project the
focus is on the use of outcome measures to inform service delivery and enhance continual
quality improvement.
An important distinction is that outcome measures differ to performance measures that
typically compare the delivery of outputs by an organisation against targets or performance
indicators. In comparison, outcome measures generally provide a measure or scale against
which the results of an action, activity, plan, process, or program can be assessed for
comparison with the intended or desired outcomes.
Outcome measures may be generically applicable concepts (such as personal wellbeing or
quality of life) or specific to a particular sub group and/or items (such as carer stress in
relation to people with dementia).
In addition to the range of different uses, outcome measures can be used across different
levels - at the individual, organisational, or systemic levels. The World Health Organisation
International Classification of Functioning (ICF) Disability and Health2 provides a standard
language, framework and classification of health and health-related outcomes. In discussing
its applications, the ICF notes that outcomes can be considered at multiple levels:
Individual consumer level, for example: What are the outcomes of the treatment?
How useful were the strategies?
Organisational level, for example: How well do we service our clients? How can the
service be improved for better outcomes at a lower cost?
Social or systems level, for example: Can we measure this improvement and adjust
our policy accordingly?
In Measuring outcomes in community care: an exploratory study (Centre for Health Service
Development, 2010)3 the authors noted a distinction in outcomes measures as to the
element of time and the level at which outcomes might be achieved. For example, strategies
may have short, medium and long term results, and that outcome measurement can occur
at the individual level, provider (agency or organisational) level and program or system
level.
At the individual level, outcome measures can provide evidence about how effective care
or treatment is, for example through the use of clinical, psychological and functional
outcome measures. For example, in clinical services, measuring health outcomes is central
to assessing the quality of care.4
2 See http://www.who.int/classifications/icf/en/ 3 See http://www.adhc.nsw.gov.au/__data/assets/file/0005/241664/51_Measuring_outcomes_in_community_care_report.pdfhttp://www.adhc.nsw.gov.au/__data/assets/file/0005/241664/51_Measuring_outcomes_in_community_care_report.pdf 4 See http://www.qualitymeasures.ahrq.gov/tutorial/HealthOutcomeMeasure.aspx
HDG Consulting Group 6
Outcomes data is collected through the use of instruments designed to assess and measure
either a particular aspect or item or more global measures. For example, the Cleveland Scale
for Activities of Daily Living measures in detail specific activities of daily living in individuals
with dementia; whereas the Assessment of Quality of Life (AQoL) is a generic health-related
quality of life instrument which provides a profile relative to four dimensions of life.
At the organisational level, outcome measures can provide evidence about the
effectiveness of programs, services and supports. Data can be aggregated to a group or
service level to provide practice based evidence and generate information to support
continual quality improvement.
At the systemic level, outcomes measures can be used to assess the results of policies and
service delivery on specific cohorts of the broader population.
Within the context of this project and the development of a framework to better enable the
monitoring of outcomes for carers, including the timeliness of services and support, the
different perspectives and uses of outcome measures need to be acknowledged:
carers and care recipients can benefit from the use of outcomes to inform the
effectiveness and quality of service provision
practitioners can benefit from an outcomes approach, as part of assessment and to
inform support and care planning for individuals
managers can benefit from aggregated outcomes data to provide practice based
evidence to inform continual quality improvement
funding organisations can benefit from analysis of global aggregated outcomes to
inform policy and funding decisions.
2.2 Selected literature
While a comprehensive literature review was not within the project scope, the following
selected literature is provided to inform consideration of the development of a framework to
better enable the monitoring of outcomes for carers, including the timeliness of services and
support. Key international approaches are summarised as well as Australian approaches to
outcome measurement in other sectors.
a) Adult Social Care Outcomes Framework
The United Kingdom (UK) Department of Health, Adult Social Care Outcomes Framework
(ASCOF) launched in 2011/12, has been designed to focus on ‘measures that reflect the
outcomes which matter most to users of adult social care services and carers’ and is a tool
to support local improvement in care and support by the sector. ASCOF data were used to
analyse improvement in adult social care services across the following themes:
I am happy with the quality of my care and support and I know that the person
giving me care and support will treat me with dignity and respect
I am supported to maintain my independence for as long as possible
I understand how care and support works, and what my entitlements are
I am in control of my care and support
I feel safe and secure.
HDG Consulting Group 7
The 2014/15 updated ASCOF is designed to demonstrate improvement in adult social care
services and is based on a suite of four domains, associated outcomes measures and
technical definitions of the measures5:
Enhancing quality of life for people with care and support needs – with an
overarching measure of social care-related quality of life; and three specific outcome
measures (one of which is carer reported quality of life).
Delaying and reducing the need for care and support – with an overarching measure
that people who use social care and their carers are satisfied with their experience of
care and support services; and three specific outcome measures (one of which is
that carers feel that they are respected as equal partners throughout the care
process).
Ensuring that people have a positive experience of care and support – with an
overarching measure of social care-related quality of life; and three specific outcome
measures (one of which is when people develop care needs, the support they receive
takes place in the most appropriate setting, and enables them to regain their
independence).
Safeguarding adults whose circumstances make them vulnerable and protecting
them from avoidable harm – with an overarching measure of social care-related
quality of life; and one specific outcome measure.
Prior to inclusion in the framework, outcome measures were assessed against the following
principles.
Relevant and meaningful to the public – measures should be intelligible and reflect
what matters to people.
Influenced by social care – measures must be relevant to councils’ adult social care
functions, including effective joint working with local partners.
Can be compared between local areas over time (with the exception of national-only
measures) – measures must be consistent to promote transparency.
A measure of social care related outcomes, or consistent with an outcomes focus.
A robust measure – data used to populate the measure are statistically robust and
the measure does not create perverse incentives.
Supported by evidence – evidence exists that suggests there are cost effective
strategies that would make a positive change.
Disaggregable by equalities – measures should be able to be broken down to support
a focus on equalities.
New requirements – where there are new requirements, these will be estimated and
funded by the UK Department of Health.
5 To be published
HDG Consulting Group 8
National measures – must meet all the above criteria, as well as: local authorities
have local (or regional) level information available against which to compare
themselves to the national picture; the measure would help inform national policy
development; and, there is consensus that the outcome is sufficiently significant that
its omission from the framework on the grounds of a lack of local-level data is not
justifiable.6
b) Personal Outcomes Approach
The Scottish Community Care Outcomes Framework (2008)7 includes four national
outcomes, being: improved health, improved wellbeing, improved social inclusion and
improved independence and responsibility. Associated with these are 16 performance
measures that cover people’s satisfaction with services, waiting times, quality of
assessment, shifting the balance of care, carers' well-being, unscheduled care and
identifying 'people at risk'. In 2012, a review of the framework8 and its effectiveness
recommended the development and use of a single suite of outcomes and indicators, and
continued use of the Personal Outcomes Approach.
The Personal Outcomes Approach defines outcomes as ‘what matters to people using the
service, as well as the result or end impact of activities.’ Personal outcomes are identified
through conversations within assessment and care planning processes to ensure that people
using services and their carers are at the centre of the support and that service providers
understand what outcomes are most important to them. Service supports and activities are
then planned and delivered to progress towards the achievement of the outcomes at the
individual level. This appears similar to the person-centred approach adopted through the
ASM in Victoria, and the identification of goals specific to the person. Goals are recorded in
the person’s support plan so that service provision can be reviewed within the context of
whether there is progress towards these. In addition, organisations can use this information
to determine effectiveness at the organisational level.
Personal outcome approaches suggest that while outcomes can be defined as the result of
the activity or support services, personal outcomes should be defined as ‘what matters to
the person’. This is comprised of the combination of elements – inputs, processes and
outputs – that together result in an outcome for the service user. The three key elements of
this approach are described as:
engagement about what the service user wishes to achieve (where the person wants
to get to, what resources and skills they can apply and how the service provider can
assist)
recording of information about the service user’s desired outcomes
use of information to inform decision making and service improvement within
organisations.
6 See https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/263783/adult_social_care_framework.pdf 7 Community Care Outcomes Framework. review of the Framework 2006 – 2011. Joint Improvement Team April 2012. http://www.jitscotland.org.uk/about/community-care-outcomes-framework/ 8 Talking Points. Personal Outcomes Approach. Practical Guide. Joint Improvement Team 2012 http://www.jitscotland.org.uk/action-areas/talking-points-user-and-carer-involvement/
HDG Consulting Group 9
The Personal Outcomes and Outcomes Framework approach classifies outcomes that are
important to people into three broad categories:
Quality of life outcomes – that relate to aspects of a person’s life that they are
working to achieve or maintain e.g. feeling safe, maintaining health and wellbeing.
Process outcomes – that relate to the experience that people have in seeking,
obtaining and using services and supports e.g. being listened to, service reliability.
Change outcomes – that relate to improvements in physical, mental or emotional
functioning that people are seeking from any particular service strategy or support
e.g. improved skills, reduced symptoms.
In addition to these three categories, for unpaid carers four additional categories of
outcomes were identified as important:
quality of life for the care recipient
quality of life for the carer
managing the care role e.g. feeling informed, making choices
process outcomes e.g. accessible services, service flexibility, respect.
c) Mental health services
Victoria’s mental health services define outcomes measures as ‘the process of measuring
health outcomes of consumers of mental health services by undertaking a series of ratings
and comparing the results over time.’ The stated purpose of measuring outcomes is to
determine whether services are effective and mental health service users experience
improved mental health as a result of the services they receive. The routine use of outcomes
measures is considered essential to clinical practice and empowering consumers by
participation in decision-making about their treatment and care. A national set of measures
has been implemented (following field testing and development of a collection protocol),
with de-identified ratings submitted by each state and territory to the Commonwealth
Government. However, in adult and aged persons mental health services there is no carer
questionnaire because no suitable instrument has been identified that can collect carer
feedback within the context of outcome measurement.9
d) Disability services
A literature review conducted in 2011 on outcomes measures in disability services defined
outcomes as: ‘the impact of the service on the status of individuals or a group.’ The report
noted that there were simply too many existing outcome measurement tools to review, and
therefore, focussed on those within set categories:
major conceptual or classification tools (such as the ICF)
therapy and psychology tools (such as the Australian Therapy Outcomes Measures
and the Personal Wellbeing Index)
life outcome tools for adults with disabilities receiving services and supports (such as
the Personal Outcomes Measures), or goal setting tools (such as the Goal
Achievement Scale).
9 See http://www.health.vic.gov.au/mentalhealth/outcomes/about.htm
HDG Consulting Group 10
The authors concluded that capturing outcomes is a complex process, that each outcomes
tool has particular strengths and limitations, and that most do not include information in
relation to service improvement (although they may not be designed to do so and service
improvement feedback can be collected through other methods).
The authors concluded that no current outcomes tool suited the purpose of outcome
measurement as being sought by the commissioning organisation and thus proceeded to
develop its own tool, titled the Measurement of Services and Supports.10
e) Community care
In NSW the Centre for Health Service Development, Measuring outcomes in community
care: an exploratory study (2010)11 included a review of best practice measuring outcomes.
The authors note that after examining the international literature, the measurement of
outcomes is easier where there are well-defined strategies and ways of defining episodes of
care so that standardised measures are repeatable at different points in time. The report
concludes that although measuring outcomes in community care is complex and technical,
the shift to an outcomes (rather than outputs) focus is essential for quality measurement
and understanding how well services are contributing to the outcomes desired by service
users. It suggests that carefully selected and standardised data items should be collected as
part of normal practice at the client level to monitor change over time. The report notes that
the design and implementation of an outcomes focussed system is a complex undertaking
and requires a long timeframe.
f) Local Government Performance Reporting Framework
The Draft Local Government Performance Reporting Framework & Indicators Working Paper
(Victorian Department of Planning and Community Development, 2013)12 defines and lists a
series of measures for services provided by local government in Victoria. The guide outlines
draft indicators to underpin comprehensive performance information and to support
strategic decision making and continuous improvement. Indicators have been developed
across three thematic areas: service performance, financial performance and sustainability;
and are based on SMART13 principles. Indicators reflect the dimensions of appropriateness,
quality, cost and service effectiveness.
The current, updated proposed measures for HACC services are:
time taken to commence the HACC service – defined as the number of days it has
taken for a new client to commence the domestic, personal or respite care service
compliance with Community Care Common Standards (CCCS) – defined as the
percentage of CCCS expected outcomes met
cost of domestic care service – defined as the direct cost to council of the domestic
care service per hour of service delivered
10 See http://www.scopevic.org.au/index.php/cms/frontend/resource/id/1425/name/moss_lit_review.pdf 11 See http://www.adhc.nsw.gov.au/__data/assets/file/0005/241664/51_Measuring_outcomes_in_community_care_report.pdf 12 Suggested for inclusion in the literature scan by a member of the Project Reference Group. 13 SMART: specific to the service, activity and/or dimension of performance being measured; based on measurable factors that can be observed, documented and verified (either by audits or other mechanisms); aligned with objectives; relevant to the service, activity and/or dimension of performance being measured; and time-bound with an appropriate and clear timeframe specified for measuring performance.
HDG Consulting Group 11
cost of personal care service – defined as the direct cost to council of the personal
care service per hour of service delivered
cost of respite care service – defined as the direct cost to council of the respite care
service per hour of service delivered
participation in the service – defined as the percentage of the municipal target
population that receive a domestic, personal or respite care service
participation in the service by culturally and linguistically diverse people that receive
a domestic, personal or respite care service – defined as the percentage of the
municipal target culturally and linguistically diverse population that receive a
domestic, personal or respite care service.
g) Commonwealth Home Support Programme
The recently released Key Directions for the Commonwealth Home Support Programme,
basic support for older people living at home (May 2014)14 groups current HACC service
types within a smaller number of service groups or outcome streams, that reflect the
outcome that each service is intended to contribute towards. These outcomes are listed as:
increased independence, nutrition, social participation, assistance at home, access to the
community and care relationships.
2.3 Instruments
In the Australian context, the health and wellbeing of carers has been investigated in detail.
Numerous studies have investigated different factors using a range of methodological
approaches and modes. For example, an Australian literature review (2004) noted that
increased carer strain appears to be more predominantly associated with care recipients
factors (such as dependence, cognitive impairment, incontinence, sleep disturbance) rather
than carer or care situation factors; other studies suggest that respite services need to
directly address the psychological effect of caring by providing psychological services that
address depression, anxiety, and stress associated with the caring process, and to
encourage carers to maintain or improve their levels of social participation.15 Some studies
suggest that respite is a key strategy to support carers to sustain their carer role; and
respite has been reported as improving the quality of life.16 However, results from evaluative
research and systematic reviews of the literature on carer support programs have been
inconclusive in relation to reducing carer strain (but may assist a carer to feel better or more
capable of sustaining their care role) and the results cannot be easily reviewed and
compared because of the different methods being used.17 This highlights the challenge of
developing a broad outcomes framework that can be used in conjunction with, or
independently of, validated instruments and scales used to measure change of a single
variable (e.g. carer strain) at an individual level.
14 See http://www.dss.gov.au/sites/default/files/documents/05_2014/final_key_directions_for_the_commonwealth_home_support_programme_discusison_paper.pdf 15 Cited in Tang J, Ryburn B, Doyle C, Wells Y 2011, ‘The Psychology of Respite Care for People with Dementia in Australia’, Australian Psychologist, vol. 46 pp.183-189. 16 Evans D 2011, ‘Exploring the concept of respite,’ Journal of Advanced Nursing pp. 1-11 17 Eagar K et al. (2007) Effective Caring: a synthesis of the international evidence on carer needs and interventions. Centre for Health Service Development, University of Wollongong.
HDG Consulting Group 12
Instruments such as surveys and questionnaires are the method by which the data
underpinning outcomes measures are collected and recorded. It is generally recognised that
the properties of the outcome measures within these are important. These include reliability
(i.e. the degree to which the results remains unchanged on test and retest) and validity (i.e.
the degree to which a measure assesses what it is intended to measure); and in some cases
responsiveness (i.e. the ability of a measure to detect change over time).
A full analysis of existing instruments was not within the scope of this project, however,
selected examples were considered in order to inform the project. These are summarised
below.
The Dementia Outcome Measurement Suite, a project commissioned under the Australian
Government's National Dementia Initiative, was designed to develop a standard suite of
instruments that would be promulgated throughout Australia to encourage clinicians to 'talk
the same language' by using the same instruments as much as possible. The outcome
measures are stated as important for monitoring progress of the dementia, for evaluating
the effects of strategies and for service planning.18
The Wellbeing of Australians – Carer Health and Wellbeing (Deakin University 2007) study
used three major outcome measures, being the Personal Wellbeing Index which scores the
level of satisfaction across seven aspects of personal life – health, personal relationships,
safety, standard of living, achieving in life, community connectedness, and future security;
and two sub-scales taken from the Depression, Anxiety and Stress Scale (Lovibond and
Lovibond, 1995) to measures outcomes and present the average score for each item
between comparative groups.
Health economics research into supporting carers of people with dementia: A systematic
review of outcome measures (2012) study into supporting carers of people with dementia
undertook a systematic review of outcome measures.19 The authors identified and extracted
a list of over 228 separate instruments that included carer outcome measures, and then
categorised them as a measure of:
carer burden – 44 measures consisting of responsibility, stress and strain
mastery – 43 measures encompassing the family carer’s coping, self-efficacy and
competence
mood – 61 mood measures covering anxiety, depression, sleep and general mental
well-being
quality of life – 32 measures
social support and relationships – 27 measures
staff competency and morale – 21 measures.
Each of these instruments was designed to screen, assess and measure one or more
domains and the associated specific items and/or strategies.
This is indicative of the high volume of existing instruments that are designed to be used by
clinicians or practitioners to screen or assess carer needs and inform the subsequent
strategies.
18 See http://www.dementia-assessment.com.au/ 19 See http://www.hqlo.com/content/10/1/142
HDG Consulting Group 13
A sample of these were reviewed to ascertain if they were relevant to this project and could
provide a simple, uniform set of outcome measures applicable within the project context.
Table 1: Examples of instruments to measure carer variables
Title Summary Description Project relevance*
Modified Caregiver Strain Index (MCSI)20
13 question self-administered tool to measure carer strain
Validated tool. Single focus on carer strain. Questions are deficit based i.e. focus on negative aspects of care (e.g. I feel completely overwhelmed, yes/sometimes/no).
Specific to a single domain rather than a broad set of carer outcome measures. Not applicable in the context of this project.
Zarit Burden Interview21
22 question self-reported tool to assess carer burden
Validated tool. Single focus on carer burden. Questions are deficit based i.e. focus on negative aspects of care (e.g. Do you feel that your social life has suffered because you are
caring for your relative?) Five point scale with scoring.
Specific to a single domain rather than a broad set of carer outcome measures. Not applicable in the
context of this project.
Adult Carer Quality of Life Questionnaire (AC-QoL)22
40 item self-reported instrument to measure quality of life
Has eight domains (and associated subscales): support for caring; caring choice; caring stress; money matters; personal growth; sense of value; ability to care; and carer satisfaction. Questions are negatively or positively worded. Higher score indicates greater quality of life.
Covers multiple domains and can be used as a pre and post measure. Does not include satisfaction with services, timeliness and other items required in the context of this project.
CarerQol instrument23
Measures and values the effect of informal care
Validated tool. CarerQol measures well-being (CarerQol-VAS) and subjective burden (CarerQol-7D). Well-being is measured in terms of happiness using an analogue scale with endpoints 'completely unhappy’ (0) and
'completely happy’ (10). Subjective burden is measured on seven dimensions (CarerQol-7D): fulfillment, relational problems, mental health, daily activities problems, physical health and support against a scale of no / some / a lot of.
Covers multiple domains and measures the overall effect of informal care. Does not include satisfaction with services,
timeliness and other items required in the context of this project.
* Project relevance - Is the tool or individual questions applicable to the project requirements i.e. does it provide uniform
outcome measures across the domains of specific interest to this project; and/or questions that can be used across multiple
service provider organisations and service types, in relation to carer outcomes?
As noted in literature24, there is the need in the longer term for ongoing or standardised
data collection based on a carer outcomes measurement suite, to allow continuous
evaluation, quality management, and comparison between services; and supported through
a suite of formally endorsed resources for measures of activities, change and outcomes.
However, none of the examples reviewed were found to suit the purposes of this project.
The majority of instruments tend to reflect a medical or clinical model to assess functional
and psychological outcomes, or are specific to a single domain rather than a broad set of
carer outcome measures. This approach differs markedly from the Personal Outcomes
approach described in the literature, which is similar to the HACC Active Service Model
approach in Victoria (that uses a person centred approach to engage the consumer in a
conversation about their desired outcomes) and the SCP approach of supporting health and
wellbeing.
20 See http://consultgerirn.org/uploads/File/trythis/try_this_14.pdf 21 See http://www.rgpc.ca/best/GiiC%20Resources/GiiC/pdfs/3%20Caregiver%20Support%20-%20The%20Zarit%20Burden%20Interview.pdf 22 See http://professionals.carers.org/health/articles/carer-quality-of-life-tool-launches,7010,PR.html 23 See http://www.hqlo.com/content/11/1/173 24 Eagar K et al. (2007) Effective Caring: a synthesis of the international evidence on carer needs and interventions. Centre for
Health Service Development, University of Wollongong.
HDG Consulting Group 14
Therefore, adoption of a single, existing tool was deemed not applicable in the context of
this project, given the requirement for the tool to be simple, applicable across a range of
organisation and program types, be able to be practitioner or service user completed,
include one or more measures of timeliness in accordance with the VAGO recommendation,
and include the mix of domains important to carers. However, based on the literature scan,
and with a focus on outcome measures and timeliness, a list of items was developed for use
in the consumer consultation component of the project.
HDG Consulting Group 15
3 Consumer consultation
3.1 Approach
Targeted consumer input was gained through direct consultation with people with dementia
and their carers through Alzheimer’s Australia Vic:
Younger Onset Dementia Group
Consumer Advisory Committee.
Based on the literature scan, a series of potential questions to generate outcome measures
were drafted. The first group discussed the draft questions from a consumer perspective
and suggested a range of changes. The questions were then revised and presented to the
second group who then discussed the revised questions and provided comments and
recommendations in relation to the question wording, response scale, and application.
3.2 Consultation
The consultation commenced with general discussion in relation to service users’
experiences and their views about what they considered important when considering
outcomes from a consumer perspective. This resulted in participants sharing their stories
and experiences with the service system and how these experiences could be viewed as
opportunities for service improvements. The consultation then focused more specifically on
the types of question that could be used to measure outcomes.
Members of the first group, comprising 15 people with younger onset dementia and their
carers, raised a diverse range of matters as part of the broad context to inform the more
specific discussion. Key points included the following:
there needs to be greater community awareness in relation to younger onset
dementia
the process of diagnosis is ‘convoluted and stressful’
a diagnosis of dementia should be an automatic trigger for an ACAS assessment ,
however, this service needs to be defined differently for younger people
all the information can be overwhelming – well designed info-graphics could assist
people to identify their stage on the care pathway and the types of services or
supports to assist
it was difficult to achieve a coordinated or seamless approach because funding for
support appeared to be from a wide range of sources
there are many different forms to be completed and this is time consuming
there are numerous interviews and applications and it is unclear as to what may be
offered as a result of each
the system of supports is not well coordinated and it is difficult to find activity and
support groups, and respite options suitable for younger people
people with younger onset dementia have different issues and there appear to be
fewer services for this group – this group desire meaningful activities with a
rehabilitative focus
HDG Consulting Group 16
more flexible funding is needed to allow people to self-select and purchase their
desired supports.
The handout (see below) as developed following the literature scan, formed the basis for
the second part of the discussion. Participants made comments and suggestions in relation
to the draft questions.
In recognition of the diversity of service users, service types and organisations, it was noted
that a standard set of outcome measures would need to be generically applicable, relatively
simple, limited to one page (if possible), include a ‘not applicable’ box, and allow space for
the respondent to provide additional comments against each question.
Participants suggested that the questions should incorporate concepts such as:
accessing information from a variety of sources
knowing what services may be available
effective communication / listening to what the consumer says
understanding needs / diversity
getting the support you need / receiving support during the care journey
meeting the needs of the person with dementia
person and staff – good match / were they suitable
suitability of the service for the carer.
Figure 2: Consumer consultation handout version 1
HDG Consulting Group 17
Participants also noted that it was important for:
measures and results at an agency level to be able to be collated to form an industry
wide perspective
the approach to be outcome focussed, rather than output based, which appears to
be the current practice.
Following feedback, the handout and draft questions were revised for consideration by the
second group.
Consultation with the second group, comprising nine carers of people with dementia,
followed a similar process. The group commenced with general discussion in relation to
service users’ experiences and their views about what they considered important when
considering outcomes from a consumer perspective, followed by more specific discussion
about the handout (see below).
Similar to the previous group, participants raised a diverse range of matters as part of the
broad context to inform the more specific discussion:
issues in relation to information and access to services
the need for proactive contact, service navigation and advice
issues in relation to fragmented care and the need for a more holistic approach
need for more GPs trained in aged care and dementia
that family carers are not trained to cope with behaviours
variability in the quality and volume of services
use of the Dementia Behaviour Management Advisory Service to provide behaviour
management strategies and techniques
emergency planning for carers (i.e. if something happens to the carer)
need for more in-home respite
need for support in transition to residential care
need for training that is dementia and delirium specific for staff in residential care
(low care)
need for family support post residential care placement
that the disability sector is more advanced in terms of offering choices and clients
directing their own care.
The handout was provided and participants made comments and suggestions in relation to
the draft questions. Key points were to:
keep it simple, as carers are time poor
have a single page if possible, so people can see the questions ‘at a glance’
adjust the scale to be never / some of the time / most of the time (or a lot of the
time) / always; and the options for unsure or not applicable
to include:
o the name of the service
o the type of service or support received
o space for comments to further explain the context for the response
o date and return instructions.
HDG Consulting Group 18
Figure 3: Consumer consultation handout version 2
Following the meeting, additional email feedback from a consumer was that the
questionnaire looked simple and straightforward to complete; with the request to add an
optional comments section so that respondents could qualify their comments with additional
detail if they wished to.
The feedback from the consultation underpinned the development of questions in relation to
current practice, to be used for consultation with service providers through a service
provider survey.
HDG Consulting Group 19
4 Service provider consultation
4.1 Survey development
In accordance with the project brief, consultation with service providers occurred primarily
though the design and administration of a survey with HACC and SCP providers. The survey
questions were informed by the literature scan and consumer consultation, and refined
though feedback from HACC regional contacts, SCP contacts and relevant department staff.
The survey was tested by members of the project Reference Group prior to finalisation.
The survey included a mix of multiple choice questions and free text questions in relation to
carer supports, and explored current practice in relation to three types of outcome
measures:
process outcomes, that reflect the experience of the person in seeking, obtaining
and using services and supports
supporting the care role outcomes, that explore the capacity, confidence and
sustainability of the care relationship in relation to the services provided
quality of life outcomes, that relate to how satisfied the person is with aspects of
their life and what matters to them.
Timeliness was defined in the survey as the extent to which the service or support was
provided at a time that suited the individual situation and the needs of carers and care
recipients (not whether it was provided in accordance within a set number of days).
The survey was administered using an online survey tool and distributed by the Department
of Health to organisations that receive funding through HACC (Respite and Planned Activity
Groups) and the SCP. Targeted follow up occurred with specific stakeholder groups.
A summary of the survey results is included below, with a more detailed analysis including
comparison between HACC and SCP providers, included in Appendix 1.
4.2 Sample size and organisation type
A sample size of 129 survey respondents was sought to provide a confidence level of 95 per
cent with a confidence interval of +/-7.25 While not a statistically based project, this number
of responses was seen as preferable to enable confidence in the results. A total of 163
surveys were received, however, following removal of incomplete and or duplicated surveys
a total of 149 survey responses were used for the analysis. The number of responses thus
met the desired sample size.
The respondent organisations represented 117 HACC (Respite and Planned Activity Groups)
service providers and 32 SCP service providers. This is equivalent to 36 per cent of all HACC
(Respite and Planned Activity Groups) providers and 65 per cent of all SCP providers in
Victoria.
25 Calculation based on a total of 323 HACC Respite and PAG and 49 SCP providers.
HDG Consulting Group 20
Organisation types included a mix of local government, health services or community health
services, community service or welfare organisations, Aboriginal Community Controlled
Organisations (ACCOs) and ethno-specific or multi cultural organisations.
Table 2: Respondent organisation type
Organisation type Program
Total Per cent HACC SCP
Aboriginal Community Controlled Organisations (ACCO) 4 0 4 2
Ethno-specific or multi cultural organisations 11 2 13 9
Community service organisations 30 10 40 27
Health services / community health services 32 12 44 29
Local government 40 8 48 33
Total 117 32 149 100
4.3 Type of support delivered
Service providers were asked to indicate the type of support options provided26, whether the
support was provided on a planned and/or emergency basis, and whether it was provided
for the carer, care recipient, or both together.
Analysis of survey results indicated the following:
advice and information (planned) is the most common service option for both HACC
and SCP
community based outings and activities in a group setting are the second most
common service option for both HACC and SCP
support or brokerage funds is the least offered service option by HACC
residential respite overnight in the organisation's own care facilities is the least
offered service option by SCP.
For HACC services the focus of the support was primarily the care recipient (45 per carer
recipient, 29 per cent carer, 26 per cent both together) whereas for the SCP the focus of the
supports was the carer and care recipient equally (38 per cent care recipient, 38 per cent
carer, 24 per cent both together). Overall, both HACC and SCP survey respondents offered
around one quarter of the listed support options to the carer and care recipient both
together. However, it should also be noted that the differentiation between the carer and
care recipient as used in the survey questions is somewhat arbitrary, as it is reasonable to
assume that the provision of services to either person has a benefit to both people.
26 Multiple choice list: advice and information; counseling; in-home, daytime, individual respite; in-home, overnight, individual respite; community based, individual respite; community based, group respite (e.g. outings and activities); centre based, group respite; community based, overnight respite; community based, overnight respite in the respondent organisation’s facility; support or brokerage funds (e.g. vouchers, aids, equipment, co-payments for residential respite).
HDG Consulting Group 21
4.4 Current practice – carer satisfaction
Service providers were asked about their current practices in measuring satisfaction for the
services and support options provided. The chart below shows the feedback method as a
proportion of the total responses (i.e. out of all methods used and including multiple
methods per single organisation). As shown, the triad of satisfaction surveys, verbal
feedback (informal) and goal achievement/care planning reviews are similar proportionally
at around one quarter each of all responses. This finding supports the concept that
organisations tend to use a combination of approaches to measure satisfaction.
Figure 4: Feedback methods
Further analysis indicates the:
most commonly used method as verbal feedback gained informally through
ongoing service provision and typically used in conjunction with other methods (91
per cent with carers, and 86 per cent with care recipients)
frequent use of gaining feedback through service reviews as part of ongoing
service delivery and goal achievement (68 per cent with carers, and 64 per cent with
care recipients)
use of annual satisfaction surveys by around half of respondents (55 per cent
with carers, and 48 per cent with care recipients)
use of feedback at service exit by around half (55 per cent)
least use of focus groups (13 per cent carers, and 9 per cent with care recipients)
range of other practices such as feedback at events, suggestion boxes, online
comments and so forth.
In relation to the best method to generate information for service improvement, the
majority (around 70 per cent) of respondents noted that verbal feedback and information
gathered through informal conversation was the most useful; and that a combination of
multiple (rather than single) methods was beneficial.
These survey results indicate that in developing an outcomes framework, an important
consideration is the ability to collect the data through a range of methods - formally through
survey type processes, as well as informally through discussion embedded within ongoing
service delivery practice.
Satisfaction
survey 23%
Focus groups
9%
Verbal
feedback (informal)
26%
Goal
achievement, care
planning, reviews 27%
Service exit
8% Other
7%
Feedback methods
HDG Consulting Group 22
4.5 Current practice - process outcomes
Respondents were asked to identify which process27 related items they currently collect or
measure, and how. Across all items and response methods (including multiple methods per
single organisation), the most commonly used method was ‘other feedback’ (e.g.
verbal/discussion), followed by the use of carer satisfaction surveys.
Figure 5: Feedback methods – process related items
It is interesting to note that the collection method varied across the different items28. For
example:
carer satisfaction surveys were more likely to include the items of communication
and respect and information provision and less likely to include suitable staff
or timely access
service system capacity and the positive experience were more likely to be
collected through other feedback (e.g. verbal/discussion) and slightly less likely to
be collected through a survey process
timely access was the item least collected through a survey or outcomes tool
process and second least collected through other feedback (e.g. verbal/discussion)
processes (14 per cent of respondents do not currently collect feedback in relation
to timely access).
Some respondents (21 per cent) indicated that while they did not currently collect feedback
about a particular item, they would do so in future.
These findings highlight the range of practices across organisations, both in relation to the
questions asked and the methods and frequency of collection. The results support the need for
the development and use of a standardised outcomes framework and questions.
27 Process related outcomes were defined as outcomes that reflect the experience that individuals have in seeking, obtaining and using services and supports. For list of items see Appendix 1. 28 For detailed analysis see Appendix 1.
Other
feedback 39%
Carer
satisfaction survey
34%
Will collect in
future 13%
Carer
outcomes tool 9%
Do not collect
5%
Methods - Process outcomes
HDG Consulting Group 23
4.6 Current practice - care relationship outcomes
Respondents were asked to identify which care relationship29 related items they currently
collect or measure, and how. Across all items and response methods (including multiple
methods per single organisation), the most commonly used method was ‘other feedback’
(e.g. verbal/discussion), followed by ‘will collect in future’.
Figure 6: Feedback methods – care relationship items
In general these care relationship items are less frequently collected in comparison to the
process measures discussed above, particularly through more formal processes such as
satisfaction surveys.
However, analysis indicates the range of practices:
other feedback (e.g. verbal/discussion) processes were used by up to half of
respondents pending the particular item
the item stress tends to be informally monitored30
carer satisfaction surveys were more likely to be used in relation to the items of
effectiveness and less likely to include partnership or skills and confidence
the use of carer outcomes tools was least common
the least collected item collected via a carer outcomes tool was sustaining the
care relationship and effectiveness
the item skills and confidence was least considered by any method.
These findings highlight the limited use of carer outcomes tools in current practice, and the
high level of reliance on informal methods. In addition, few organisations currently collect
information about sustaining the care relationship and effectiveness in supporting
such. There appears to be a level of preparedness to collect these items with around one
quarter of respondents indicating that that they will do so in future.
29 These outcomes relate to supporting the care role and explore the capacity, confidence and sustainability of the care
relationship, in relation to the services provided. For list of items see Appendix 1. 30 Three services reported using a Carer Stress Index to assess the carer and/or inform priority of access.
Other
feedback 38%
Will collect in
future 19%
Do not collect
19%
Carer
satisfaction survey
18%
Carer
outcomes tool 6%
Methods - Care relationship items
HDG Consulting Group 24
4.7 Current practice - quality of life outcomes
Respondents were asked to identify which quality of life31 related items they currently collect
or measure, and how. Across all items and response methods (including multiple methods
per single organisation), ‘other feedback’ (e.g. verbal/discussion) was the method that
accounted for the largest proportion of responses, followed by ‘do not collect’.
Figure 7: Feedback methods – quality of life items
Similar to the previous sections, the most commonly used method is through other
feedback, such as informal discussion that occurs during service delivery.
However, analysis indicates the range of practices:
other feedback (e.g. verbal/discussion) is most likely to consider social contact and
least likely to consider satisfaction with care role
carer satisfaction surveys were more likely to be used in relation to the items of
meaningful activities and less likely to include satisfaction with care role
the use of carer outcomes tools was least common
the most collected item through carer outcomes tool was social contact and the
least collected item was staying well.
Similar to the previous section (care relationship measures) some respondents (up to 32 per
cent) indicated that while they did not currently collect feedback about a particular item, they
would do so in future.
These findings indicate that quality of life measures are collected slightly more than care
relationship measures and slightly less than process measures.
While up to half of respondents collected information about some items through informal
measures, they are less likely to be measured though formal methods, and there is little use
of carer outcomes tools in current practice. The provision of a uniform outcomes framework
and measures would assist organisations to collect quality of life information in a consistent
manner to inform service improvement.
31 Quality of life outcomes relate to how satisfied the person is with aspects of their life and what matters to them. For list of items see Appendix 1.
Other
feedback 38%
Do not collect
20%
Carer
satisfaction survey
17%
Will collect in
future 15%
Carer
outcomes tool 10%
Methods - Quality of life items
HDG Consulting Group 25
4.8 Most important items
Respondents were asked to identify the four most important outcomes to collect. The free
text response boxes allowed respondents to enter multiple concepts into each response
field.
Based on a content analysis, and in descending order:
the overall most important item was considered to be whether the service was
effective and assisted to meet the person’s needs and goals - this included
whether the service met their needs and expectations and was responsive and timely
in doing so
measures in relation to quality of life with a focus on carer health and wellbeing,
inclusive of carer stress
a cluster of items categorised under the theme of communication and respect –
this included respect for care relationship, respect for the carer’s knowledge,
effective communication and the provision of information about the service system
measures in relation to care roles and relationships, such as whether the person
has been assisted in their care role, and to maintain and sustain the care relationship
items to do with service quality and satisfaction
other supports and links such as whether the carer was using other services and
social connections
meaningful activities for the carer and/or care recipient.
Respondents were also asked to suggest other outcomes measures that they considered
important to collect, for example to meet the requirements of the Victorian Carers
Recognition Act. Responses tended to reflect the items above and included some additional
items:
sustaining the care relationship, stress, and the ability to take a break (26 per cent of
respondents)
quality of life, wellbeing, and social connections (24 per cent of respondents)
diversity related measures, respect for cultural identity and cultural responsiveness
(19 per cent of respondents)
other suggestions included skills, consumer directed care plans, the number of
referrals made, confidence in services and staff, whether they had a positive
experience and activities were meaningful and so forth.
These findings indicate that respondents consider the most important carer outcomes to
collect relate to: effectiveness in responding to needs, carer health and wellbeing,
communication and respect, and care roles and relationships.
The inclusion of these items in a standardised outcomes framework and questions would be
of benefit to service providers to assist in their understanding of the effectiveness of service
provision and to inform continual quality improvement.
HDG Consulting Group 26
4.9 Requirements of the Victorian Carers Recognition Act
The survey included a question about what could assist the organisation to meet the
requirements of the Victorian Carers Recognition Act. The most common suggestion
accounting for around one third of all suggestions, was for further promotion of the Act to
the general public, to senior management and to staff; the provision of information
resources such as fact sheets, checklists, best practice examples; and associated training.
The development of uniform outcomes tools and measures accounted for another 36 per
cent of suggestions, for example:
‘A standardised set of questions/questionnaire developed by the Department so that
consistent data can be gathered across agencies.’
‘Universal tools for evaluation i.e. wellbeing survey for all SCP funded agencies.’
‘Set forms to complete - it is easier to obtain information from carers / clients if the
form is a requirement.’
Around 12 per cent of respondents identified the need for financial resources and/or more
flexible funding; while around six per cent reported that assistance was not needed and that
their organisation was meeting the requirements of the Act.
These findings indicate that a generic outcomes tool would enhance the capacity of some
organisations to meet the requirements of the Act.
4.10 Barriers, enablers and suggestions
Barriers
Almost half of respondents identified time and resources as a key barrier - for the service
provider organisation and the carer, who was often time poor and experiencing stress. The
next most frequently identified barrier was carer reluctance to reveal personal information or
provide feedback. Likewise, literacy and language were identified as barriers. Poor quality
feedback due to poorly designed questions, the inability to find appropriate feedback tools,
or feedback tools that were not easy to use by staff or carers were also identified as
barriers.
Enablers
The most frequently identified enabler was organisational capacity in terms of a supportive
policy and the associated processes, information technology systems and feedback tools.
This was followed by staff skills and the development of a trusting relationship with the
carer. Adequate time and resources were also identified as enablers.
HDG Consulting Group 27
Suggestions
The most common suggestion for how the service system and organisations could improve
the collection and measurement of carer outcomes was the development and
implementation of a standardised tool/survey for use by funded organisations (to access via
the Department’s website to enable organisations to download and use as needed, and with
the flexibility for organisations to add their own) and/or an increased focus on the
measurement of outcomes, including the reporting of such.
Comments of this nature accounted for just over half of all suggestions. For example:
‘Develop a universal carer satisfaction survey to be used by all HACC providers
thereby enabling data to be collated and compared accurately across regions. This
would also reduce the need for agencies to develop their own templates.’
‘Would be good to have a standardised survey tool that all services use. Great for
benchmarking and would save everyone re-inventing the wheel and spending time
reviewing survey tools as we are currently doing.’
‘A standardised set of questions/questionnaire developed by the Department so that
consistent data can be gathered across agencies. As an organisation that delivers a
range of services to diverse communities this data would be very useful as a broader
measure.’
Around one third of responses reflected the ongoing (and improved) use of a mix of survey
and less formal verbal/discussion based approaches within their organisations.
4.11 Summary of survey findings
Methods
The most commonly used method for carer satisfaction feedback is verbal feedback. More
formal approaches such as satisfaction surveys tend to be used annually by around half of
the service providers and goal achievement is monitored annually by around one quarter of
the service providers. Focus groups with carers were the least used method.
Most service providers use a range of methods and do no not rely on any one single method
alone. Therefore a key learning in relation to outcome measures is the importance of being
able to collect outcomes data using various methods (to suit the person / situation). For
example, both formally through survey type processes as well as through discussion as part
of ongoing service delivery; ensuring that both approaches result in recorded data.
In relation to collecting feedback for service improvement purposes, the majority of
respondents reported their experience that verbal feedback and information gathered
through informal conversations was the most useful information.
HDG Consulting Group 28
Process outcomes
There is an inconsistent approach to the collection of process related information – both
between organisations and between items. Measures such as information provision or
communication and respect were more likely to be collected in carer surveys; while items
such as service system capacity or whether the experience had been positive, were more
likely to be collected through verbal feedback. Timely access was the item least collected
through a survey or outcomes tool process and second least collected through other
feedback processes (e.g. verbal/discussion).
Supporting the care relationship
There is an inconsistent approach to the collection of this information – both between
organisations and between items. However, in comparison to process related measures,
items about supporting the care relationship were less likely to be collected, and where they
are collected, less formal methods tend to be used. The item ‘stress’ was more likely to be
included on a carer outcomes tool.
Quality of life
There is an inconsistent approach to the collection of this information – both between
organisations and between items. Quality of life measures are collected slightly more than
care relationship measures and slightly less than process measures. While up to half of
respondents collected information about some items through informal measures, they are
less likely to be measured though formal methods. The item social contact was most
collected and satisfaction with care role least considered.
Important outcomes
Survey findings indicate the items considered to be most important to collect as:
service effectiveness - assisting to meet the person’s needs (including carer
perceptions of service responsiveness and timeliness)
quality of life – a focus on carer health and wellbeing, inclusive of carer stress
communication and respect
care role and relationship - whether the person has been assisted in their care role,
to maintain and sustain the care relationship
service quality and satisfaction with service provision
use of other services, social connections, meaningful activities.
Conclusion
The development of a uniform set of questions as part of a standardised outcomes tool, that
could be administered via multiple methods (i.e. formal / less formal) would be beneficial to
achieving greater consistency in practice to inform outcomes and ultimately service
improvements.
Given the importance of health and wellbeing (including stress) at least one or more
questions underpinning the proposed outcomes measures should reflect one or more
measures of carer perceptions of timeliness, quality of life and/or health and wellbeing.
HDG Consulting Group 29
The survey findings indicate that there would be a reasonable level of support for the
introduction of a uniform tool and the associated questions to measure carer outcomes, and
that there would be a voluntary take-up rate by service provider organisations, especially if
it met the following criteria:
able to be implemented using a mix of methods (paper, online, verbal)
supported by an associated information resource/guide/instructions for use
able to be integrated into existing feedback or practice tools and/or added to by
individual organisations (while retaining the core items).
HDG Consulting Group 30
5 Feedback tools used by service providers
5.1 Current practice
Service providers were invited to forward copies of their existing feedback tools used to
determine carer or care recipients outcomes and/or gain feedback in relation to satisfaction
with service provision. A variety of forms were provided, with a range of different titles, such
as:
carer feedback form
telephone questionnaire
client satisfaction survey
client exit interview form.
These were analysed in relation to their structure, content, inclusion of outcome measures,
use of measurement scales and applicability to the development of uniform outcome
measures.
In addition, four examples of screening and/or assessment tools were provided. These
included a Service Coordination Tool Template (Health and social needs), carer self-
assessment tool to rate carer stress, a carer screening tool used to prioritise allocation of
care, and an assessment tool.
5.2 Content items
Content analysis of seventeen feedback tools revealed the use of five main categories of
information. These were demographic information, opinions and rating about various
aspects of the service, satisfaction in relation to various aspects of the service, whether
expectations had been met and/or the service made a positive difference. The ‘other’
category included a mix of items.
Most of the feedback tools included most of the categories, however, the questions within
each category varied, as did the response scales.
Table 3: Analysis of feedback tools forwarded by service providers
Organisation type Demographic information and service type
Opinions and ratings about various aspects of the service
Satisfaction ratings of various aspects of the service
Met expectations / difference made
Other
Community service organisation
Community service organisation
Community Service Organisation
Community Service Organisation
Community Service Organisation
Health or
community health service
Health or community health
HDG Consulting Group 31
Organisation type Demographic information and service type
Opinions and ratings about various aspects of the service
Satisfaction ratings of various aspects of the service
Met expectations / difference made
Other
service
Health or community health service
Local government Local government Local government Local government Local government Local government Local government Multicultural, ethno-specific
Multicultural, ethno-specific
Total 15 14 13 10 13
a) Demographic information and service type
Demographic information and service type information typically included items such as:
who completed the survey e.g. client, paid carer, relative, friend, other
gender, age, postcode, ATSI
category of respondent e.g. older person, person with dementia, disability, mental
health and so forth
a listing of the types of service/s provided e.g. information, emotional support,
residential respite, in-home respite, out-of-home respite/day centre, education,
other.
b) Opinions and views
Most feedback tools sought opinions, views and ratings about various aspects of service
provision. These have been further classified into the three sub-themes shown below.
Table 4: Examples of items seeking views and ratings about various aspects of service provision
Sub-category Examples of the item or question as included on the feedback tool
Timeliness The first time you contacted us, how long did you have to wait to speak to someone
who could help you?
Are you confident that when you contact the office your query is responded to in a
professional and timely manner?
Were you allocated a worker within a reasonable amount of time?
The service started in a timely manner.
Service is provided at a time that suits me.
My Community Care Worker stays for the entire rostered time.
I am advised in plenty of time if my usual service is to change.
Communication
and respect
Were you treated with courtesy and dignity?
Did your advocate, coordinator or facilitator offer you an interpreter?
Sensitivity and responsiveness to the customs and traditions of your nationality and
culture.
How do you rate the way that the [service] communicates with you about the
following [list of items].
Explaining things in a way you understand.
HDG Consulting Group 32
Sub-category Examples of the item or question as included on the feedback tool
Involving you in decisions affecting you and the person you care for.
My carer listens to my ideas and suggestions.
Was your plan written and explained in a way that you could easily understand?
The extent to which the Council listens when you have a problem with the service.
Understanding
needs
Understanding your needs and issues.
Offering support/respite options that meet your needs.
The service considers my individual needs.
Ensuring that they have a good understanding of your needs and issues.
Providing you with flexible options in relation to your needs.
Helping you to manage your problem or health need(s).
Doing their best to improve your health and well-being.
I am involved in making decisions about my care.
What else could we do to support you?
c) Satisfaction
Most feedback tools included satisfaction ratings for a range of items. Although phrased
differently, the questions tended to be similar in intent. Most questions relate to overall
service provision, although some sought feedback in relation to specific aspects of the
service provision. For example:
Overall how satisfied are you with the service provided?
In general, were/are you satisfied with our service?
Overall, how satisfied are you with the program?
Please rate Council’s Home Care Service overall.
I am satisfied with the service I am receiving.
Overall how satisfied are you with the service provided by [service]
Are you satisfied with the quality of service you receive?
Are you satisfied with the regularity of service you receive?
How satisfied were you with the respite?
How satisfied are you with discussions about your needs and preferences?
How satisfied are you that information is explained in a way that is easy to
understand?
With what aspects of the initial home visit that was provided by an office staff
member were you most satisfied?
d) Met expectations / difference made
In contrast to the categories described above, fewer feedback tools included specific items
in relation to whether the service user’s expectations and/or needs had been met or the
service provision had made a difference to them and/or their quality of life. Examples
included:
What is it that you hope to get from the services here?
How well is your requirement for each of the following being met?
I receive a sufficient number of hours of service to meet my needs.
To what degree did these services assist you to remain living in your home?
Do you feel that our service encourages you to remain active and independent?
I feel I have benefited from the program.
HDG Consulting Group 33
Before you first used the service how did you rate your quality of life? And now how
would you rate your quality of life? If your quality of life has not improved since you
started using the service, why hasn’t it improved?
How has your quality of life changed since using the service?
e) Other
The ‘other’ category included a mix of items such as:
confidentiality of personal information
awareness of rights and responsibilities in relation to the service
awareness about how to make compliments or complaints
feedback about staff
transport
questions about the cost of the service and invoicing
whether they would recommend the service to others.
5.3 Measurement scales
The feedback tools analysed included a range of response or measurement scales, both
quantitative and qualitative. Generally speaking there was little if any congruence between
these, even when the questions were similar across the different feedback tools. Examples
of the response or measurement scales included:
Yes / No tick boxes
Unsatisfactory / satisfactory / very good
Improved / unchanged / got worse / unsure
Always / usually / sometimes / never
Always / most of the time / rarely / never / don’t know or not applicable
More confident / no different / less confident
This group helps me achieve my goals / I haven’t set goals / This group didn’t help
me achieve my goals
1 = Poor, 2 = Just Alright, 3 = Satisfactory, 4 = Good, 5 = Excellent
Excellent / Good / Needs some improvement / Needs a lot of improvement / Don’t
Know/ Not applicable
Not at all satisfied / Partly satisfied / Satisfied / Very satisfied
Disagree Strongly / Disagree Somewhat / Neutral / Agree Somewhat / Agree
Strongly
Better than you expected / As you expected / Not as good as you expected / Unsure
10 point Likert scales, e.g. where 1 is very dissatisfied and 10 is extremely satisfied;
or where 1 is poor and 10 is excellent.
The scales ranged from simple yes / no responses to more detailed ten point scales. The
lack of a consistent response or measurement scale (in combination with the lack of uniform
questions) means that it is challenging to develop a baseline or benchmark between similar
services.
HDG Consulting Group 34
5.4 Summary
Based on the analysis provided above, it is evident that:
between service provider organisations, feedback tools vary in terms of structure,
content and depth of questions
questions are worded in a variety of ways, even when the intent of the question is
similar in nature
a range of different response and measurement scales are used, both quantitative
and qualitative
the feedback tools serve a range of purposes, for example, to gain feedback about
service quality, to assess service users’ satisfaction with service provision, and to
suggest service improvements
none of the feedback tools provided the set of outcomes measures being sought
within the context of this project.
It is clear that, if developed, a uniform set of outcomes measures is likely to be included by
service provider organisations within feedback tools seeking other information they wish to
collect. This reinforces the thinking that a simple set of questions with a uniform response
scale, will both enable benchmarking within and between organisations, and contribute to
continual quality improvement by service providers.
HDG Consulting Group 35
6 Conclusion
6.1 Findings
The project scope included four elements. The findings from each element are summarised
below. There is clearly both the opportunity for, and a level of interest in, the development
and introduction of a carer outcomes framework with a uniform set of outcome items and
measures, that can be administered using a range of methods, and that reflect the key
points below.
Table 5: Summary of findings by project element
Project element Key findings
Targeted
literature scan
Outcome measures can be used at different levels – individual (e.g. change
as a result of a strategy), organisational (e.g. program effectiveness) and
systemic (e.g. inform policy, population outcomes).
The UK ASCOF and Scottish POMs approaches provide examples of how
outcomes can be used across these three levels.
Several studies in Australia have concluded that capturing outcomes is a
complex process.
While there is a high volume of existing instruments designed to screen or
assess carer needs for specific variables (e.g. stress), none were identified
that reflected the intent of this project.
Consumer
consultation
The approach should be outcome (rather than output) focussed.
There should be a standard set of generically applicable questions to enable
benchmarking.
To include the key concepts of: finding out about services, receiving
information, communication and respect, understanding of needs, service
availability, matching to needs, quality, and the difference made to the carer.
A tool should be short (one page), with simple questions, a simple response
scale and room for explanatory comments.
Service provider
consultation
Current practice is varied with an inconsistent approach between
organisations.
Most service providers use a range of methods and do not rely on any one
single method alone, with a high use of verbal feedback during service
delivery.
Process items are most collected; and quality of life items are collected more
than care relationship items.
Process items tend to be collected in a more formal way, while quality of life
and care relationship items tend to be collected through verbal feedback.
The most important items to collect in relation to carer outcomes were
identified as service effectiveness, quality of life, communication and respect,
care role and relationship, and satisfaction with service quality.
There was minimal difference between HACC and SCP responses suggesting
that the same questions can be used by both programs.
The development of a uniform set of questions as part of a standardised
outcomes tool would be beneficial to achieving consistency in practice.
Key design considerations are multi-method administration (e.g. survey,
verbal), user instructions/guidelines, and capacity for integration into existing
feedback tools and processes.
HDG Consulting Group 36
Project element Key findings
Feedback tools
analysis
Feedback tools currently in use by service providers tend to be specific to
each organisation.
Content reflects five categories: demographic information, opinions and
ratings about various aspects of service provision, satisfaction, expectations
and other.
While there is some congruence between feedback tools the wording of
items varies.
There is little consistency in the measurement scales.
None of the feedback tools reflected the set of outcome measures being
sought within the context of this project.
6.2 Proposed logic model
Based on the project findings, a logic model and associated outcome items and measures
have been developed for consideration.
In developing the logic model, the key assumptions were that it should:
focus on outcomes for carers and care recipients
assist service providers to meet their obligations under the Victorian Carers
Recognition Act 2012 for reporting and improving quality of services
enable information collection as part of routine assessment, care planning and
review practice
provide information that will contribute to accreditation against the Community Care
Common Standards
add value to organisations in their continual quality improvement of services
enable reporting against the recommendations of the VAGO report
not constitute onerous additional reporting requirements
be applicable to the diversity of HACC and SCP organisations.
HDG Consulting Group 37
Figure 8: Proposed logic model
Vision Inputs Outputs Outcomes
For carers and care recipients to be supported in their care role and care relationship.
For the positive health and wellbeing
of carers and care recipients to be enhanced.
Legislation
Victorian Carers Recognition Act 2012
HACC Act
Policy and guidelines
Victorian charter supporting people in care relationships
Victorian health priorities framework 2012-2022
HACC program manual HACC initiatives (ASM, diversity) SCP guidelines.
Funding: HACC, SCP
Delivery
- Service providers
- Workforce, infrastructure, quality systems
Target groups
SCP: carers of older people, older carers and carers of younger people with dementia
HACC: frail older people, younger people with a disability and their carers
Types of support
Information and advice Counselling Respite (in-home,
community, individual, group, daytime, overnight)
Support or brokerage funds
Carers and care recipients are informed and able to access services that are responsive and effective in meeting their needs.
Carers and care recipients receive respect for, and support to maintain and sustain, their care role and relationship.
Carers and care recipients receive services that make a positive contribution to their quality of life, health and wellbeing.
The high level outcomes shown in the last column of the logic model map to the proposed
outcomes items and measures as described in the outcomes framework and outcomes tools,
and thus underpin data collection and the monitoring of outcomes, to inform service
improvement.
Figure 9: Flow from logic model to outcomes monitoring
Logic model
Defines high level outcomes being sought.
Outcomes framework
Lists outcome domains, items, questions and measurement scale.
Carer support outcomes tools
Staff administered version with guidelines and self administered version.
Data collection
Collection of outcomes data through formal methods, ongoing service delivery and everyday practice, in accordance with guidelines.
Monitoring
At an individual level, organisation level and systems level.
HDG Consulting Group 38
6.3 Outcomes framework
Congruent with the logic model outlined above, the following outcomes framework, items
and measures reflect the findings of the project elements as described in previous sections
of this report. It has been structured so as to:
take into account service system and organisational barriers and limitations
take into account system and organisational enablers such as the use of a simple,
uniform set of questions
be implemented using a mix of methods (paper, online, verbal)
be integrated into existing feedback tools and processes, and/or added to by
individual organisations
include one or more measures of timeliness
contribute to continual quality improvement.
The proposed framework provides a uniform set of questions32 that can be used by
organisations in their everyday practice (e.g. assessment, care planning, review) as well as
in more formalised feedback processes for quality improvement purposes. The framework
therefore provides a set of questions that can be used as part of everyday practice during
informal conversation with carers and care recipients as well as through more formal
processes. This approach will complement existing reporting requirements in that funded
organisations can incorporate the items/questions into their existing feedback tools at an
organisational level, as well as use them in everyday conversation with carers and care
recipients.
In addition to the approach shown, the mapping of specific instruments (e.g. stress index,
personal wellbeing index) against these items could assist those organisations that wished to
measure change for specific variables at an individual level, and generate data to inform
these broader measures. Thus the use of validated instruments and scales to measure
change for specific items (such as a carer stress index, or goal attainment) can be
integrated into the framework and practice where appropriate to the presenting situation
and to the role of and qualifications of the practitioner; however, they are not a pre-
requisite or barrier to implementing the proposed framework, items and measures.
The proposed outcomes framework, items and measures developed as a result of this
project, is shown over page.
32 It should be noted that these questions have not been independently tested for reliability, validity or responsiveness; and this could be an important part of pilot testing.
HDG Consulting Group 39
Figure 10: Proposed carer outcomes measures
Domain Measure
type #
Indicative questions –
for use in a written
survey
Indicative questions – for
use in a verbal survey
Indicative questions – for use in everyday
practice
Desired result33
Other individual level
instruments, variables
and/or questions used
by organisations may
further inform this
item, e.g.34
Information Process 1
I received information
when I needed it about
the types of services
available.
Did you receive information
when you needed it about
the types of services
available?
How are we going in providing you with
the information you need when you need
it? Is there any other information you
need?
Informed decision
making.
Ease of service
navigation
Connection to other
services
Respect Process 2
The staff I spoke with
listened to me and
respected my opinions
as a carer/care
recipient.
Did the staff listen to you
and respect your opinions
as the carer/care recipient?
How are we going in listening to you and
understanding your needs? Is there
something you feel we have missed
about your situation? How could we get
to know you and your needs better?
Person centred
practice.
Access Process 3
The services were
available when I most
needed them.
Were the services available
when you most needed
them?
How does this time suit you? Is this
service at a good time for you? If
available, would a different time be
better?
Timeliness of access.
Effectiveness Care role 4
The services were
effective in supporting
my needs and goals as
a carer/care recipient.
Were the services effective
in supporting your needs
and goals?
How are we going in supporting you? Are
our services making it easier or harder
for you? In what ways? Meeting needs.
Social isolation
Goal attainment
Quality Satisfaction 5
The services I received
were satisfactory and of
good quality.
Were the services
satisfactory and of good
quality?
How are you finding the quality of
services? How could we make the service
better for you?
Service quality.
Care
relationship Care role 6
The services I received
helped me to continue
in my care role and/or
relationship.
Did the services you
receive help you to
continue in your care role
and/or relationship?
How is the service helping you to manage
your situation? Are there other things
that you would like or that could assist
you?
Sustaining care roles. Carer stress index
Carer knowledge and
life skills
Quality of
life QoL 7
The services I received
made a positive
difference to my life.
Did the services make a
positive difference to your
life?
Do you think this service is making a
difference? What else could we do to
make a difference to you?
Psychological health,
wellbeing and quality
of life.
The response scale is the same for all questions:
Never Rarely Sometimes Most of the time Always; with additional boxes for Unsure and Not applicable
However, because this scale has a limited degree of sensitivity, and/or to be congruent with a specific service provider’s survey or quality practice, some
organisations may wish to use a more sensitive scale. This is acceptable provided the responses can be consolidated into the proposed scale for comparison
and benchmarking purposes.
Provision for additional comments or feedback should be made for each item.
See Appendix 2 for survey example.
33 These should be interpreted as contributing towards the desired result or a future benchmark. 34 For example, tools that measure specific variables (e.g. stress, social isolation) may be used by some organisations and provide longitudinal data in relation to these outcomes.
HDG Consulting Group 40
6.4 Benefits and recommendations
The approach outlined above is beneficial in that it reflects the findings from each of the
four project components and other key considerations; is relatively simple and thus more
likely to be achieved within existing resources; and caters for the diversity of organisations.
The table below summarises the benefits of the proposed approach against the findings of
each project element.
Table 6: Benefits of proposed approach by project element
Project element
/ considerations Benefits of proposed approach Addressed
Literature Reflects the intent of an initial overarching framework at a systems level. Yes
Can link (in the longer term) to standardised and routine data collection and a
suite of formally endorsed resources for measures of activities, change and
outcomes.
Partial*
Generates service improvement information that can be used at an
organisational level.
Yes
Generates service improvement information that can be used as part of
practice at an individual carer or care recipient level.
Yes
Consumer
(carer and care
recipient)
consultation
Provides a simple, uniform approach. Yes
Able to be implemented through paper-based, verbal or online methods. Yes
Reflects the concepts identified as important by carers and care recipients. Yes
Simple concepts and response scales that can be translated into community
languages and are appropriate for people with low literacy.
Yes
Service provider
consultation
Diversity of collection methods – can be achieved via multiple collection
methods.
Yes
Diversity of organisation types – can be achieved by multiple organisation
types and sizes.
Yes
Interface with other practices – can be implemented as a stand-alone process
or integrated with an organisation’s assessment, care planning and review
processes, quality improvement processes and existing feedback tools.
Yes
Can be administered by a worker or self-administered by a consumer. Yes
Service
improvement
Contributes towards service improvement and continual quality improvement. Yes
Achievable Not onerous in terms of additional information collecting or reporting. Yes
VAGO report Provides a monitoring framework. Yes*
Provides a core level response to the VAGO recommendations by the inclusion
of one or more measures in relation to timeliness.
Yes
Victorian Carers
Recognition Act
Can be used by the HACC and SCP to assist in meeting the requirements of
the Act.
Yes
*Additional and/or further work required beyond the scope of this project e.g. suite of formally endorsed resources;
coordinated system that allows for statewide collection, collation and analysis of data for reporting and benchmarking
purposes; incorporated into service provider care management systems for routine data collection and analysis.
HDG Consulting Group 41
Recommendations
Based on the project scope and findings, the following recommendations are suggested in
order to further progress the development and implementation of a monitoring framework
and outcomes measures for HACC (Respite and Planned Activity Groups) and the SCP.
Recommendation 1
That the indicative questions as detailed in the proposed carer outcomes measures
be designed into a range of formats (staff administered, carer administered), and the
associated guidelines developed.
Indicative timing: July – December 2014
Recommendation 2
That further feedback is sought from the consumer groups that provided input at the
commencement of the project, regarding the proposed carer outcomes measures,
formats and guidelines.
Indicative timing: July – December 2014
Recommendation 3
That consideration is given to pilot testing the proposed carer outcomes measures,
formats and guidelines. The aim of the pilot would be to test the carer outcomes
measures and items for analysis of reliability, validity and usability purposes. The
pilot would be designed to include HACC and SCP and to test the formats with a mix
of organisation types, staff categories and implementation modes, and include
practitioner and carer evaluation.
Indicative timing: January – June 2015
Recommendation 4
That the project outcomes are used to inform the HACC ASM project that plans to
investigate and identify service user outcome measures from an ASM perspective.
Indicative timing: To occur over 2014/15.
Recommendation 5
That the project findings are made available and communicated to project
participants and other key stakeholders.
Indicative timing: July – December 2014
These actions will progress the use of carer support outcome measures at an individual and
organisation level, and will contribute to a systemic approach in the longer term. The
proposed approach will respond to the VAGO recommendations and inform continual quality
improvement of services to support the essential role and wellbeing of carers.
HDG Consulting Group 42
Appendix 1
Detailed survey results
1. How carer satisfaction is currently measured
Service providers were asked about how they measure carer satisfaction for the services and
support options they provide.
The chart below shows the feedback method as a proportion of the total responses for
carers and care recipients combined and including multiple methods per single organisation.
As shown, the triad of satisfaction surveys, verbal feedback (informal) and goal
achievement/care planning reviews are similar proportionally at around one quarter each of
all responses.
Figure 11: Feedback methods
Further analysis indicates the most commonly used methods.
The most commonly used method was verbal feedback gained informally through ongoing
service provision. This method was used with carers by 91 per cent of respondents, and
with care recipients by around 86 per cent of respondents (most commonly in conjunction
with other methods).
Use of feedback through service reviews as part of ongoing service delivery was reported
by 68 per cent of respondents, and through monitoring goal achievement by 64 per cent
of respondents. Thirty per cent of respondents conducted annual feedback through care
planning or service reviews; and 26 per cent conducted annual monitoring of goal
achievement.
Around half, 55 per cent of respondents undertook satisfaction surveys annually with
carer recipients and a slightly lesser proportion at 48 per cent with carers. Quarterly
satisfaction surveys were less frequent, used by four to seven per cent of respondents.
Around half (55 per cent) of respondents used feedback at service exit.
Focus groups with carers were the least used method. They were used by 13 per cent of
respondents with carers and by nine per cent of respondents with care recipients.
Satisfaction
survey 23%
Focus groups
9%
Verbal
feedback (informal)
26%
Goal
achievement, care
planning, reviews
27%
Service exit
8% Other
7%
Feedback methods
HDG Consulting Group 43
Some providers reported using other methods such as feedback at events, suggestion
boxes, online and so forth.
In relation to any differences (across all methods) between HACC and SCP:
SCP providers were more likely than HACC providers to use focus groups with carers
(variance of 15 per cent), carer satisfaction surveys (variance of 9 per cent), and
other approaches such as suggestion boxes, events etc (variance of 16 per cent)
HACC providers were more likely than SCP providers to use feedback through care
planning or service reviews (variance of four per cent).
The data above indicates that there is a high use of verbal feedback methods through
ongoing service delivery to monitor satisfaction. More formal approaches such as satisfaction
surveys tend to be used annually by around half of the service providers; and goal
achievement is monitored annually by around one quarter of service providers. In
considering future outcome measures a key consideration will be the ability to collect
outcomes data through a range of methods - both formally through survey type processes
as well as through discussion as part of ongoing service delivery.
A follow up question asked about the methods that provide organisations with the best
information for service improvement purposes.
Around 70 per cent of respondents typically noted that verbal feedback and information
gathered through informal conversations was the most useful information to assist with
improving service quality.
Forty-three per cent of respondents identified surveys or other forms of written feedback
such as evaluation. Around 35 per cent of respondents commented on information gathered
through care plan and service reviews, including information gathered during assessment or
at exit. Fourteen per cent of respondents identified focus groups or other types of group
meetings.
Around half of all respondents identified that a combination of multiple (rather than single)
approaches were beneficial to service improvement. Examples of comments include:
‘Annual client surveys and verbal feedback through ongoing service delivery provide
the best information for measuring carer outcomes.’
`Verbal feedback is useful but more difficult to analyse and provides an individual
point of view rather than highlighting trends or a broader perspective. Satisfaction
surveys provide a broader view, however I often think this format makes it difficult
to measure client outcomes - a person may be satisfied with the services - but that is
not to say that the outcomes were met.’
`Feedback through care planning during service reviews and monitoring forms.’
Based on this feedback it is evident that the questions underpinning the proposed outcome measures should be able to be used and applied via a range of methods.
HDG Consulting Group 44
2. Collection of process related outcome measures
Respondents were asked to identify which process related items they currently collect or
measure. Process related outcomes were defined as outcomes that reflect the experience
that individuals have in seeking, obtaining and using services and supports.
The items listed were:
Information provision (e.g. I was provided with information about services and
supports in a timely manner)
Communication, respect (e.g. I was recognised, listened to and respected by staff
who valued my input, expertise and preferences)
Person-centred (e.g. I was encouraged to take an active role in decision making and
develop a solution to suit me)
Service system capacity (e.g. I was informed about what services and supports were
available, how much may be provided, and my options)
Timely access (e.g. I was able to access carer supports in a timely manner)
Suitable staff (e.g. I was allocated care or support staff that understood and were
suitable to my needs)
Positive experience (e.g. I had a positive experience with the services and support
provided).
Across all items and response methods, other feedback (e.g. verbal/discussion) was the
method that accounted for the largest proportion of responses, followed by carer
satisfaction surveys.
Figure 12: Feedback methods – process related items
Between 39 per cent and 51 per cent of respondents reported collecting the process related
items using a carer satisfaction survey. A similar proportion at 42 to 52 per cent of
respondents reported collecting these items using other forms of feedback (e.g.
verbal/discussion).
The use of carer outcomes tools was less common with only eight to 16 per cent of
providers using this method, pending the particular item. For example, the item ‘person-
centred’ was more likely to be included on a carer outcomes tool (16 per cent of providers)
than the item ‘timely access’ (eight per cent of providers).
The type of collection method also varied depending on the particular item.
Other
feedback 39%
Carer
satisfaction survey
34%
Will collect in
future 13%
Carer
outcomes tool 9%
Do not collect
5%
Methods - Process outcomes
HDG Consulting Group 45
Carer satisfaction surveys were more likely to include the items of communication and
respect and information provision (48-51 per cent of providers) and less likely to include
suitable staff or timely access (39 per cent).
Service system capacity and the positive experience were more likely to be collected
though other feedback (e.g. verbal/discussion) (52 per cent) and slightly less likely to be
collected through a survey process (40 per cent).
Timely access was the item least collected through a survey or outcomes tool process and
second least collected through other feedback (e.g. verbal/discussion) processes. Fourteen
per cent of respondents do not currently collect feedback in relation to timely access.
Up to 21 per cent of respondents indicated that they would collect one or more items in
future. It was not specified as to whether this would be as well as, or instead of, items
currently collected.
Table 7: Collection of process related outcome measures
Outcome measure
Collection method
Carer
satisfaction
survey
Carer
outcomes
tool
Other
feedback
Do not
collect
Will
collect in
future
a) Information provision (e.g. I was
provided with information about services
and supports in a timely manner)
48% 8% 51% 4% 16%
b) Communication, respect (e.g. I was
recognised, listened to and respected by
staff who valued my input, expertise and
preferences)
51% 9% 48% 3% 19%
c) Person-centred (e.g. I was encouraged to
take an active role in decision making and
develop a solution to suit me)
41% 16% 40% 4% 21%
d) Service system capacity (e.g. I was
informed about what services and supports
were available, how much may be provided,
and my options)
40% 11% 52% 6% 15%
e) Timely access (e.g. I was able to access
carer supports in a timely manner) 39% 8% 42% 14% 18%
f) Suitable staff (e.g. I was allocated care or
support staff that understood and were
suitable to my needs)
39% 9% 46% 9% 16%
g) Positive experience (e.g. I had a positive
experience with the services and support
provided)
44% 8% 52% 6% 14%
Comparison between HACC and SCP indicates that across all items and methods (excluding
do not collect and will collect in future):
SCP providers are more likely to use a carer satisfaction survey compared to HACC
providers, at 50 per cent to 40 per cent respectively
HACC providers are more likely to use other feedback (e.g. verbal/discussion)
compared to SCP providers, at 50 per cent to 38 per cent respectively
Both HACC and SCP use carer outcomes tools at similar rates, at 10 and 13 per cent
respectively.
HDG Consulting Group 46
Analysis of the difference between service types indicates that there are differences in
practice between organisation types, as summarised below.
Table 8: Process outcomes – collection method by organisation type
Organisation type Method: Across all the process related items listed above, this type of
organisation is most likely to:
Aboriginal Community Controlled
Organisation Collect this type of information in future
Community service organisation
Use a carer satisfaction survey (40 per cent) followed by other feedback
(e.g. verbal/discussion) (32 per cent) followed by a carer outcomes tool
(12 per cent)
Health service / Community health
service
Use other feedback (e.g. verbal/discussion) (47 per cent) followed by a
carer satisfaction survey (37 per cent) followed by will collect this
information in future (11 per cent)
Local government
Use other feedback (e.g. verbal/discussion) (38 per cent) followed by a
carer satisfaction survey (30 per cent) followed by will collect this
information in future (19 per cent)
Multicultural / ethno-specific
Use other feedback (e.g. verbal/discussion) (38 per cent) followed by a
carer satisfaction survey (34 per cent) followed by a carer outcomes tool
(13 per cent)
In relation to process measures, carer satisfaction surveys and other feedback (e.g.
verbal/discussion) were more often used (up to 52 per cent of respondents) whereas carer
outcomes tools were less often used (up to 16 per cent). This is congruent with qualitative
feedback from other questions suggesting that the development and use of a standardised
outcomes tool would be beneficial.
3. Collection of care relationship outcome measures
Respondents were asked to identify which care relationship related items they currently
collect or measure. These outcomes relate to supporting the care role. These outcomes
explore the capacity, confidence and sustainability of the care relationship, in relation to the
services provided.
The items listed were:
Skills, confidence (e.g. The information, services and support I received increased
my care skills and confidence in my care role)
Stress (e.g. The services and support decreased the stress or intensity of my care
role)
Partnership (e.g. The services and support worked beside me to support my care
role)
Sustaining the care relationship (e.g. The services and support helped me to
maintain, or if relevant, improve my care relationship)
Effectiveness (e.g. The services and support were effective in supporting my care
role and relationship).
Across all items and response methods, other feedback (e.g. verbal/discussion) was the
method that accounted for the largest proportion of responses, followed by will collect in
future.
HDG Consulting Group 47
Figure 13: Feedback methods – care relationship items
Analysis indicates that these items are less frequently collected in comparison to the process
measures discussed above, particularly through more formal processes such as satisfaction
surveys.
Between 16 per cent and 29 per cent of respondents reported collecting these items using a
carer satisfaction survey whereas 39 to 48 per cent reported collecting these items using
other feedback (e.g. verbal/discussion), indicating that less formal processes are more
commonly used.
Carer satisfaction surveys were more likely to be used in relation to the items of
effectiveness (29 per cent of providers) and less likely to include partnership or skills and
confidence (16-17 per cent of providers).
The use of carer outcomes tools was less common with only six to 11 per cent of providers
using this method, pending the particular item. For example, the item ‘stress’ was more
likely to be included on a carer outcomes tool (11 per cent of providers) than effectiveness
(six per cent of providers). The least collected item collected via a carer outcomes tool was
sustaining the care relationship and effectiveness.
Other feedback (e.g. verbal/discussion) processes were used by 39-48 per cent of
respondents pending the particular item. The item stress was most considered (48 per cent
of respondents), indicating that there is a significant amount of discussion regarding carer
stress although it is monitored informally. Three respondents (one health service, one
community health service and one community service organisation) reported using the Carer
Stress Index to assess the carer and inform priority of access.
The item skills and confidence was least considered.
Between 15 and 27 per cent of respondents did not collect these items.
Other
feedback 38%
Will collect in
future 19%
Do not collect
19%
Carer
satisfaction survey
18%
Carer
outcomes tool 6%
Methods - Care relationship items
HDG Consulting Group 48
In the Australian context, the health and wellbeing of carers has been investigated in detail
in numerous studies. An Australian literature review (2004) noted that increased carer strain
appears to be more predominantly associated with care recipients factors (such as
dependence, cognitive impairment, incontinence, sleep disturbance) rather than carer or
care situation factors. Studies suggest that respite services need to directly address the
psychological effect of caring by providing psychological services that address depression,
anxiety, and stress associated with the caring process, and to encourage carers to maintain
or improve their levels of social participation.35 However, results from evaluative research
and systematic reviews of the literature on carer support programs have been inconclusive
in their effect on carer strain and the results cannot be easily reviewed and compared
because of the different methods being used.36
Table 9: Collection of care relationship outcome measures
Items
Collection method
Carer
satisfaction
survey
Carer
outcomes
tool
Other
feedback
Do not
collect
Will
collect in
future
a) Skills, confidence (e.g. The information,
services and support I received increased
my care skills and confidence in my care
role)
17% 7% 39% 4% 16%
b) Stress (e.g. The services and support
decreased the stress or intensity of my
care role)
18% 11% 48% 3% 19%
c) Partnership (e.g. The services and
support worked beside me to support my
care role)
16% 7% 43% 4% 21%
d) Sustaining the care relationship (e.g.
The services and support helped me to
maintain, or if relevant, improve my care
relationship)
22% 6% 44% 6% 15%
e) Effectiveness (e.g. The services and
support were effective in supporting my
care role and relationship)
29% 6% 40% 14% 18%
Comparison between HACC and SCP indicates that across all items and methods (excluding
do not collect and will collect in future):
SCP providers are more likely to use a carer satisfaction survey compared to HACC
providers, at 32 per cent to 27 per cent respectively
HACC providers are more likely to use other feedback (e.g. verbal/discussion)
compared to SCP providers, at 64 per cent to 55 per cent respectively
Both HACC and SCP use carer outcomes tools at similar rates, at 9 and 13 per cent
respectively.
Analysis of the difference between service types indicates that there are differences in
practice between organisation types, as summarised below.
35 Cited in Tang J, Ryburn B, Doyle C, Wells Y 2011, ‘The Psychology of Respite Care for People with Dementia in Australia’, Australian Psychologist, vol. 46 pp.183-189. 36 Eagar K et al. (2007) Effective Caring: a synthesis of the international evidence on carer needs and interventions. Centre for Health Service Development, University of Wollongong.
HDG Consulting Group 49
Table 10: Care relationship outcomes – collection method by organisation type
Organisation type Method: Across all the process related items listed above, this type of
organisation is most likely to:
Aboriginal Community Controlled
Organisation Collect this type of information in future
Community service organisation
Use a carer satisfaction survey (40 per cent) followed by other feedback
(e.g. verbal/discussion) (32 per cent) followed by a carer outcomes tool (12
per cent)
Health service / Community
health service
Use other feedback (e.g. verbal/discussion) (47 per cent) followed by a
carer satisfaction survey (37 per cent) followed by will collect this
information in future (11 per cent)
Local government
Use other feedback (e.g. verbal/discussion) (38 per cent) followed by a
carer satisfaction survey (30 per cent) followed by will collect this
information in future (19 per cent)
Multicultural / ethno-specific
Use other feedback (e.g. verbal/discussion) (38 per cent) followed by a
carer satisfaction survey (34 per cent) followed by a carer outcomes tool
(13 per cent)
Care relationship measures are not collected as much as process measures. While almost
half of respondents collected information about some items through informal measures they
are less likely to be measured though formal methods. Carer outcomes tools were least
often used (up to 16 per cent). The least collected items using a carer outcomes tool were
sustaining the care relationship and effectiveness.
There appears to be a level of preparedness to collect these items as 18 to 27 per cent of
respondents suggest that they will do so in future.
4. Collection of quality of life outcome measures
Respondents were asked to identify which quality of life related items they currently collect
or measure. Quality of life outcomes relate to how satisfied the person is with aspects of
their life and what matters to them.
It has been well documented that caring is a demanding role and that respite is a key
strategy to support carers to sustain their care role. Respite has been reported as improving
the quality of life.37
The items listed were:
Social contact (e.g. How satisfied are you with your opportunities for social
participation or time out for rest and recreation?)
Meaningful activities (e.g. How satisfied are you with your opportunities for having
things to do and undertaking meaningful activities, such as employment,
volunteering or hobbies?)
Satisfaction with care role (e.g. How satisfied are you with the aspects of your care
role and the care relationship that matter most to you?)
Staying well (e.g. How satisfied are you with your own health and wellbeing?)
37 Evans D 2011, ‘Exploring the concept of respite,’ Journal of Advanced Nursing pp. 1-11
HDG Consulting Group 50
Across all items and response methods, other feedback (e.g. verbal/discussion) was the
method that accounted for the largest proportion of responses, followed by do not collect.
Figure 14: Feedback methods – quality of life items
Similar to the previous sections, the most commonly used method is through other (e.g.
verbal/discussion) feedback.
Between 16 per cent and 26 per cent of respondents reported collecting these items using a
carer satisfaction survey. Carer satisfaction surveys were more likely to be used in relation
to the items of meaningful activities (26 per cent of providers) and less likely to include
satisfaction with care role (16 per cent of providers).
The use of carer outcomes tools was less common with only 11 to 14 per cent of providers
using this method, pending the particular item. For example, the item social contact was
more likely to be included on a carer outcomes tool (14 per cent of providers) than staying
well (11 per cent of providers).
Other feedback (e.g. verbal/discussion) processes were used by 39-51 per cent of
respondents pending the particular item. The item social contact was most considered (51
per cent of respondents) and satisfaction with care role least considered.
Similar to the previous section (care relationship measures) 17 to 25 per cent of
respondents did not collect these items. One quarter of respondents do not collect
information (via any method) in relation to meaningful activities, satisfaction with care role
and staying well, although up to 32 per cent of providers indicate they will do so in future.
Other
feedback 38%
Do not collect
20%
Carer
satisfaction survey
17%
Will collect in
future 15%
Carer
outcomes tool 10%
Methods - Quality of life items
HDG Consulting Group 51
Table 11: Collection of quality of life outcome measures
Items
Collection method
Carer
satisfaction
survey
Carer
outcomes
tool
Other
feedback
Do not
collect
Will
collect in
future
a) Social contact (e.g. How satisfied are you
with your opportunities for social participation
or time out for rest and recreation?)
22% 14% 51% 17% 18%
b) Meaningful activities (e.g. How satisfied are
you with your opportunities for having things
to do and undertaking meaningful activities,
such as employment, volunteering or
hobbies?)
26% 12% 43% 25% 13%
c) Satisfaction with care role (e.g. How
satisfied are you with the aspects of your care
role and the care relationship that matter
most to you?)
16% 12% 39% 25% 23%
d) Staying well (e.g. How satisfied are you
with your own health and wellbeing?) 19% 11% 43% 25% 18%
Comparison between HACC and SCP indicates that across all items and methods (excluding
do not collect and will collect in future):
both HACC and SCP use satisfaction surveys at similar rates, at 27 and 24 per cent
respectively
both HACC and SCP use other feedback (e.g. verbal/discussion) at the same rate, at
57 per cent
SCP providers are marginally more likely than HACC to use a carer outcomes tool, at
19 per cent and 16 per cent respectively.
Analysis of the difference between service types indicates that there are differences in
practice between organisation types, as summarised below.
Table 12: Quality of life outcomes – collection method by organisation type
Organisation type Method: Across all the process related items listed above, this type of
organisation is most likely to:
Aboriginal Community Controlled
Organisation Collect this type of information in future
Community service organisation
Use other feedback (e.g. verbal/discussion) (32 per cent) followed by do
not collect (23 per cent) followed by a carer satisfaction survey (17 per
cent)
Health service / Community health
service
Use other feedback (e.g. verbal/discussion) (43 per cent) followed by do
not collect (20 per cent) followed by a carer satisfaction survey (15 per
cent)
Local government
Use other feedback (e.g. verbal/discussion) (40 per cent) followed equally
by a carer satisfaction survey, do not collect and will collect in future (all
17 per cent)
Multicultural / ethno-specific Use carer satisfaction survey and other feedback (e.g. verbal/discussion)
(both 28 per cent) followed by do not collect (21 per cent)
HDG Consulting Group 52
Quality of life measures are collected slightly more than care relationship measures and
slightly less than process measures. While up to half of respondents collected information
about some items through informal measures, they are less likely to be measured though
formal methods. Similar to the other sections, carer outcomes tools were least often used.
The least collected item is satisfaction with the care role and relationship. There appears to
be a level of preparedness by the sector to collect these items as 13 to 18 per cent of
respondents suggest that they will do so in future.
5. Improvement
Service providers were asked for suggestions about how the service system and/or their
organisation could improve the collection and measurement of carer outcomes, and what
would be practical and achievable from their perspective.
Respondents replied to this question in various ways - some respondents reported how they
collected useful information; while other respondents reported what they collected that was
most useful.
In relation to how the most useful information was collected, one third of respondents
identified a survey (such as a satisfaction survey) or tool (such as a care planning tool or
SCTT); and 22 per cent reported general verbal feedback and discussion. Examples include:
‘Gauging through the Carer Satisfaction Survey whether we are offering support and
options that meet the needs of our carers is integral to our service delivery.’
‘Feedback during care planning and service reviews has been the most valuable
information as the carer and care recipient are currently in the midst of receiving
services and as we review their goals we can also identify what worked well, what
they would like to change and what their goals should be for the future.’
‘SCTT38 for care recipients to assess needs, functional capacities and set achievable
and measurable goals; Carer Profile Tool to assess carer needs, stress levels and
sustainability of carer and care relationship.’
‘Combination of surveys and care planning tools.’
‘Introduction of goal centred care plans.’
In relation to what type of information was most useful to collect, 21 per cent of
respondents suggested service quality related information – for example, opportunities for
improvement, compliments and complaints, and information collected through a range of
feedback forms.
‘Regular surveys assist in determining the outcomes/successes/opportunities for
improvement of our services for care recipients and their carers.’
‘Information regarding the quality of the direct services is of value to ensure that we
are continuing to provide high quality support for carers. Asking carers what other
38 Service Coordination Tool templates (SCTT) is a suite of templates developed to facilitate service coordination. They support
the collection, recording and sharing of initial contact, initial needs identification, referral, information exchange and care planning information in a standardised way. See http://www.health.vic.gov.au/pcps/sctt.htm
HDG Consulting Group 53
supports would be useful to them in their caring role is also valuable as this provides
an opportunity for us to adapt services to meet these needs.’
‘Have your Say forms.’
‘Compliments and complaints offer the most valuable information.’
An additional 13 per cent of respondents identified the type of information as the activities
people enjoyed, social engagement and coping skills:
‘Understanding what activities people find most meaningful.’
‘Ease of access, quality of information provision, and quality of emotional support
and advice offered …’
‘Feelings of belonging (social engagement).’
This information indicates that organisations believe they could improve the collection and
measurement of carer outcomes by various means – a survey and/or tool and verbal
feedback and discussion, with a focus on the quality of support, opportunity to improve
supports and satisfaction.
6. Most important items to collect
Respondents were asked to identify the four most important care outcomes to collect. The
free text response boxes allowed respondents to enter multiple concepts into each of their
four most important items. Based on a content analysis, the nominated items for the first
and second most important items are shown below.
As indicated, the overall most important item was considered to be whether the service was
effective and assisted to meet the person’s needs and goals. This included whether the
service met their needs and expectations and was responsive and timely in doing so.
The next most important item was considered to be measures in relation to quality of life
with a focus on carer health and wellbeing, inclusive of carer stress.
The next most important item was considered to be a cluster of items categorized under the
theme of communication and respect. This included respect for the care relationship,
respect for the carer’s knowledge, effective communication and the provision of information
about the service system.
This was followed by measures in relation to care roles and relationships, such as
whether the person has been assisted in their care role, and to maintain and sustain the
care relationship.
This was followed by items to do with service quality and satisfaction.
Two items emerged in the second choice that did not show in the first – being other
supports and links such as whether the carer was using other services and social
connections; and meaningful activities for the carer and/or care recipient.
HDG Consulting Group 54
Table 13: Most important items to collect
Category/theme Per cent of respondents
First Second Combined
Service was effective - needs/goals met 30 27 57
QoL, carer health and wellbeing including carer stress 23 17 40
Communication, respect 11 16 27
Care role and relationship 16 6 22
Service quality / satisfaction 11 11 22
Other supports and links 14 14
Meaningful activities 11 11
Respondents were also asked to suggest other outcomes measures that they considered
important to collect, for example to meet the requirements of the Victorian Carers
Recognition Act.
Around 30 per cent of survey respondents completed this question. Responses tended to
reflect the items above and included some additional items:
26 per cent of respondents to this question commented about sustaining the care
relationship, stress, and the ability to take a break
24 per cent of respondents to this question made comment in relation to quality of
life, wellbeing, and social connections
19 per cent of respondents to this question made comment in relation to diversity
related measures, respect for cultural identity and cultural responsiveness
A range of other suggestions included skills, consumer directed care plans, the number of
referrals made, confidence in services and staff, whether they had a positive experience and
activities were meaningful and so forth.
Examples include:
‘Measuring carer stress and quality of life are the two main points that would be
good to measure.’
To ‘Have their social wellbeing and health recognised in matters relating to the care
relationship.’
‘Yes, it would be beneficial to collect data relating to cultural diversity and other
factors which may impact …’
‘Confidence and trust in staff.’
This information indicates that respondents consider the most important care outcomes to
collect are items about: effectiveness in responding to consumer needs, carer health and
wellbeing, communication and respect, and care roles and relationships.
HDG Consulting Group 55
7. Barriers and enablers
Respondents identified a range of barriers and enablers.
Almost half of respondents to this question identified time and resources as a key barrier -
for the service provider organisation and the carer, who was often time poor and
experiencing stress.
The next most frequently identified barrier was carer reluctance to reveal personal
information or provide feedback.
Likewise, literacy and language were identified as barriers.
Examples include:
‘Time, resources, capacity of same information to be collected across service sector.’
‘Barriers can include language and literacy, carer stoicism, staffing resources, our
current Client Management System.’
‘Finding the best outcome measure tool to use is the biggest barrier, as well as
having the same tool used by all HACC providers.’
‘An annual survey tool for the existing carers would be valuable to collect information
about their experience with our service.’
Poor quality feedback due to poorly designed questions, the inability to find appropriate
feedback tools, or feedback tools that were not easy to use by staff or carers were also
identified as barriers.
The most frequently identified enabler was organisational capacity in terms of a supportive
policy and the associated processes, information technology systems and practice tools.
This was followed by staff skills and the development of a trusting relationship with the
carer.
Adequate time and resources were also identified as enablers. Examples include:
‘Enablers: Skilled staff, prioritising reflective practice.’
‘The enabler would be to have a standardised consumer and carer satisfaction survey
to collect their views on the issues of being recognised, supported and be heard etc.’
There did not appear to be any key differences to barriers or enablers by organisation or
program type, with the exception of multi-cultural and ethno-specific organisations that
were more likely to identify literacy/language as a barrier.
In summary, key barriers were considered to be time and resources, combined with carer
reluctance to reveal personal information.
The key enablers were considered to be organisational capacity (policies, processes, and
systems), staff skills and time.
HDG Consulting Group 56
8. Suggestions for meeting the requirements of the Victorian Carers Recognition Act
Of those who responded to this question, the most common suggestion was for further
promotion of the Act to the general public, to senior management and to staff; the provision
of information resources such as fact sheets, checklists, best practice examples; and
associated training. These accounted for 35 per cent of suggestions. For example:
‘Having a greater presence of the Act in the media.’
‘Education for staff and management.’
‘Information session for Community Support Workers that summarises the Act.’
‘A centrally located e-learning module that all direct care workers are mandated to
view to understand their obligations under the Act.’
‘Simple dot points on how HACC services should modify / change their practice to
meet these requirements.’
‘A workshop on the Act and its relevance would be very helpful and this would
support the Active Service Model discussions.’
‘Information available in community languages.’
Around twenty six per cent of respondents made suggestions relation to the development of
uniform feedback tools and outcomes measures. For example:
‘Carer outcome tool.’
‘Generic carer / care recipient survey.’
‘Set forms to complete - it is easier to obtain information from carers / clients if the
form is a requirement.’
‘It would be an advantage to have a standard questionnaire/ survey to ensure the
answers are all measuring the same.’
‘A standardised set of questions/questionnaire developed by the Department so that
consistent data can be gathered across agencies.’
‘Universal feedback tools for evaluation i.e. wellbeing survey for all SCP funded
agencies.’
Twelve per cent of respondents identified the need for financial resources and/or more
flexible funding. For example:
‘The implementation of a formal system to assess, monitor, collect and record details
of carer's needs would require additional resources eg staff hours, ongoing review,
carer support.’
‘Increased funding for increased staff resources, training of staff.’
‘More flexible funding and guidelines allowing for more flexible service delivery.’
Six per cent of respondents reported that assistance was not needed and that their
organisation was meeting the requirements of the Act.
HDG Consulting Group 57
These suggestions indicate that there is scope for ongoing information distribution and
promotion in relation to the Act and how funded organisations can meet the requirements of
the Act.
A uniform set of questions/generic tool is considered to enhance the capacity of
organisations to meet the requirements.
9. Suggestions to improve the collection and measurement of carer outcomes
Of those who responded to this question, the most common suggestion was for the
development of a standardised tool/survey for use by funded organisations (to access via
the Department’s website to enable organisations to download and use as needed, and with
the flexibility for organisations to add their own) and/or an increased focus on the
measurement of outcomes, including the reporting of such. These accounted for 53 per cent
of suggestions. For example:
‘Develop a universal carer satisfaction survey to be used by all HACC providers
thereby enabling data to be collated and compared accurately across regions. This
would also reduce the need for agencies to develop their own templates.’
‘Would be good to have a standardised survey tool that all services use. Great for
benchmarking and would save everyone re-inventing the wheel and spending time
reviewing survey tools as we are currently doing. Would also be good to have a
"good survey" practice book/manual that has potential questions grouped under
headings, best ways of obtaining information, i.e. focus groups, written surveys,
telephone surveys or a combination of these.’
‘Perhaps standardise the service survey content to ensure consistency in the
information being gathered by all providers.’
‘A standardised set of questions/questionnaire developed by the Department so that
consistent data can be gathered across agencies. As an organisation that delivers a
range of services to diverse communities this data would be very useful as a broader
measure …’
‘This would assist the sector to have access to rigorous information about service
quality and performance that was sourced consistently. It is of little value to anyone
to ask the questions about service quality and then not use it or feed it back into the
service system.’
‘HACC MDS reporting could be improved to collect information based on consumer
outcomes as opposed to total service delivery hours provided.’
‘Create a Carer Feedback website.’
Around 32 per cent of responses reflected the ongoing (and improved) use of a mix of
survey and less formal verbal/discussion based approaches within their organisations.
‘Multifaceted (qualitative/quantitative) data collection methods.’
‘More regular collection of carer information.’
HDG Consulting Group 58
‘Accessible and easy to use feedback tools. Offer a variety of ways for carers to feed
information back.’
‘Verbal surveys, informal conversations, quick survey tools administered annually -
example via survey monkey.’
Five per cent of responses related to resources and/or staff training. For example:
‘A separate dedicated formal system and process for assessing, monitoring, collecting
and recording carer support needs and outcomes is required. This additional formal
requirement is outside the current resource capacity of PAGs.’
‘Ability for flexibility in targets / spending funds, as long as there is justification and
outcome measures.’
There were also a mix of other comments, such as:
Two respondents identified the ‘most significant change’ technique39.
‘Regarding Support for Carers Program funding targets - I think we need some more
clarification around whether the funded targets are more about hours of service
delivered, the number of carers benefitting or both. I very much look forward to
hearing the outcomes of this survey and any actions that result from it. Thank you.’
These responses indicate that there would be a reasonable level of support for the
introduction of a uniform tool/questions to measure carer outcomes; and that there would
be a voluntary take-up rate by service provider organisations, especially if it met the
following criteria: able to be implemented using a mix of methods (paper, online, verbal);
supported by an associated information resource/guide/instructions for use; able to be
integrated into existing feedback tools and/or added to by individual organisations (while
retaining the core items).
A combination of items that embrace carer quality of life, the quality and sustainability of the
care relationship, and service delivery process items can underpin an outcomes approach
and can be used for multiple purposes.
39 The Most Significant Change (MSC) technique is a tool for collecting, discussing and selecting stories about the significant
changes that people experience as a result of a program. It is designed to be used in conjunction with other evaluation
methods.
HDG Consulting Group 59
Appendix 2
Sample carer outcomes survey - Guidelines for use
Introduction
In Victoria, unpaid family members and friends are the key providers of care to people who
require support in their daily lives. These unpaid care roles are essential to the wellbeing of
the population and the economy. Services such as those provided through Home and
Community Care (HACC) and the Support for Carers Program (SCP) contribute to the
capacity of carers to continue in their care role.
Measuring outcomes is essential to understanding the extent to which services make a
positive difference and contribute to meeting the support needs of carers. Measuring
outcomes informs continual quality improvement and assists organisations to meet the
requirements of the Victorian Carers Recognition Act.
Carer outcomes measures
The ‘Carer supports: timeliness and outcomes measures’ project conducted in 2014
developed a logic model, key domains and seven questions that can be used by a diverse
range of organisations to measure outcomes for carers (see report pages 37-39).
A key recommendation of the project is to pilot test the carer outcomes measures and items
for analysis of reliability, validity and usability purposes. In the interim, organisations may
wish to consider and use this sample survey, developed as part of this project, and subject
to further testing and refinement.
General instructions
The sample carer outcomes survey is for use by HACC and SCP funded organisations to
collect feedback from carers. It is not for assessing needs.
The purpose of the carer outcomes survey is to collect feedback from carers about whether
services and supports are responsive and effective in meeting their needs, supporting their
care role and relationship, and making a positive contribution to their quality of life, health
and wellbeing. The questions have been purposefully designed to seek feedback about
outcomes across seven key domains (information, respect, access, effectiveness, quality,
care relationship, quality of life).
Each organisation should use their knowledge of their consumer group to determine how
best to engage with carers to seek feedback (i.e. which process or combination of
processes, such as discussion, interview, written survey etc is most suitable).
This should include consideration of how best to involve carers who may be reluctant to
reveal personal information or provide feedback. For example, by offering a choice of
feedback options (discussion, telephone interview by independent third party, written
survey, online, feedback via another a family member and so forth) the carer can select
their preferred option and retain anonymity of their responses.
HDG Consulting Group 60
Likewise, when using the survey practitioners may adapt the language and wording to suit
the carer’s literacy skills, presenting situation, cultural background and experience with
services.
How to use the survey
The survey can be used in multiple ways, depending on the processes and practices of each
organisation. The questions can be:
used in everyday conversation with carers (see indicative questions on page 39)
integrated into usual re-assessment, care planning and review processes
integrated into the organisation’s existing feedback processes.
These approaches are unlikely to require any additional organisational resources.
The seven questions have been formatted into two versions of a carer outcomes survey:
Practitioner administered that can be used in an interview or verbal feedback
process, such as during care planning, review or feedback processes (see over)
Consumer administered that can be used in multiple ways, for example by mail-out,
a discussion group, or an online process (see over). This can be printed as A3 size
for ease of reading and to enable adequate room for consumer comments. If being
used in a mail-out process organisations should include a stamped, addressed
envelope for return of the completed survey.
The carer outcomes survey uses a five point rating scale that is the same for all questions:
Never Rarely Sometimes Most of the time Always
There are additional boxes for Unsure and Not applicable.
Space is included for additional comments or feedback.
Because a five point scale has a limited degree of sensitivity, and/or to be congruent with a
specific service provider’s survey or quality practice, some organisations may wish to use a
more sensitive scale. This is acceptable provided the responses can be consolidated into the
five point scale for comparison and benchmarking purposes.
Recording and interpreting the results
Information from the carer responses to the seven questions (whether collected using
discussion or the carer outcomes survey) should be collated, reviewed and analysed to:
inform practice and support with individual carers
inform planning and quality improvement changes to the program or service
combined with other sources of information (i.e. from other feedback or quality
processes) to inform continual quality improvement
contribute to benchmarking purposes with other programs, services, or
organisations.
Further information
For further information about the development and/or use of the carer outcomes questions
please contact the Department of Health: SCP or HACC program in the central office or
regional offices.
Version 1.0 This is a sample survey subject to further testing and refinement.
Carer outcomes survey Staff administered
Purpose: To assist service providers to assess outcomes for carers and
contribute to service improvement.
Consumer
Name:
Date of Birth: dd/mm/yy
Sex:
UR Number:
Age of person/s cared for:
I would like to ask you some questions about the services you have received.
Your answers will help us to improve the services and support we provide to you and other
carers.
Your participation in completing this is voluntary, and we treat your information in the
strictest confidence, in accordance with privacy legislation.
Thinking about the service you have received…
Never
Rare
ly
Som
etim
es
Most
of
the t
ime
Alw
ays
Unsu
re
Not
applic
able
1 Did you receive information when you needed it about
the types of services available?
2 Did the staff listen to you and respect your opinions as
the carer?
3 Were the services available when you most needed
them?
4 Were the services effective in supporting your needs and
goals?
5 (a) Were you satisfied with the services you received?
(b) Were the services of good quality?
6 Did the services you receive help you to continue in your
care role and/or relationship?
7 Did the services make a positive difference to your life?
Note any additional comments or explanations by the consumer
This information collected by: Staff member name:
Position/Agency:
Sign:
Date: Contact number:
Ca
rer o
utc
om
es s
urv
ey –
sta
ff ad
min
iste
red
