Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Real Time Feedback System Pilot for People Using Mental Health and
Addiction Services
Final Pilot Report
September 2014
Provided to the Health and Disability Commission and Advisory Group
TA B L E O F CO N T E N T S
TABLE OF CONTENTS...............................................................................................2
1. INTRODUCTION AND BACKGROUND............................................................3
2. SETUP........................................................................................................................6
3. SITE VISITS..............................................................................................................8
3. DATA COLLECTION.............................................................................................10
4. ADDITIONAL QUESTIONS................................................................................13
5. LANGUAGES..........................................................................................................16
6. SURVEY SOFTWARE...........................................................................................17
6. ANALYTICS.............................................................................................................21
7. SUMMARY OF QUESTION MODIFICATIONS.............................................23
8. CONCLUDING COMMENTS..............................................................................25
2CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
1 . INTRODUCTION AND BACKGROUND
This is the final pilot report provided by CBG Health Research (CBG) on the Real Time Feedback Pilot
for People Using Mental Health and Addiction Services 2014. This report summarises and references
key findings from the first and second qualitative reports and literature review. It is limited to the set
up and running of the pilot, and review of pilot findings to inform recommendations and further
discussion for a pilot extension and/or future roll out.
The Health and Disability Commission (HDC) contracted CBG to develop an electronic survey system
that will allow the public to provide feedback directly to service providers in real-time. Development
included collaboration with service users, their families / whānau, and service providers. We have
also worked closely with the HDC, an Advisory Group and other key stakeholders.
Preliminary work began in December 2013 and implementation began in January 2014, with data
collection using tablet devices commencing in April 2014. A six month data collection period will be
complete by the 28th September 2014. An independent evaluation commissioned by the HDC is
currently in process implemented by Malatest International and is scheduled for completion in
October 2014.
The HDC recruited seven pilot sites to represent a range of provider types, consumers, families /
whānau, and demographic groups. Twenty devices (Samsung Galaxy Tab 3) were purchased and
provided to the sites, some sites purchased additional devices of their own. The table below lists the
participating sites, the provider types utilising the survey and where the devices are located.
3CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
Pilot Site Name Devices Provider Types Device Locations
Counties Manukau DHB 2 Adult Mainstream, Pacific, Community Outpatient
Front desk, outpatient clinics
Waikato DHB 4 Adult Mainstream Inpatient and Outpatient
Front desk, outpatient clinics
Front desk, inpatient ward
Northland DHB 4 Adult Mainstream Inpatient, Outpatient and Community
Front desk, outpatient clinics
Front desk, inpatient ward
Waitemata DHB 7 Adult Mainstream Inpatient, Outpatient, Community and Residential
Front desk, outpatient clinics
Front desk, inpatient ward
Turuki Healthcare 2 Mental Health Support Services within Adult Primary Healthcare
Front desk, outpatient clinics
Offered at home visits
Odyssey House 6 Child, Youth and Family Admission’s, Assessment, Residential and Schools
Offered during individual or group sessions / assessments
Youth Horizons 10 Child, Youth and Family Community, Functional Family Therapy (FFT)
Offered during family therapy sessions
4CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
An online project plan was formally developed using a Gantt chart format, showing progress and
milestones. The plan has been accessible at any time and has been used to share relevant
documentation and communications.
5CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
2. SETUP
ORIENTATION VISITS
Post recruitment, an initial round of orientation visits were arranged by CBG with a HDC
representative attending where possible. The primary purpose was to commence engagement with
the key contacts, outline the project, and collect some preliminary information on systems and
context. The meetings identified and stimulated discussion on pilot site considerations, such as
where to place devices and how to manage their distribution.
INFORMATION PACK
An information pack was developed and distributed at the start of the implementation in January
2014. It provided a system and project overview, hardware specifications and some considerations
to inform placement of devices in service areas.
The HDC developed and released a consumer, family /whānau information sheet which was released
at the same time. This was later replaced with a public leaflet.
QUESTION DEVELOPMENT AND DESIGN
A programme of research and literature review was completed In February 2014. Considerable effort
was made to harmonise the proposed questions with existing work already being undertaken in New
Zealand, in particular by the Health Quality and Safety Commission on collecting data about in-
patient patient experience, based on question sets from the Picker Institute. A range of other
national and international research was also accessed and summarised in a literature review.
Possible domains of patient experience were developed and draft questions proposed.
To engage providers in project development a discussion guide was developed, including a bank of
possible questions. This was sent to pilot sites for internal distribution, using their own processes for
engaging consumers and their families / whānau. The discussion guide gave a summary of the
reasons for including the referenced domains and questions, which aimed to provide a starting point
to stimulate ideas, but not to limit the discussion about possible questions or the scope of the pilot
questions.
6CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
Consultation with consumer and family / whānau groups at each pilot site took place mostly
independently of CBG and elicited a significant amount of feedback, most of which was presented
and discussed with CBG at the first site visits. The feedback is reported in detail using thematic
analysis in the First Qualitative Report and had a significant influence on final question design. In
particular it confirmed the domains of patient experience considered important and the
development of alternative terminology for words identified as ‘labelling’, ‘negative’ and ‘illness
orientated’.
SET UP -RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE
The orientation visits were instrumental in generating enthusiasm and starting the planning process
for the sites. The information pack and leaflets would need to be reviewed and updated for any new
sites coming on board in the future, citing updated questions and titles more relevant to
preventative support services e.g. youth services. We would recommend that this information be
integrated into a larger resource pack which collates all information and training materials
developed throughout the pilot.
Including the consumer and family / whānau in the question design was key in setting up the survey.
Not only did it contribute significantly to the initial core questions, it provided guidance when
reviewing the questions, and also when deciding on individual site questions at a later stage in the
pilot. This process also resulted in an engagement process with consumer groups, many of which
then went on to support and promote the survey throughout the pilot. A longer period of
consultation would have allowed for a more varied approach to giving feedback, targeting those who
would not necessarily feel comfortable in a group situation or in a face to face interaction. For this
reason, we cannot presume that feedback given at this stage of the pilot represents the majority,
but clear themes emerged from those who contributed and all feedback was considered in the
question design. The planned independent evaluation methodology using online surveys will
hopefully allow a greater number of consumers, family / whānau to contribute to the developed
questions.
The demographics of the people and groups providing feedback was not always possible to
determine as this information was not always collected. For any future independent feedback run by
7CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
the sites, it would be advisable that more detailed guidance be provided to the sites requesting they
collect this additional information where feasible to ensure as many ethnicities and ages are
represented as possible.
It could be beneficial to have a means of on-going feedback on the core questions and additional
questions. How this would be implemented and the results considered would need further
discussion and a commitment from the sites, but would be a useful resource to continue to build a
provider, consumer and family / whānau sense of ownership and enthusiasm longer term.
3. S ITE VISITS
FIRST SITE VISIT
All sites were visited in March 2014 and as detailed above, the primary aim of the site visits was to
gather in feedback and present to the group, allowing further opportunity to contribute and
validation of existing feedback.
Other agenda items covered at the site visits are listed below. The initial plan to provide
demonstration and training on installation of the survey was in most cases not appropriate. The
attendees for the first site visit had a greater consumer focus and the appropriate person/s to be
taking responsibility for this were not always present, as well as practical constraints i.e. lack of Wi-Fi
at the meeting venues. There was also limited time and some items were given less attention than
ideal. Despite earlier communications detailed above, a lot of unplanned time was spent orientating
people to the survey and explaining the purpose of the pilot. The short time frame did not allow
internal communications to occur as effectively as they should.
Feedback from consumer, family / whānau representative’s and pilot site representative’s
on sample questions/patient domains as per discussion guide
Discussion on information pack
Distribution of allocated devices
Demonstration of sample analytics for feedback and discussion
SECOND SITE VISIT
8CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
This round of site visits allowed for at least six weeks of data collection and primarily aimed to gather
in feedback on the first set of core questions, response options and initial analytics as well as the
opportunity to identify additional questions specific to their services. Please see Second Qualitative
Report for feedback analysis and p.20 ‘Summary of Question Modifications’ for the revised
questions. All sites were then contacted to ensure they understood the purpose of the meeting,
discuss who should attend and to encourage internal discussion on any additional questions prior to
the meeting. The main agenda items covered are listed below.
Nature of core questions including, other questions or question types that should be
included
Possible additional questions to be added
The way that response options were presented
Managing the device/updating the survey and its use within the service
Usefulness of analytics
PILOT SITE WORKSHOP
Additional to the original project plan, a workshop was scheduled for mid-way through the pilot in
July. Two to three representatives from each site and all advisory group members were invited to
attend. This was primarily arranged and facilitated by the HDC with CBG collaboration into the
agenda and content. The workshop received good feedback and allowed attendees to talk directly
with the HDC and advisory group members, and to confirm their feedback was being heard. Giving
opportunity for further feedback into the revised questions, was a valuable exercise which modified
the questions further and promoted a sense of ownership.
SITE VISITS- RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE
For the pilot process, the site visits were invaluable, not only on a practical level covering the agenda
items listed, but to fully engage the sites. The need for future site visits would depend on the
purpose, now that the questions have been modified. If further detailed qualitative research and
evaluation was required then future site visits would be recommended, but on a wider scale would
be costly and time consuming. The site visits also limit discussions to those attending. Methodology
9CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
that captures feedback on a wider scale e.g. online, would be a more cost effective means to
continue this process longer term for a larger number of sites.
Maintaining the face to face interaction via continued site workshops maybe a way forward, to allow
the sharing of experiences, ideas and ensure a continued connection with the HDC, developers and
advisory group. This would also allow for key decisions to be made that impact all sites and where
useful to hear the perspectives of other organisations i.e. data ownership or strategies that have
worked to increase survey uptake longer term.
3. DATA COLLECTION
FIRST SET OF CORE QUESTIONS
The First Qualitative Report analysing the provider, consumer, family / whānau feedback was
presented to attending advisory group members and the HDC on teleconference. A core question set
was subsequently proposed with further opportunity for feedback given to all advisory group
members. Some minor changes were made to reflect the feedback and a final core question set was
approved by the HDC. See table p.23 ‘Summary of Question Modifications’.
COMMENCING DATA COLLECTION
The first version of the survey was released on the 28th April. Instruction on downloading the survey
initially was provided and telephone support. Once the survey was loaded this would not need to
reoccur throughout the pilot, future versions required only an update, which is a much simpler
process. It took up to 4 weeks before all sites were actively returning results and up to 7 weeks for all
devices to be in field. The majority of the sites were returning some data within a few weeks.
MANAGING THE DEVICE AND TRAINING
1 0CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
At the early stages of the pilot, wider internal staff training using the ‘train the trainer approach’ was
limited due to time constraints, but in reality there was very little ‘training’ involved. The need was
more around orientation and awareness. As the pilot progressed the term ‘training’ was avoided as it
painted a picture of time consuming formal training that was not required. Other issues arising
detailed below:
Poor access to Wi-Fi
Internal staff orientation and training
Management decisions on where to place the devices
Ownership of the devices
A significant arising issue was the need to upload results completed offline manually, currently
automatic upload only occurs when results are complete online. CBG were not aware that this was a
process that would need to be done manually and therefore, this was not fully communicated to sites
initially. This resulted in results being left on devices and staff being unaware. The process to
manually upload is very simple, but a task to remember to complete on a regular basis. It also means
results can be lost if a survey is updated and old off-line results are not manually uploaded.
With some exceptions, most sites had not fully established ownership of the devices and person/s
responsible for updating and managing the devices when data collection commenced. The fast pace
of the pilot left sites unsure as to where to place the devices right up until the release of the first
survey, this made identifying the key people to manage the devices a last minute decision for some
and a significant amount of support was required at the early stages. As the pilot progressed device
ownership improved, with key person/s identified to be responsible for managing all the devices.
This has worked much more effectively with the latest version of the survey being updated quickly
and easily with minimal intervention.
DATA COLLECTION- RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE
1 1CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
Factors influencing the sites ability to roll out the survey quickly was mostly influenced by barriers
typical and often unavoidable within organisations, staff sickness, resources etc…. The tight timeline
of the pilot required pressure to be placed on the sites to get the data coming in as quickly as possible
to inform the initial stages of the pilot and allow further evaluation of the core questions. Under
normal circumstances with less time constraints, less support contact would be required, reducing
costs in bringing other sites on board.
Other issues arising discussed above, such as ownership, where to place devices, training etc… could
be detailed as part of the larger resource pack, provided post initial recruitment, increasing
awareness at the early stages and thus minimising their impact as barriers to implementing the
survey in their chosen service areas.
Although some advice could be provided on the issue of Wi-Fi availability, detailing strategies used by
other pilot sites, this issue would need to remain with the site to manage. Unfortunately it does
impact how ‘real time’ the results are for some services who are not able to access Wi-Fi on a daily
basis, but is an organisational issue different for each service that will need to be managed internally.
The steps to load the survey initially, and update in the future are simple enough for a person with a
novice level of IT literacy to complete, by following step by step instructions, with telephone support
if required. We would recommend that although a higher level of IT support within each organisation
would be required to manage the technical aspects of the device, e.g. passwords, security etc… a
person/s should be identified at the service level as well. The time taken to gather in devices and
have an ‘IT’ person update and send out again is wasteful, as the process of updating is simple and
can be completed at the service level easily.
The functionality for results completed offline to automatically upload when a Wi-Fi connection is
made, is a recommended software development task. We would consider this development essential
for a wider roll out. Once this functionality is available, the staff in service areas would only need to
update surveys when required.
The level of support required for the pilot could be significantly reduced on a wider scale, by ensuring
sites have identified staff, to take ownership over these tasks. Instructions at each step were emailed
out and feels fragmented. The person/s identified at the early stages of data collection have also
changed as the pilot progressed. Before the end of the pilot period the instructions for the task’s
1 2CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
listed below will be collated, updated and provided in one pdf document to all sites. This can be
updated in a wider roll out situation to reflect any software changes and incorporated into a manual
suggested earlier.
Loading the survey software and settings
Manually uploading offline results
Updating the survey
Managing setting on the device
Other considerations e.g. anti-virus, security etc…
4 . ADDITIONAL QUESTIONS
All sites were given an opportunity to add in additional questions specific to their service area. They
were all provided with a discussion guide to support sites in the design and implementation of
optional additional questions following on from the core question set. The content of this guide is
mostly reiterated below. In order to accommodate these additional questions a new survey was
required for each site.
Sites were not expected to come up with the ‘perfect’ question. They were mostly able to identify
the domains of patient experience, service or intervention to be evaluated, CBG assisted at the
second site visits and in collaboration with the HDC to design the appropriate question/s. Some sites
were very clear on their questions and needed little or no support in developing.
The bank of questions provided in the discussion guide were mostly selected from two sources:
Discussion guide of sample questions used to facilitate feedback when designing the core
questions
Suggested questions from the first pilot site visits and consumer, family / whānau groups
Some questions were omitted or amended to reflect the feedback given to date, and remove repeat
questions. These questions provided a starting point and were amended further to perfect the
wording and ensure relevance to the service.
1 3CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
Where some questions were considered irrelevant or too ‘medical’ to be applied to all pilot sites for
the core question set, we still included in the bank of questions. For sites wishing to evaluate an
inpatient service or treatment clinic, for example, they may well be useful. The word ‘hospital’ could
be replaced with residential centre, clinic or other service. Likewise a ‘group’ or therapy’ could be
replaced with a specific name.
There were a number of limitations set, listed below:
A maximum of 2 additional questions
All questions will be designed in collaboration with CBG and pilot sites and will ideally take
into account consumer, family / whānau feedback recently provided
All questions will need to be approved by CBG in collaboration with the HDC
Sites can have only 2 surveys running at the same time, it is not possible for the pilot to have
different additional questions on different devices.
- Core question set only.
- New survey with the core questions set plus additional questions.
Response options for the additional questions will be the same as the core questions, using
the smiley faces rating scale or selecting from a pre-determined list. Free text responses will
not be available.
ADDITIONAL QUESTIONS- RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE
We would recommend the option to add in additional questions that are outside of the national core
set remains. Developing a core set of questions relevant to a range of organisations and service
areas has been challenging, and although they have been developed to be as applicable to all, for
some sites especially those outside of mainstream secondary care services, they may never be ideal.
Additional questions designed with the specific service or client group in mind, targeting key
programmes or initiatives will be a significant motivation for organisations to stay engaged with the
survey and drive change. It also allows for results from the core questions to be explored in more
detail, allowing services to target areas of need more specifically.
1 4CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
The option of additional questions creates the need for a survey per site, which comes at an
additional cost. A survey containing the core questions only can always be running and some sites
may prefer to have this, with no additional questions. The advantage of a separate survey per site, is
customisation with their own logos, tailoring the wording on the front pages to name their
organisation and other surrounding text. How much control is retained over the level of
customisation would need to be further discussed.
The limitations placed on adding in additional questions listed above we consider reasonable and
should mostly remain, with the exception of the following:
A maximum of 2 additional questions
All questions will be designed in collaboration with CBG and pilot sites and will ideally take
into account consumer, family / whānau feedback recently provided
Once a new survey is created there is no limit to the number of questions and therefore, no
additional cost incurred per question, but the feedback in the first round of qualitative research was
clear that consumers wanted the survey to be short, especially if being asked to complete each
episode of care. It would be worthwhile waiting to see if the independent evaluation identifies
length as a barrier to completing the survey, given it has increased since the first version. There may
be some scope to extend beyond two questions.
We would recommend that the final choice of question be approved by the HDC for inclusion, but
control over this should be limited, mainly ensuring anonymity is maintained. The survey
management team could provide an advisory role in developing the questions if required, using the
feedback from the First and Second Qualitative Reports and other experience in survey question
design. The bank of possible questions listed in the pilot could be reviewed and provided to sites as a
guide.
5 . LANGUAGES
The surveys have been made available in 7 different languages
English
1 5CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
Māori Tongan Samoan Chinese (simplified) Korean Hindi
The software offers different languages, but NZ International was used, which offers a high quality
translator services with extensive quality checks. The languages were then programmed using skip
logic. The translations are limited to the questions only. The front page and text introducing the
survey, as well as the final page remains in English only, due to software restrictions.
The languages were not made available until the third version of the survey at the end of July, due to
continued development of the questions. There was a significant amount of work and time involved
with lining up translators to be available to translate the revised set of core questions and additional
questions within the time frame proposed. This quick turnaround may not be guaranteed in the
future and time would need to be allowed for any additional translations in the future to occur.
This is going to be particularly relevant if sites wish to add in or change additional questions. Having
them translated into multiple languages means they cannot have the questions implemented as
quickly as if English only. There is also a significant cost as most quality translator services work on a
minimum word count for the minimum base charge per language. The cost to get one question
translated will be the same as approx. ten or more.
LANGUAGES- RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE
Translations for any further changes to the core questions will become less as the final changes are
made and will not, therefore, incur a significant issue or on-going cost. Any further translations for
additional questions could be implemented in batches and set opportunities be given for question
review. It is not sensible or cost effective for one or two questions to be sent off for translation ad
hoc. The other option has already been briefly discussed with the HDC and advisory group;
translating the bank of questions mentioned earlier in this report. In developing a final list the sites
could have opportunity to add other questions. Any questions developed outside of this list would
need a process of development, authorisation, and translation. This process could be managed in its
entirety by the sites, and the questions provided for programming.
1 6CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
Ideally the survey front page should allow a language selection and then all text, information as well
as the questions continue and finish in that same language. If this is possible using the current survey
software would need further discussion, but we would recommend this as a future development.
6. SURVEY SOFTWARE
Having reviewed the NHS technical solution and a range of other options, detailed in our proposal,
including building a system from scratch, our preferred option at that time was a service provided by
another local New Zealand company. The company was “Kinross” and the product is called “AMS”.
Another New Zealand company called droidSURVEY came to our attention at a late stage, they
offered all the same desired functionalities listed below, and had a significant advantage over AMS
for increased customisability and ease of use, for this reason we decided to adopt the droidSURVEY
platform.
https://www.harvestyourdata.com/
FEATURES
Does not require a particular operating system
Device independent
Collects data both online and offline, so that if a Wi-Fi or mobile broadband connection fails
or is unavailable data can is stored and uploaded when the connection is restored.
Collects a wide range of data types including single and multi-choice selections (sliding scale
or smiley face selection), text, numbers, email capture, images, photos, videos, signatures,
barcodes and locations.
Provides skip logic functionality without the need for any coding language (to implement
questionnaire branching).
Multiple surveys can be loaded on to a device and you can quickly switch between them
when it's offline, without losing any of your captured results.
Each purchased survey can be loaded on an unlimited number of iOS and Android devices,
no charge per device or question
1 7CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
Can manage all devices centrally
Supports data protection that is implemented at the software level and works with the
hardware and firmware encryption to secure data collected in the event that a physical
device is stolen or lost.
All export options needed for downloading data collection results in the SPSS format or in
CSV (comma separated value) format, which is 100% compatible with Microsoft Excel.
Automate the downloading of results with the ‘Developer API’ and integrate data with other
software.
Real time charts providing analysis of results that are displayed in a customized dashboard,
automatically refreshing every minute with the latest questionnaire data. (this link can be
embedded in other online sites, newsletter or displayed as a live link in waiting rooms)
You can set up surveys to record time stamped GPS co-ordinates for each survey result
No programming is required to create your survey or form.
Automatically restart the survey after a certain amount of idle time that is specified. You can
choose to display an optional warning message prior to the survey restarting.
Control whether the device saves or discards partial results.
User interface that automatically proceeds to the next question after a user answers a single
select question. Users can quickly move to the next screen without having to select the next
screen button.
DEVELOPMENTS MADE
There were a number of developments we needed initially before the surveys could be
implemented.
1 8CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
Landscape option
Customised response option displaying 5 smiley faces on one screen
RECOMMENDED FURTHER DEVELOPMENTS
The table below summarises recommended software developments that are needed to improve the
overall look of the survey and ease of use from a management perspective and at the user level. The
cost and time frame for these developments to be done would need further discussions with
droidSURVEY and at this stage of the pilot a further review of other systems including developing a
tailored system from scratch that meets all the requirements.
Requirement Details
Improved editing capability Increased editing capability for the 'look' e.g.
different font sizes, colour on different screens,
currently selections are the same for all. Also
ability to change positioning/ spacing.
1 9CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
Original option available Current option
Improved look for landscape Although the survey was developed to use
landscape mode, many other features designed
originally to look good in portrait mode were not
changed. e.g. positioning of multi select to the
right of the question rather than underneath, and
text for first and final pages split across the page.
Preview screen available for landscape For the same reason above, this functionality only
displays in portrait. When programming it would
save a significant amount of time to be able to
preview the survey on screen, rather than
repeatedly loading to the device to test.
Manually remove results from droid database The real time results on the public page accesses
the droid database before the data is exported to
our own systems for further analysis. Occasionally
false results are entered by sites for demo
purposes and we cannot manually remove
without asking droidSURVEY
Automatic upload of completed offline surveys
when Wi-Fi connection is made
This was a feature we originally thought possible
and is a priority development for the future.
Should be the default option that can be turned
off if required.
The next button for text response questions
needs to be more prominent
Currently there needs to be instruction to tap the
‘Next’ button as it’s not immediately obvious.
Combining droid databases The real time results on the public page start
again when a new survey is published, you cannot
combine the databases.
Translations available for all survey screens not
just the questions
Option on the front pages to select language and
all screens to follow in that language
6. ANALYTICS
Feedback on the nature of reporting was minimal in the early stages. All sites were given the
opportunity and provided with sample analytics for discussion at the site visits. Generally feedback
on what we were presenting and proposing seemed adequate at that stage of the pilot. Sites were
2 0CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
not ready to address analytics in detail at this stage. The focus was more on the questions design
and where to place the devices within services.
For others lack of experience in analysing data or looking at graphs and charts made consideration of
the analytics overwhelming and there was a reluctance to offer any feedback at the site visits. In
most cases the person who would be responsible and more interested in the analysis was not
present.
The public results update every minute and can be viewed www. patientexperiencesurvey.co.nz
With the latest version, sites now have their own surveys and the live overall results per site will be
displayed in the same format, by going to the same website and clicking on the heading ‘Public
Results per Organisation’ and selecting their organisation. Any site that does not want their results
accessible to other sites can opt to have a separate link which they can embed in their own website.
The sample analytics for the organisations were presented to the advisory group and the HDC and
were implemented once a reasonable amount of data was returned from each site to ensure
anonymity. We have used an analytic software solution called Yellowfin, presented in real time
dashboards via a secure log for each site. Reports can be easily exported, emailed and printed. Links
can also be created to allow users to embed in other sites, newsletters or other means of circulating
the results.
At the second site visit feedback was gathered on the analytics and at this stage a number of
suggested improvements and requests were identified. Having a visual starting point made this
process easier to conceptualise. The majority of these improvements have been made and we will
continue to work through the rest for completion before the end of the pilot period.
ACCESS TO RAW DATA
Currently the sites have not been given the raw data for any of the questions. Concerns around risk
management of the raw data for the text response questions led to this being withheld initially. The
2 1CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
following paragraph was communicated to the public in the leaflet and is stated prior to the text
response question on the survey screen.
“Any additional comments as text feedback will not be made fully available to services, but
presented in themes analysed automatically by a word search using computer software. This is to
ensure anyone completing the survey or staff members are not identified. Any requests for assistance
will not be viewed by services and cannot, therefore, be followed up. If you feel you are in distress or
need help please contact your local mental health services, GP or nearest Accident and Medical”
This makes this question on screen look very crowded, but it was decided that this was required until
further decisions were made on this issue.
ANALYTICS - RECOMMENDATIONS AND CONSIDERATIONS FOR THE FUTURE
Yellowfin would continue to be our preferred analytical software solution and can be expanded to
allow users with a higher level of technical expertise to utilise additional functionality, in particular
“self service” business intelligence, including the creation of site specific reports – tables, reports
and dashboards. Further customisations can be made very easily to the existing suite analytics
easily as required in the future.
Discussions on access to the raw data has already progressed and final decisions on this will be made
by the HDC following further discussion with participating sites and management of risk. The
advisory group consensus indicates that the raw data, in particular for the text response questions
can provide very rich and useful information that can be instrumental in driving change.
The use of the results and how effective they will be at driving change should be a key focus
following on from the pilot. Working with sites to encourage the dissemination of the results at
multiple levels should be a key priority. Included in this, should be access to the results for the public
to keep the people motivated to complete and see how the results are being used to improve the
services. The trend data will allow an overview of the uptake rates week by week, overall and across
services. This information should help inform sites as to where their devices may be best placed and
enable them to continue to improve survey administration and delivery.
2 2CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
To date, sites have not described an unwillingness to complete the survey more than once, although
there are real concerns that people will become less enthusiastic as time goes on. A longer period of
data collection is required for this to be evaluated. The survey is still progressing and recent
improvements to the questions may increase the likelihood that people will be willing to complete
more than once, or on a regular basis. Keeping up provider interest in administering the survey is
also going to be a challenge and sites have already started to think about moving devices around
more often, and advertising results. Advice on long term management of real time surveys will likely
be sought post pilot period and each approach will be dependent on the organisation, the services
and even personalities involved, but the analytics will play an important part for all.
7. SUMMARY OF QUESTION MODIFICATIONS
The table below show the original core questions implemented in the first version of the survey and
the modified questions implemented in the current version at the end of July. Please note question
five was omitted from the first version in error. A second version was published in June to include
this question, an evaluation question and an opening question asking if they are answering as a
consumer or family / whānau member. Additions to version2 listed below.
1. I am completing this survey as… a consumer or service user or person seeking support family, whānau or friend
5. Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement….
I have the support I need for the future
12. Did the questions in this survey cover things about the service that are important to you? Yes completely Some things Not really Not at all
13. What else would you like to have been asked about? (text option if ‘Not really’ or ‘Not at all’)
First Set of Core Questions, April 2014 Modified Core Questions, July 2014
The people I see within the service value my opinions and are compassionate, non-
Thinking about your most recent experience with the service/people who support you, how much
2 3CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
judgemental and respectful of my cultural needs.
do you agree or disagree with the following statement….
I feel respectedI am sufficiently consulted and involved in decisions about services to meet my identified needs
Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement….
I am involved in decision makingThe services I use communicate well with each other and are fully informed about my needs.
Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement…. (Don’t know option)
The people I see communicate with each
other when I need them to
My family / whānau or someone close to me has been given information and support to be involved in my care and recovery.
Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement…. (Additional Option: I didn’t want them to be involved)
My family / whānau are given information and encouraged to be involved
The services have supported me and my family move forward in recovery.
Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement….
I have the support I need for the futureI would recommend this service to friends and family if they needed similar care or treatment.
Thinking about your most recent experience with the service/people who support you, how much do you agree or disagree with the following statement…. (Optional)
I would recommend this service to friends and family if they needed similar care or treatment.
Is there anything you want to say about your recent experience with the service or anything you think we can improve on?
Thinking about your most recent experience with the service/people who support you… (Optional)
2 4CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL
What don’t you like?
Thinking about your most recent experience with the service/people who support you… (Optional)
What do you like?
8. CONCLUDING COMMENTS
This has been a very rewarding project in a number of ways. There have been a number of technical
challenges that have been interesting and satisfying to resolve. The idea of measuring and acting
upon real time patient experience data has been universally supported by a wide range of providers.
During site visits, advisory group meetings and in the meeting of all participants, we have observed
stakeholders develop ideas for leveraging the potential of the infrastructure, and observed support
for widespread adoption of the system within organisations.
Having youth services and primary care on board as pilot sites has challenged the core questions
significantly to be relevant outside of mainstream mental health services. Consequently the
developed core questions are now more applicable to wider range of services.
CBG is looking forward to the next stage of this project and expanding the number and range of
providers using this tool. One challenge will be developing a coordinated management structure for
real time feedback, and defining the role of this system in relation to other national services.
2 5CBG HEALTH RESEARCH LTD . F INAL P I LOT REPORT HDC REAL T IME FEEDBACK P I LOT SEPTEMBER 2014 FINAL