Ju ly 2014 The F riends and Family Test - Ipsos · Jo 13-0 Ju T Qu b number ; Title 89420-01 | FINAL ly 2014 he F alitative of document : D v1 | PUBLIC rien researc raft status ds

Jo

13-0

Ju

TQu

ob number ; Title

089420-01 | FINAL

uly 2014

The Fualitative

e of document : D

v1 | PUBLIC

Friene researc

Draft status

nds ah

and Fammily Test

t

1

The Friends and Family Test: Qualitative research

13-089420-01 | FINAL v1 | PUBLIC

© 2014 Ipsos MORI – all rights reserved.

The contents of this report constitute the sole and exclusive property of Ipsos MORI. Ipsos MORI retains all right, title and interest, including without limitation copyright, in or to any Ipsos MORI trademarks, technologies, methodologies, products, analyses, software and know-how included or arising out of this proposal or used in connection with the preparation of this report. No license under any copyright is hereby granted or implied.

The contents of this report are of a commercially sensitive and confidential nature and intended solely for the review and consideration of the person or entity to which it is addressed. No other use is permitted and the addressee undertakes not to disclose all or part of this report to any third party (including but not limited, where applicable, pursuant to the Freedom of Information Act 2000) without the prior written consent of the Company Secretary of Ipsos MORI.

This work was carried out in accordance with the requirements of the international quality standard for Market Research, ISO 20252:2012, and with the Ipsos MORI Terms and Conditions which can be found at http://www.ipsos-mori.com/terms. © Ipsos MORI 2014.


13-089420-01 | FINAL v1 | PUBLIC

Contents

1 Executive Summary ................................................................................................. 1

2 Background and context ........................................................................................ 8 2.1 The Friends and Family Test ...................................................................................................... 8

3 Our approach ....................................................................................................... 11 3.1 Online feedback ...................................................................................................................... 11

3.2 Case studies ............................................................................................................................ 13

3.3 Stakeholder and expert interviews ......................................................................................... 17

4 Overall views of the Friends and Family Test........................................................ 20 4.1 The importance of collecting feedback on patient experience ........................................... 20

4.2 Perceptions of the FFT .............................................................................................................. 22

4.3 Implementing the FFT .............................................................................................................. 24

4.4 Knowledge and understanding of the FFT .............................................................................. 25

4.5 FFT and patient experience measures ................................................................................... 26

4.5.1 Integrating FFT with existing measures ......................................................................................... 27

5 The Friends and Family Test Methodology ........................................................... 30 5.1 The question wording .............................................................................................................. 31

5.1.2 Opinions of the response codes ................................................................................................... 33

5.1.3 The FFT question for vulnerable groups ........................................................................................ 34

5.1.4 Focussing on specific wards ......................................................................................................... 34

5.1.5 The follow-up question was particularly valuable ....................................................................... 34

5.1.6 Considerations when answering the FFT question ....................................................................... 36

5.2 The scoring system .................................................................................................................. 36

5.2.1 Reactions to the score .................................................................................................................. 37

5.2.2 Participants found the scoring system confusing and difficult to understand ........................... 37

5.2.3 Many thought the scoring system was unfair .............................................................................. 39

5.2.4 Participants thought the scoring system restricted choice ......................................................... 40

5.2.5 Preferences for calculating the FFT score .................................................................................... 40

6 Data collection ..................................................................................................... 43 6.1 Data collection methodology ................................................................................................. 44

6.1.1 Primary data collection methods ................................................................................................. 45

6.2 Choosing a data collection method ...................................................................................... 46

6.2.1 Familiarity with the methodology ................................................................................................. 46

6.2.2 Influence on the response rate ..................................................................................................... 47

6.2.3 Ease of use .................................................................................................................................... 48

6.2.4 Capturing honest responses ......................................................................................................... 49

6.2.5 Capturing a representative cross-section of patients ................................................................. 50


13-089420-01 | FINAL v1 | PUBLIC

6.2.6 Real time feedback ....................................................................................................................... 50

6.3 Making FFT data collection work ............................................................................................ 52

6.3.1 Making the FFT routine .................................................................................................................. 52

6.3.2 Engaging staff in the FFT ............................................................................................................... 53

6.3.3 Assigning responsibility ................................................................................................................. 53

6.3.4 Introducing a Trust-wide strategy ................................................................................................. 54

6.4 The need for flexibility in data collection ............................................................................... 56

6.5 When should the FFT be asked? .............................................................................................. 57

6.6 The eligible population ........................................................................................................... 58

6.6.1 Logistical constraints ..................................................................................................................... 59

6.6.2 Sensitivity and capability .............................................................................................................. 60

6.6.3 Deliberately not asking the FFT ..................................................................................................... 61

6.6.4 Asking the FFT more than once .................................................................................................... 62

6.7 Implementation and guidance ............................................................................................... 62

6.8 Using suppliers ........................................................................................................................ 64

6.9 The A&E challenge .................................................................................................................. 65

6.9.1 Volume and turnover .................................................................................................................... 66

6.9.2 Type of patients ............................................................................................................................. 67

6.10 Solutions to A&E data collection issues .................................................................................. 68

6.10.1 Greater publicity ....................................................................................................................... 68

6.10.2 Engaging medical staff ............................................................................................................. 69

6.10.3 Distributing the FFT at admission .............................................................................................. 69

6.10.4 Use the most appropriate methodology .................................................................................. 69

7 Communicating the results ................................................................................... 72 7.1 Sharing the results ................................................................................................................... 73

7.1.1 With frontline staff .......................................................................................................................... 73

7.1.2 With the board ............................................................................................................................... 74

7.1.3 With patients and the public ........................................................................................................ 75

7.1.4 Presenting the feedback from the follow-up question ................................................................. 79

7.2 What do patients and the public want? ................................................................................. 80

7.2.1 Intuitive and easy to understand .................................................................................................. 81

7.2.2 The need for supporting information ............................................................................................ 82

8 Using the data ....................................................................................................... 85 8.1 Who uses the data? ................................................................................................................. 85

8.2 Service improvement, not comparison .................................................................................. 86

8.2.1 Comparability is not just about methodology ............................................................................. 88

8.3 The FFT and patient choice ..................................................................................................... 89

8.3.2 Trusts and staff doubted patients were using FFT ........................................................................ 91

8.4 Use of the FFT data in commissioning ..................................................................................... 92


13-089420-01 | FINAL v1 | PUBLIC

8.5 Using data from the follow-up question .................................................................................. 93

8.5.1 Boosting morale and pride ........................................................................................................... 93

8.5.3 A need for more thematic analysis .............................................................................................. 96

9 Conclusions ........................................................................................................... 98

10 Appendices ........................................................................................................ 101 10.1 Online feedback form ........................................................................................................... 101

10.2 Guidance for case study Trusts ............................................................................................. 111

10.3 Discussion guide for case study staff interviews .................................................................. 114

10.4 Discussion guide for case study frontline staff interviews.................................................... 121

10.5 Discussion guide for patient and public groups .................................................................. 128

10.6 Stimulus material for patient and public groups .................................................................. 135

10.6.1 The FFT question ...................................................................................................................... 135

10.6.2 The scoring system .................................................................................................................. 135

10.6.3 Comparing scores .................................................................................................................. 136

10.6.4 Alternative scoring set ............................................................................................................ 137

10.6.5 Presenting the FFT score set .................................................................................................... 139

List of Figures

Figure 1.1 ---- How the FFT score is calculated ......................................................................................... 3 Figure 3.1 ---- Response rates for the online form ................................................................................... 13 Figure 3.2 ---- Case study sample selection ........................................................................................... 15 Figure 3.3 ---- Profile of stakeholder interviews ....................................................................................... 18 Figure 4.1 ---- The impact of the FFT on patient experience ................................................................... 22 Figure 4.2 ---- Successful implementation of the FFT .............................................................................. 24 Figure 5.1 ---- How the FFT score is calculated ....................................................................................... 37 Figure 5.2 ---- Understanding of the FFT score ......................................................................................... 38 Figure 6.1 ---- The different data collection methods used by Trusts ..................................................... 44 Figure 6.2 ---- Primary data collection methods ..................................................................................... 46 Figure 6.3 ---- Promotion of the FFT at the main reception ..................................................................... 56 Figure 6.4 ---- Happy face token collection ............................................................................................ 61 Figure 6.5 ---- Use of external suppliers for different tasks ..................................................................... 64 Figure 6.6 ---- Ease of set-up of the FFT .................................................................................................... 66 Figure 6.7 ---- Receptionist with an FFT t-shirt .......................................................................................... 69 Figure 7.1 ---- Publicly available board report dashboard .................................................................... 75 Figure 7.2 ---- Poster advertising FFT results ............................................................................................ 76 Figure 7.3 ---- Ward board using a combined percentage alongside other metrics ............................ 77 Figure 7.4 ---- Ward board bar showing the detailed FFT results ........................................................... 77 Figure 7.5 ---- How the FFT score is calculated ....................................................................................... 78 Figure 7.6 ---- Presentation of the FFT score in line with the guidance .................................................. 79 Figure 7.7 ---- ‘‘You said, we did’’ display ............................................................................................... 80 Figure 8.1 ---- Audiences using the FFT data ........................................................................................... 86 Figure 8.2 ---- Ways in which the FFT data is used .................................................................................. 87


13-089420-01 | FINAL v1 | PUBLIC

1 Executive Summary

The Friends and Family Test: Qualitative research 1

13-089420-01 | FINAL v1 | PUBLIC

1 Executive Summary This report presents the findings of qualitative research into the Friends and Family Test (FFT) conducted by the Ipsos MORI Social Research Institute on behalf of NHS England.

Six months after the FFT was introduced for Inpatients and Accident and Emergency (A&E) patients, NHS England carried out a review of the FFT. The review had the following specific purposes:

To assess the implementation, publication, use and reception of FFT during the first six months of national data collection, focusing on what has worked well and what might be improved;

To collect and present evidence of best practice across the settings;

To judge whether the test in its present incarnation matches its original ambitions, and, if necessary, to highlight any changes that would be required to fulfil all of these functions, specifying any associated risks or costs; and

To ensure that future NHS England guidance on FFT is evidence-based and reflects best practice.

The qualitative research undertaken by Ipsos MORI formed one part of NHS England’s review. It comprised several distinct elements:

Detailed case studies conducted in nine Trusts incorporating interviews and discussion groups with senior staff, frontline staff, commissioners, patients and the public, as well as a chance to observe the FFT being used in practice;

20 telephone and face-to-face depth interviews with experts and other stakeholders; and,

A structured online feedback form sent out to all 156 NHS/Foundation Trusts in England; 95 responses were received, representing a response rate of 61%.

The key findings emerging from this qualitative study are described below.

There was real enthusiasm for collecting patient experience data. The FFT was mostly, although not universally, seen to have an important role to play in embedding this and encouraging cultural change, and was seen to have increased the emphasis on patient experience within Trusts. Almost four in five Trusts responding to the


13-089420-01 | FINAL v1 | PUBLIC

online survey said the FFT had increased the emphasis on patient experience (78%).

The FFT was felt to offer the best yet approximation of real time patient feedback. It allowed more immediate feedback than national surveys and could be mined for ward-level detail.

Triangulation with other measures is necessary to achieve an understanding of the whole picture. The FFT was not useful for senior staff and commissioners without being seen alongside other measures. FFT was more useful for wards when displayed alongside other measures (for example, other patient experience measures or metrics such as pressure ulcers and falls). Patients and the public also preferred presentations of the FFT data that placed it alongside other information.

There were differing opinions about the appropriateness of the question wording. Some of those we spoke to believed that the notion of “recommending” services was appropriate. Consultants spoke of how this matched the way they talked to patients, while patients themselves understood this as how they themselves spoke about hospitals to their friends and family. That said, others thought the concept of recommendation did not work in all healthcare settings as it was interpreted at times as recommending that a friend or family member be ill or need care. There were also suggestions that the use of the word “recommend” might be less suitable to emergency settings. Staff and patients alike believed that no-one would recommend that someone should go to A&E. Additionally, the wording was taken to imply patients had a choice about which A&E departments they could attend, a choice that few patients or staff believed existed.

The free text follow-up question was believed to be the most valuable aspect of the FFT. Almost all Trusts were asking a follow-up question that allowed patients to explain their answer in their own words. It was seen as vitally important that Trusts were able to identify the “why” behind FFT responses. The free text question provided richer, more useable data. Stakeholders, in particular, pointed out that this is how the net promoter concept is used in the commercial world. Much of the feedback from the free text was positive, providing a much appreciated morale boost for frontline staff.

The calculation of the FFT score was disliked and confusing. This was especially true for frontline staff and patients. The use of a net score was less easily understood than a percentage or star rating. All questioned why “likely” responses were dismissed and neutral responses were classified as detractors. It gave rise to cynicism


13-089420-01 | FINAL v1 | PUBLIC

about manipulation of data; some staff felt it was doing the NHS down, while the public worried that data could be manipulated to mislead them. Senior staff and stakeholders also disliked the FFT score. The following chart illustrates how a score would be calculated in a fictional Trust.

Figure 1.1 ---- How the FFT score is calculated

The FFT was primarily being used by Trusts as an internal tool to drive service improvement. It was primarily seen as an internal tool rather than a public facing instrument by Trust staff, while patients and the public instinctively assumed that the question was being asked to help identify and drive service improvement. It was perceived by Trust staff – both frontline staff and senior staff – as being best at providing a reliable time series to assess performance rather than as a Trust-to-Trust comparison. The free text follow-up question was seen as particularly valuable to frontline staff for identifying issues and then publicising actions taken as a result (“you said / we did”). Staff tended to be aware of the issues raised through the free text question, but having the written evidence provided an added impetus to address the problem. It also provided evidence highlighting issues that staff themselves had identified, but needed more ammunition to persuade senior colleagues to act upon.

The FFT data was not perceived to be strictly comparable between Trusts. Staff, commissioners and stakeholders insisted that the FFT data should not be compared across Trusts. This was primarily due to their awareness of mode effects, as a result of different data collection methodologies, varying response rates, and

Score represents everyone who said they are extremely likely to recommend MINUS the respondents who would not recommend (“neither likely nor unlikely”, “unlikely” & “extremely unlikely”) divided by total responses minus “don’t know” responses

48 – (6+10+16)100 - 2

Response SCORE

A Extremely likely 48 48

16

B Likely 18

C Neither / nor 6

32D Unlikely 10

E Extremely unlikely 16

F Don’t know 2

FFT score: = 16


13-089420-01 | FINAL v1 | PUBLIC

the belief that no two Trusts were exactly comparable. However, senior staff did see it as important from a reputational perspective, and acknowledged that they undertook informal “at-a-glance” comparisons between their own FFT scores and those of peer Trusts.

Clinical staff had little awareness of, or exposure to, FFT. While one of the stated aims of the FFT was to produce access to patient feedback data for staff from “board to ward”, usage was, in reality, based on it being used by “ward and board”. Nurses and senior staff used it most frequently and were best able to discuss it at length. Clinical staff, by contrast, had very little knowledge of the FFT. Unless they also had executive or board-level responsibilities, they knew almost nothing about it.

A&E posed a challenge to the FFT. The FFT was hard to successfully implement in a hectic and fast-moving environment like A&E; response rates were often lower, leading to questions about the credibility of the data. It was difficult for staff, stakeholders and patients to see the virtue of the recommendation question in this setting.

There was little consistency between Trusts. In line with the permissive methodology outlined in the guidance, Trusts were administering FFT in a variety of different ways.

Exactly how the FFT was administered to patients was inconsistent within Trusts. Where the distribution of the FFT relied on frontline staff (for example, to hand out FFT postcards), senior staff acknowledged it was almost impossible to police for consistency due to the large numbers of staff involved. This meant, for example, that the FFT was not always asked at the correct time (i.e. at or within 48 hours of discharge) or of the full population eligible to take part in the FFT.

There were issues with the practical definition of the eligible population. Questions were raised over who should and who can be asked the FFT. It was widely believed that it was not practical to ask those with dementia and those who have recently had a highly traumatic experience, among others. Stakeholders suspected, and some frontline staff confirmed, that on occasions particularly difficult (i.e. awkward) or unpleasant patients were not always being offered the chance to complete the FFT. However, this was not universal and there was insufficient evidence to support the view that FFT was being “gamed” deliberately in this way. For example, the staff who admitted not always offering difficult patients the chance to complete the FFT did so in order to avoid confrontation with patients rather than to manipulate the results.


13-089420-01 | FINAL v1 | PUBLIC

Focusing on the response rate was understandable but can lead to perverse outcomes. The use of targets for response rates was widely acknowledged as understandable from the perspective of driving implementation and ensuring the data was credible. However, the case studies Trusts tended to have expended more effort in achieving required response rates than in using the data itself. Similarly, commissioners have focused on enforcing the response rate CQUIN payment as a performance metric. Commissioners had little contact with the frontline staff in Trusts and so knew little about how the FFT was being used “on the ground” to deliver service improvements. This focus on the response rate was also responsible for some Trusts adopting the “Waitrose boxes” or token system. This methodology was believed to offer a higher response rate than other methods, particularly in A&E. However, it was defined by stakeholders and staff as being less useful for identifying areas for service improvement, primarily due to the lack of a follow-up question.

Commissioners paid little heed to the FFT beyond the response rate CQUIN. They did not look at the feedback in detail to directly examine how services might be improved. Some suggested that they would like to make more use of the data from the free text follow-up question and that a CQUIN recognising how data has been used to make improvements would be advantageous.

Presentation and communication of FFT data was inconsistent. The formal Net Promoter Score (NPS) was often not used when displaying the FFT results. Patients and the public wanted a simple and understandable presentation of the results. There was some enthusiasm for a star rating system as this was seen as intuitive and familiar. Patients and the public also appreciated the option of looking at other contextual data alongside FFT data, including other metrics, FFT sample sizes and responses to the FFT free text follow-up question.

The FFT’s relevance to patient choice was questioned. With low awareness among the public, few thought to use the FFT when choosing a hospital. This was exacerbated by the scoring mechanism, which was confusing and difficult for the public to understand. A clearer metric and greater awareness could increase its potential to support patient choice. Some patients and members of the public were keen on choice, others less so. Staff, the public, patients and stakeholders alike all suggested that other factors beyond FFT would be significant influences on choice; for example, location, GP recommendation or past personal experience. Staff and stakeholders suggested that choice was highly constrained much of the time. In particular, there was a question mark over the relevance of choice to A&E.


13-089420-01 | FINAL v1 | PUBLIC

There were questions about the sustainability of the FFT. Successfully conducting the test requires a large amount of effort with limited staff resources. Even those Trusts that were most positive about and most committed to the FFT questioned whether they would continue with it in its current form if it was not mandated that they do so. This was a particular concern relating to the roll-out of the FFT to outpatients with the additional volumes of patients this would include. Some Trusts working with suppliers for data collection that offered low cost or free contracts were also concerned about the sustainability of funding once their current contracts end.


13-089420-01 | FINAL v1 | PUBLIC

2 Background and context

Th

13-0

2

AscoseavaPriacho

Anreg(A&aroIt wtheof merolpathe

Th

“Hthe

An(inseThincthegeusetheopma

Rarespro

he Friends and F

089420-01 | FINAL

2 Bac2.1 The

laid out in Nllecting feedrvices is an eailable to all.me Minister ross the NHSspitals in En

n initial pilot ogion prior to &E) settings ound three mwas implemee aim is that NHS fundedental health al out the FFTtients and st

e health serv

e FFT is a si

How likely areey needed si

nswers are oncluding a ‘dorvices and pey are also a

clude an optieir answer, ander, age aned ‘FFT’ to ree term ‘followen-ended qu

andated FFT

aw answers tsults can be ovider such a

Family Test: Qua

v1 | PUBLIC

ckgroe Friends

NHS Englanddback from pessential ele. The Friendon 25th May S, to improvegland.

of the FFT wanational roll-from April 2

million responented for matFFT will have

d services, inand communT to the wholetaff are offere

vices they us

mple, single

e you to recoimilar care o

n a six-point on’t know’ op

private servicable to ask aion for patien

as well as colnd ethnicity. efer to the m

w-up questionuestion to un question.

o the FFT arviewed at a as ward). Th

alitative research

oundand Fam

d’s Business patients on thement of the ds and Family

2012 as a ne patient car

as carried ou-out in inpatie013. Since inses from inpternity service been introd

ncluding GP nity servicese of the healed the oppoe or provide

question su

ommend our or treatment?

scale from eption). This dce providers additional folnts to providllect some dPlease note

main mandaten’ when refenderstand th

e converted Trust level (ahe approach

h

d andmily Test

Plan – Puttinheir experiendrive to ensuy Test (FFT) ew way of co

re and identif

ut across theent and Accts introductiopatient and Aces in Octobduced acrosand dental p, as well as oth service acrtunity to fee.

rvey:

[service/war”

extremely likdesign is marequired to ulow-up quese a free text emographic that through

ed closed qurring to the ue issues driv

into a standand at lowerh chosen for

d con

ng Patients Fnces of usingure high quawas launcheollecting sucfy the better

e NHS Midlanident and Emon the FFT hA&E settings

ber 2013. By ss thousandspractices, amoutpatients. cross 2014-1

ed back via t

rd] to friends

kely to extremandated, withuse this exac

stions to the Fresponse ex information hout this repuestion and huse of a suppving the resp

dardised scor levels withinthe scoring

ntext

First – g health ality care is ed by the ch feedback performing

nds and Easmergency has collecteds in England.April 2015

s of providersmbulance,

The aim is to15 so that all he FFT on

s and family i

mely unlikelyh local NHS ct wording. FFT that mayxplaining such as theort we have have used plementary

ponse to the

re so that n each of the FFT

t

t

d

s

o

if

y

ir

Howrec[sefriethecar

w likely acommendervice/waends and ey neededre or trea

re you to d our ard] to

family if d similar tment?

8


13-089420-01 | FINAL v1 | PUBLIC

was to calculate a Net Promoter Score (NPS)1 – although a revised version was used for the NHS – for each service provider (and lower level within each provider such as site and ward). The NPS works by splitting people into three groups depending on their response to the question: promoters, passives and detractors.

The NPS is calculated as the proportion of respondents who are promoters (defined in the FFT as those who are extremely likely to recommend the service to their friends and family) minus the detractors (defined in the FFT as the proportion of respondents who would not recommend the service to their friends and family – i.e. the sum of those selecting the neither likely nor unlikely, unlikely or extremely unlikely options). Those who say they don’t know if they would recommend the service are excluded from this calculation.

In order to help inform the roll-out of the FFT to additional settings, NHS England wanted to capture the experiences of implementing and using the FFT in its first six months. To this end, NHS England is undertaking a review of the FFT which contains two elements: a quantitative exploration of the FFT data and qualitative research into how the FFT is operating in practice. The review had the following specific purposes:

To assess the implementation, publication, use and reception of FFT during the first six months of national data collection, focusing on what has worked well and what might be improved;

To collect and present evidence of best practice across the settings;

To judge whether the test in its present incarnation matches its original ambitions, and, if necessary, to highlight any changes that would be required to fulfil all of these functions, specifying any associated risks or costs; and

To ensure that future NHS England guidance on FFT is evidence-based and reflects best practice.

This report details the work undertaken and feedback received as part of the qualitative research2 which NHS England commissioned Ipsos MORI to conduct.

The next chapter outlines the programme of research undertaken to meet the aims of this qualitative study.

1 A research technique developed, and trademarked, by Fred Reichheld, Bain & Company, and Satmetrix. 2 The results of the quantitative study of data and data reporting techniques are detailed under a separate report.

Friends & Family Test Publication Guidance


13-089420-01 | FINAL v1 | PUBLIC

3 Our approach


13-089420-01 | FINAL v1 | PUBLIC

3 Our approach In this chapter, the approach used for the Friends and Family Test (FFT) qualitative research is described. An essential element was to investigate the views and experiences of the FFT across a number of different groups, such as senior staff, frontline staff, commissioners, patients, the general public and wider stakeholders. A multi-stage approach was selected to allow the collection of the breadth and depth of experiences required. In summary our approach consisted of:

A structured online feedback form sent out to all 156 NHS/Foundation Trusts in England;

Detailed case studies conducted in nine Trusts incorporating interviews and discussion groups with senior staff, frontline staff, commissioners, patients and the public, as well as observing the FFT being implemented and used in situ;

20 telephone and face-to-face depth interviews with experts and other stakeholders; and,

The approach was iterative, with regular debrief sessions between the research team at Ipsos MORI and NHS England throughout the fieldwork period. This allowed us to amend research materials and focus on specific emerging themes where relevant.

The remainder of this chapter/section provides more detail on each stage of the study.

3.1 Online feedback

To ensure the research was as inclusive as possible all NHS and Foundation Trusts in England had the opportunity to provide feedback and share their experiences of implementing and using the FFT. This also helped ensure that the research was comprehensive and credibly covered the diversity of experiences that Trusts have had with the FFT.

An online feedback form was developed by Ipsos MORI in partnership with NHS England. The form consisted of 11 closed factual questions and 18 free text questions for Trusts to share their experiences in their own words. A copy of the online feedback form is provided in the appendix to this report. The topics covered in the feedback form included:

Where responsibility for the FFT sat within Trusts;


13-089420-01 | FINAL v1 | PUBLIC

Establishing how FFT data was collected: in what settings, what methods were used to collect the FFT and why that/those methods were chosen;

Whether any follow-up questions were asked alongside the FFT and what these were;

Whether external suppliers were used at any stage of the FFT process and views on these;

How well the FFT was working in practice;

How easy the FFT was to set-up and differences between settings;

Any suggestions on how to improve the guidance around the FFT for future roll-out to other settings;

Understanding and calculation of the FFT score;

How the results of the FFT were used, who was using them and how they were being communicated;

The benefits of the FFT to Trusts and the impact it had on patient experience; and,

What the challenges for FFT had been and how they had been overcome.

A sample of Chief Executives of the 156 NHS/Foundation Trusts in England was supplied to Ipsos MORI by NHS England for the purposes of contacting them about the research. An invitation email was sent to all 156 Chief Executives on 12th December 2013 with a unique link to the online form for their Trust. During fieldwork, two reminder emails were sent on the 8th and 21st January 2014 to all Trusts that had not responded by these dates. Fieldwork ended on the 31st January 2014.

While the emails were sent to Chief Executives, the Trusts themselves decided who would be best placed to complete the feedback form on behalf of the Trust. In order to help facilitate collation of the feedback from across the Trust into one response, a Word version of the form could be downloaded by Trusts to share with colleagues internally.

In total, 95 responses were received, which represents a response rate of 61%. Responses were received from a range of different Trusts as outlined in the table overleaf.


13-089420-01 | FINAL v1 | PUBLIC

Figure 3.1 ---- Response rates for the online form

Total number of

Trusts

Number of response

from Trusts

Online form RR %

Type

Foundation Trust

97 61 63

NHS Trust 59 34 58

Size

Small 52 30 58

Medium 52 31 60

Large 52 34 65

FFT score

Low 47 30 64

Medium 48 30 63

High 61 35 57

FFT RR%

Low 49 32 65

Medium 51 30 59

High 56 33 59

Region

London 25 13 52

Midlands and East

45 27 60

North 48 36 75

South 38 19 50

Source: Ipsos MORI.

3.2 Case studies

Case studies were conducted in nine Trusts in order to deliver a detailed and in-depth understanding of how the FFT has been introduced and used within Trusts. A case study approach had the advantage of enabling an in-depth 360o understanding of the implementation and use of the FFT, ensuring that a range of different perspectives were included and triangulated.

Case study Trusts were selected using a purposive sampling approach to ensure that they reflected the diversity of all Trusts on some key criteria. These included:

Location – at least two case studies in each NHS England region.


13-089420-01 | FINAL v1 | PUBLIC

Type and status of Trust – a range of Trust sizes and a mix of NHS and Foundation Trusts.

Response rate and score – a range and mix of FFT response rates and scores.

Methodology used – a range of Trusts that used different modes to collect FFT.

A mix of those who did and did not use an external supplier.

The table overleaf outlines the Trusts in which case studies were conducted and the selection criteria used. It must be noted that the case study Trusts were selected by key criteria based on data supplied by NHS England from December 2013. As the findings detailed in this report show, the FFT is continually evolving in Trusts; consequently, some of the information used to select the Trusts may have changed in the period between selection and the case study taking place. All case studies took place during January and February 2014. It was initially intended that 10 case studies would be completed, with one conducted at a Keogh 14 Trust. Approaches were made to three of the 14 Trusts to try and secure participation, however all declined to take part.


13-089420-01 | FINAL v1 | PUBLIC

Figure 3.2 ---- Case study sample selection

Region FT FFT RR% FFT Score

Inpatient mixed mode?

Inpatient most frequent mode

A&E mixed mode?

A&E most frequent mode Supplier

Eligible inpatient

population

1 London Yes High Low Yes Tablet/ kiosk Yes Paper/ postcard Yes Large

2 North of England Yes High High No Paper/ postcard #N/A #N/A No Small

3 South of England No Low High Yes Paper/ postcard No Paper/ postcard No Large

4 North of England Yes Low High Yes Paper/ postcard No Paper/ postcard Yes Small

5 North of England Yes High Medium Yes SMS Yes SMS Yes Medium

6 South of England No Medium Low No Paper/ postcard No Paper/ postcard No Medium

7

Midlands and East of

England No High High No Token No Token No Small

8 London Yes Medium Medium Yes Paper/ postcard Yes Paper/ postcard Yes Small

9

Midlands and East of

England No Low Medium No Paper/ postcard Yes Paper/ postcard No Large


13-089420-01 | FINAL v1 | PUBLIC

Once case study Trusts were selected NHS England provided Ipsos MORI with a lead contact at each Trust. Ipsos MORI contacted each lead to arrange convenient dates during which interviews within each case study could take place. A large part of the recruitment for the case studies was arranged by the Trusts themselves. Ipsos MORI provided each Trust with a recruitment specification which gave guidance on the number and roles of Trust staff they wished to interview. A copy of the recruitment specification is provided in the appendix to this report.

Discussion groups with patients and the public were conducted in each case study area. The recruitment for these groups was largely undertaken by Ipsos MORI using in-street recruitment. Loose quotas were set on some demographic variables such as gender, age, ethnicity and social grade to ensure that both the patient and public groups contained a good mix of participants. Patients were defined as those who had a stay in the case-study hospital or visited Accident and Emergency (A&E) since April 2013. A payment of £40 was given to all patient and public participants as a thank you for their time, and to cover any expenses such as travel and childcare to attend the groups.

The case studies were conducted over two days and comprised several elements, following established thinking on case study design3:

Individual interviews, paired depth interviews and mini discussion groups with CEOs, senior directors, management and frontline staff. The mix of interviews and groups varied across the case studies and staff types. Interviews lasted between 30 minutes and one hour;

On-site observation by Ipsos MORI researchers;

Discussion groups with patients and the general public – at each case study one group was conducted with patients and one with the general public. Each group lasted around an hour and a half;

Face-to-face interviews with expert patients – two to three interviews were conducted with expert patients at most case studies. Interviews lasted between 30 minutes and one hour; and,

Telephone interviews with representatives from Clinical Commissioning Groups (CCGs) – a total of six interviews with CCG representatives were conducted across six different case study sites. The representatives of the CCGs were suggested by the case study Trusts. Interviews lasted between 30 minutes and one hour.

The case studies explored:

3 Yin, R. K. (2003). Case study research: Design and methods (3rd ed.)


13-089420-01 | FINAL v1 | PUBLIC

Implementation of the Test, including where it has worked well and less well, the challenges that Trusts faced, and perceptions of the guidance;

Data collection methodologies, including how Trusts were collecting the FFT, how they made those choices, and how this impacted on response rates;

How results from the FFT have been communicated and used, particularly to find out where and how it has been used to drive improvements, if at all; and,

Patient and public experiences of the FFT, including views on the question wording and FFT score, responses to the way the FFT is presented, and exploring how patients and the public might use it.

The interviews and discussion groups covered a range of topics. A copy of the guide of each of the groups and interviews is provided in the appendix. In summary the topics included:

Collecting patient feedback;

The wording of the FFT question;

Implementation of the FFT;

How the FFT is collected;

Whether the FFT is used to support patient choice;

Perceptions of the FFT score and its calculation;

How the FFT results are presented and used in hospitals; and,

Perceptions of the future of the FFT.

3.3 Stakeholder and expert interviews

A total of 20 interviews were completed with a range of stakeholders and experts. These formed an important part of the project, collecting the views and experiences of some key individuals. This provided a broader view than just those who are currently involved in the administration and use of the FFT.

Ipsos MORI developed a purposive sample of stakeholders which was refined and agreed with NHS England. Stakeholders were categorised into five types based on their experience and area of expertise. Loose quotas were set on stakeholder type to ensure interviews were conducted with a good spread of different stakeholders.


13-089420-01 | FINAL v1 | PUBLIC

Recruitment was undertaken by Ipsos MORI’s team of executive recruiters, with interviews booked in at times convenient for respondents. All interviews were conducted by a member of the Ipsos MORI research team over the telephone or where specifically requested face-to-face. Interviews lasted between 30 minutes and one hour.

Figure 3.3 ---- Profile of stakeholder interviews

Category Description Interviews

Leadership: Those closest to the roll-out

Participants in this group had the most detailed expertise and overall knowledge around the FFT roll-out.

4

Implementation: Professional Groups, Patient representatives

These had an important stake in how the FFT has been implemented and used.

6

Methodological expertise

These participants were well placed to discuss methodological issues such as the advantages and disadvantages of different data collection methodologies and the comparability of the results.

3

Wider stakeholders who work closely with NHS England (such as Monitor)

Participants in this group represented organisations who work closely with NHS England and were directly affected by the introduction of the FFT.

5

Independent service providers

Participants in this group represented private or third sector service providers who had implemented FFT.

2

TOTAL 20

The report now goes on to present the findings from the qualitative research into the FFT.


13-089420-01 | FINAL v1 | PUBLIC

4 Overall views of the Friends and Family Test

Th

13-0

4

Thanvieinc

Acthase

he Friends and F

089420-01 | FINAL

4 OveFrie

is chapter sud how well it

ews of the fulcluding staff,

4.1 Thepa

cross all the aat patient exprvices. Patie

Key findi

Collectinby staff, area in walso kee

The FFT was worplaced o

While heparticulaorder to delivered

Howevesome paservicesitself.

Implemerespons

With the attentionfor identboosting

Family Test: Qua

v1 | PUBLIC

erall ends ummarises pt is thought tll range of au stakeholder

e importaatient exp

audiences inperience datents were inc

ngs

ng feedbackstakeholder

which the NHen to give fee

was, for therking well anon patient ex

elpful in its owarly useful whdevelop a dd.

r, these posiarticipants qus. It was felt

entation to dae rate target

FFT now larn to using thetifying servicg the morale

alitative research

viewand

perceptions oo be workingudiences wers, patients a

ance of cperience

ncluded in thta should becreasingly ac

k on patient ers and patienHS was seenedback.

e most part, vd was helpinxperience.

wn right, thehen triangula

detailed pictu

itive sentimeuestioned thto be too blu

ate had beets.

rgely embede FFT data. e improvemeof frontline s

h

ws of d Fam

of the Friendg in practicee engaged wand the publ

collecting

he research, collected ancknowledged

experience wnts and the p to be impro

viewed positng to increas

FFT was thoated with othure of how se

ents were note value of thunt an instrum

n driven by t

ded, Trusts wIt was seen ents, providistaff.

the mily T

ds and Family. It incorpor

with as part oic.

g feedbac

it was widelynd used to imd as importa

was seen as public. This woving; patient

tively. Trustsse the empha

ought to be her data sourervices are b

t universally he FFT in impment to do th

the need to m

were turningas particularng real time

Testy Test (FFT)rates the f this work,

ck on

y accepted mprove ant

important was an ts were

s said it asis

rces in being

held, and proving his by

meet the

g their rly useful data, and

Thefocind

Patigrou

ere is muccus on padividuals.

ent discussiup

2

ch more atients as

on

20


13-089420-01 | FINAL v1 | PUBLIC

stakeholders whose opinions should be sought when shaping the services they use.

Therefore, stakeholders and staff agreed that collecting data on patient experience should be a central part of how the NHS operates. Further, it was generally recognised that this was an area in which the NHS is making efforts to improve, with greater focus on the patient voice than has been the case in the past.

I remember hearing a doctor referring to a patient as ‘the appendix’. You wouldn’t hear this kind of thing anymore. There is much more focus on patients as individuals. Patient discussion group

As Trusts were generally aware of the importance of understanding the patient experience, collecting this kind of data was nothing new to them; many were already doing this prior to the implementation of the FFT. For instance, some had developed their own bespoke in-house survey which patients are asked to complete at some stage during a hospital visit. These tended to be surveys with multiple questions about various specific aspects of patient experience.

We were already undertaking a survey of all inpatient areas and the FFT was added to this approach. We believe it sits better within a bigger survey. NHS Trust

There were also examples where Trusts were already using qualitative data collected through, for example, patient diaries, depth interviews and discussion groups. This kind of data was felt to be particularly useful, adding depth to a Trust’s understanding of patient experiences. Nurses also mentioned the benefit of going out onto wards and talking to patients face-to-face, often feeling that they received their most valuable insights about their work this way.

Patient diaries and patient stories are very, very powerful. Actually, by just going out and talking to our patients you will get far more back. Associate Chief Nurse

Patients and the public were also enthusiastic about the concept of being asked about their experiences while in hospital. They explained that they were increasingly used to giving their opinion on other services – for example, about hotels and restaurants – and, therefore, felt comfortable being asked to do the same about the NHS.

Collecting data on patient experience should be a central part of how the NHS operates.


13-089420-01 | FINAL v1 | PUBLIC

I am always asked when I go to the garage. I should be asked when I go to the hospital. Patient discussion group

4.2 Perceptions of the FFT

Almost all participants therefore agreed with the principle of the FFT – that feedback on patient experience should be gathered – and it appears to have had a positive impact on how patient experience is viewed and embedded within Trusts. For instance, in the online feedback form four in five Trusts said that the emphasis on patient experience had increased (78%), including one in three who said it had increased a lot (36%).

Figure 4.1 ---- The impact of the FFT on patient experience

This was certainly endorsed in the free text comments provided in the online form, as well as the qualitative interviews with stakeholders and staff, who described the FFT as ‘an evolutionary step’ that was helping Trusts to further embed patient experience within their everyday work and practice, alongside other feedback mechanisms already in place. Staff suggested that the FFT already was (or had the potential to be) a useful tool for improving patients’ experiences. Others, including commissioners and Trusts responding to the online survey, also felt that the FFT had helped embed a greater focus on patient experience, especially in those places that had not done so previously as a result of FFT being mandated requirement.

If the FFT has added anything, it’s made providers think about patient experience more seriously ---- it has heightened awareness of patient experience. Commissioner

Feedback from staff in the online form and at the case studies also suggested that the FFT could offer a better approximation of real time data

36%

42%

17%

1%

2%2%

Has increased a lot Has increased a little Has stayed the same Has decreased a little

Has decreased a lot Too early to say Don't know

Source: Ipsos MORIBase: All Trusts that responded (95). Fieldwork dates: 12th December 2013 - 2nd February 2014.

What difference, if any, has the Friends and Family Test made to the emphasis on patient experience in your Trust?

Th

13-0

cothe

Fuinc

DecawapaexbemealobePasta‘te

[DhapromeAs

Fuunwhmoiminvthomocu

he Friends and F

089420-01 | FINAL

mpared to oe specific me

rther details cluded in Ch

Identifyievidenceresult, it patient eFFT feedtimes forrooms.

Boosting‘extremedo. The feedbacprevious

espite these tch-all solutio

as seen as a tients or stakperiences ofcame a pow

etrics alreadyongside othespoke patientient Advice

akeholders thmperature g

Data from thave a problroblem is. Seasure the sociate Chie

rthermore, it iversal and t

ho were mucotivations for proving patievestment neeose who wereost of it wererrent form if

Family Test: Qua

v1 | PUBLIC

other patient ethodology u

of the rangeapter 8, but

ng practicae base for idwas already

experience. dback to ider inpatients o

g frontline mely likely’ resfollow-up qu

ck – somethinsly collected

positive percon to undersblunt tool th

keholders wif a Trust. Ho

werful piece oy in place. It

er data such nt experiencand Liaison

herefore refeauge’, an ‘ad

the main FFTblem somewSo you can e temperatuef Nurse

should be nthere were Th more nega

r the introducent experieneded in time e more posit unsure of wit were not m

alitative research

experience sused (see se

e of ways in wtwo of the ke

al service imdentifying poy having a smFor examplentify the neeor more com

morale: The hponse helpe

uestion often ng frontline sin such a sy

ceptions, howstanding patat when useth all the info

owever, whenof corroboratt was at its mas the nation

ce surveys, c Services (PA

erred to the Fdd-on’, a ‘sm

FT question]where, but in use it as aure of an ar

noted that thirusts, staff (fative. Thesection of the Fnce, particula

and effort totive about th

whether they wmandated. T

h

surveys, althection 6.2.6).

which the FFey uses are h

mprovementstential servic

mall number e, frontline std for changefortable seat

high numbered remind sta

generated astaff appreciaystematic wa

wever, the Fient experiend in isolationormation theyn triangulateting evidenc

most useful wnal inpatient

complaints anALS) data. S

FFT as a ‘linemoke detecto

] will tell yoit won’t tell

a thermomerea.

s positivity tofrom ward to staff tended

FFT and did narly when weo collect the e FFT and cowould contin

This was bec

hough this de

FT is being uhighlighted b

s: The FFT pce improvemof significantaff talked abes to therapyting in A&E w

r of patients aff of the gooa great deal ated and sai

ay.

FT was not snce. Indeed,n did not proy need abou

ed with other ce among thewhen conside

survey, addnd complimeSenior staff ae of sight meor’, or a ‘litmu

ou that youl you what eter if you l

owards the Fo board) and d to point to not see its vaeighed againdata. Indeeommitted to nue with the cause the suc

epended on

sed are below.

provided an ments. As a nt impacts onbout using y session waiting

providing anod work theyof positive d was never

seen as a by itself, it

ovide staff, ut patient measures it

e many otherered

ditional ents, or and asure’, a us test’.

u might that

like to

FFT was not stakeholder

political alue in nst the d, even making the FFT in its ccessful

n

n y

r

r

rs

Theproserexp

Com

Theposgenhasthesta

e FFT has oviders thriously abperience.

mmissioner

e high volsitive feednerated bs served t

e morale oaff.

2

made ink more out patie

lume of dback by the FFTto boost of frontlin

23

ent

T

e


13-089420-01 | FINAL v1 | PUBLIC

administration and collection of the FFT was felt to require a large amount of effort from increasingly scarce resources. This was a particular concern relating to the roll-out of the FFT to outpatients with the additional volumes of patients this would include. Some Trusts working with suppliers for data collection that offered low cost or free contracts were also concerned about the sustainability of funding once their current contracts end.

Initially I didn’t want to do it; we only did it once mandated. The jury's out; it is additional work and additional resource, and we do it because we have to. Director of Quality and Safety

Reflecting this concern about funding, some Trusts responding to the online feedback form called for additional funding, for example through central funding or through CQUIN payments.

I think at some point there has to be consideration for resource allocating to FFT. Whether that is individually Trusts making allowances for this through CQUIN payments or if centrally funds are going to be made available. NHS Trust

4.3 Implementing the FFT

Six months after the FFT was initially implemented, the online feedback showed that it has largely been embedded in Trusts’ working practices, with most (88%) believing that it was working well.

Figure 4.2 ---- Successful implementation of the FFT

From the case studies and online feedback forms, it was clear that staff believed the FFT was working well when their Trust, ward or department was achieving the CQUIN targets around response rates. Conversely, when the response rates were missing or close to missing the target, staff

16%

73%

6%3% 2%

Very well Fairly well Not very well Not at all well Too early to say

Source: Ipsos MORIBase: All Trusts who responded (95). Fieldwork dates: 12th December 2013 - 2nd February 2014.

Thinking generally about the Friends and Family Test, how well, if at all, is it working in your Trust?

88%

88% of Trusts say that the FFT is working well in their Trust. Online feedback form among Trusts


13-089420-01 | FINAL v1 | PUBLIC

were more likely to say it wasn’t working well. Indeed, a significant amount of energy had been invested in achieving the required numbers of responses, with adjustments having been made to data collection methods to ensure good response rates. Now, with these targets met, Trusts were starting to pay attention to the question of how the FFT could most usefully be employed. Other key reasons why Trusts felt the FFT was working well included having successfully embedding the ethos of FFT and making use of the results to improve patients’ experiences. However, some Trusts providing online feedback also highlighted moving on to make full use of the data to drive service improvements as one of the challenges for the FFT. That is not to say, however, that this meant that Trusts were less concerned about response rates; while the CQUIN target exists, they are likely to remain a driver. However, Trusts were aware that higher response rates meant more robust data which, in turn, could increase the validity of the conclusions that they draw from it.

For a very long time the focus was very much on response rate. You can’t do anything without a robust response rate. We’re now moving into the communication of the results, both to staff and to patients. Head of Patient Experience

4.4 Knowledge and understanding of the FFT

Knowledge and familiarity had a significant impact on perceptions of the FFT. Broadly speaking, those we spoke to could be characterised as follows:

The most knowledgeable tended to be senior staff – particularly such senior staff as CEOs, medical directors and patient experience leads with board-level roles – and frontline staff, such as ward managers and nurses. This group includes those who have responsibility for the FFT within their Trust, which is often the responsibility of either the Director of Nursing or patient experience lead. As well as being most knowledgeable, these two groups also tended to be most positive about the FFT – they saw it as helpful for triangulating with other measures and for providing a broad indication of performance. Senior staff typically saw the FFT as one performance metric among many which they could triangulate with other measures against what patients were saying. The FFT offered senior staff a broad indication of how wards and departments were performing. However, other patient feedback measures were required to offer greater detail and to build a fuller picture of how the different aspects of care were experienced by patients. The FFT was perceived to offer the closest yet approximation of “real time” feedback from patients. Some stakeholders – typically those with the closest working links with NHS England, such as regional FFT leads – were closely involved in the practical implementation of the FFT and so were largely enthusiastic


13-089420-01 | FINAL v1 | PUBLIC

about the Test and its potential for embedding a culture of listening to patients. Nurses had the closest “hands-on” links to the FFT, through administering it to patients and implementing the feedback it provided. While this group saw the FFT as an administrative burden, they also valued the positive and actionable qualitative feedback provided when the FFT was linked with a free text follow-up question.

Stakeholders who were not working directly with NHS England, while knowledgeable, had a more detached and theoretical view of the FFT. While they recognised the advantages of having a standardised, mandated patient feedback tool they also raised concerns about comparability across the diverse range of methodologies used to collect the FFT.

Those with least knowledge of the FFT tended to be doctors and consultants. Without ward-specific or board-level roles, they did not personally get involved in managing the FFT (i.e. handing out questions or encouraging patients to take part) and did not pay attention to the results being displayed and used on wards. This did not stop them from being positive about the FFT in principle, but they were typically unsure as to how it might apply to their jobs. Exceptions to this were doctors or consultants attached to A&E as their work centred on one department, these clinicians had more exposure to the FFT and therefore witnessed the practical benefits and were much more positive towards the FFT than their non-A&E peers.

Most patients and members of the public had little, if any, experience of the FFT, whether through answering the question or using the feedback themselves. As a result their discussions tended to focus on how the FFT might work in theory. Patients and public, as well as NHS Trusts and frontline staff, suggested that an awareness campaign could help increase patient and public engagement with the FFT and improve response rates. During the discussion groups with patients and the public, we tested reactions to a range of stimulus materials covering the question wording, the calculation of the FFT score and alternative scoring systems, plus potential presentations of the FFT data. All of these can be found in the appendix to this report.

4.5 FFT and patient experience measures

Many Trusts were already collecting some form of patient experience data prior to the introduction of the FFT, and this posed a series of challenges: for example, whether or not to integrate the FFT into existing data collection methods and how to differentiate the FFT from other measures for patients and staff.

Awareness of the FFT is very low among patients and the public.

Many Trusts already collected patient experience feedback. Merging this with the FFT presented several challenges.


13-089420-01 | FINAL v1 | PUBLIC

Patients and the public demonstrated a limited understanding of the feedback channels available to them. Very few had completed a patient experience survey about their recent visit, used formal complaints channels, or heard of or completed the FFT.

Despite this lack of awareness, senior and frontline staff in the case studies and some Trusts responding to the online feedback form suggested that the range of feedback avenues offered to patients could seem overwhelming, with an associated risk of survey fatigue – though the public and patients themselves did not raise this as a problem. They felt that Trusts needed to be clearer with patients about the types of feedback methods that were available to them. Additionally, nurses pointed out that the FFT was adding to a perceived excess of information for the patient to contend with around the ward or department. For example, information on other patient surveys or health and safety measures is displayed to patients on posters, information booklets and other communications around wards. FFT displays were adding to this abundance of information for patients to process, which some nurses felt could be confusing.

If you’re a patient you get the GP survey, you get an inpatient stay survey, potentially you get an outpatient appointment experience survey, at some point there’s going to be fatigue. Patient experience manager

4.5.1 Integrating FFT with existing measures

Many Trusts chose to collect the FFT in a similar way to their existing patient feedback measures. In some cases the FFT was being asked alongside other questions covering patient perceptions of the treatment, care, and information they have received. Feedback from the online survey suggests that this approach was felt to both ease the burden of data collection, be relatively cost effective and smooth the implementation of the FFT within the Trust. This approach was felt to both ease the burden of data collection and be cost effective. More importantly though, it was also thought to increase staff buy-in as they were able to see FFT data in the context of other, more familiar measures which, in turn, made the findings more meaningful.

Adding some additional patient experience metrics to FFT has helped to engage the frontline in data collection; it makes the score a lot more meaningful to them. NHS Trust

This solution was not viable for all Trusts though. For example, one Trust described how they had identical inpatient and outpatient surveys, administered via pen and paper to any patients who wished to provide feedback. This Trust chose to create a separate single-question survey for the FFT, rather than create separate inpatient and outpatient


13-089420-01 | FINAL v1 | PUBLIC

questionnaires. The reasons for this decision included: a large increase in the volume of inpatient questionnaires, as all would have to be asked the FFT; the consequent financial and administrative burden of dealing with this increase; an unwillingness to have separate inpatient and outpatient questionnaires; a desire to avoid overburdening patients.

The remainder of the report goes on to discuss the findings around the FFT in more detail.


13-089420-01 | FINAL v1 | PUBLIC

5 The Friends and Family Test Methodology


13-089420-01 | FINAL v1 | PUBLIC

5 The Friends and Family Test Methodology

This chapter explores perceptions of the methodology behind the Friends and Family Test (FFT), considering how the question wording and calculation of the score were perceived and understood.

Key findings

There were differing opinions around how effective it is to use recommendation as the basis for the FFT question. To many staff, patients and the public alike, asking about whether or not they would recommend their ward or A&E was easy to understand. Others equated it with a more general satisfaction question. In contrast, some felt the term inappropriate in a healthcare setting; patients and the public said they would not recommend their ward or A&E because they would not want friends and family to need care there.

When completing the FFT, respondents tended to focus on their experiences in hospital and environmental factors rather than clinical care. This was clear from the discussion groups with the public, feedback from staff and from observation of completed FFT questionnaires.

The FFT scoring system was disliked. Participants found a net score ranging from -100 to +100 confusing, and also thought it was unfair to omit ‘likely’ responses. Finally, producing one figure alone was thought to restrict patient choice as it was not possible to explore the underlying data.

The inclusion of a follow-up question allowing patients to provide free text comments was considered particularly valuable to give context to data generated by the main question and allow areas for improvement to be identified.

Th

13-0

Th

Ththetheagclo

5.1

ThFFdecolanwitwhjudfam

ThrecCo

Fowoquthe

he Friends and F

089420-01 | FINAL

5.1 The

e FFT is a si

e FFT questie opportunityeir answer. Te, gender an

osed or free t

1.1 Recomm

ere were mixT question. partment to nsultants an

nguage that th patients. hether they wdging its quamily member

hat’s the kinecommend onsultant

r patients anording for theestion was q

em to answe

“How likely friends and

Please choo

Extreme

Likely

Neither l

Unlikely

Extreme

Don’t kn

Family Test: Qua

v1 | PUBLIC

e questio

ngle, simple

ion is often foy to provide fTrusts often and ethnicity) text question

mendation

xed views onFor some, atheir friends d stakeholdehealth profesIn addition, h

would recommality – if they r then there i

nd of thing the service

nd members e first time whquickly under.

are you to re family if they

ose from the

ely likely

likely nor unl

ely unlikely

ow

alitative research

on wordin

question:

ollowed by afurther feedbalso ask dem

while, in a sns are includ

as a basis

n the use of rsking patienand family w

ers believed ssionals usehealth profesmend a servwould not res likely to be

I would saye to my fam

of the publichen we presrstood with m

ecommend oy needed sim

e following op

ikely

h

g

a free text quback about t

mographic qusmall numbeded.

for the FFT

recommendants to recommwas felt to bethat the phr when discussionals expvice to their frecommend ae a problem w

ay to my pamily?

c, most of whented it to thmost sugges

our ward/A&milar care or

ptions:

uestion that ghe reasons uuestions (for r of cases, a

ation as the mend a warde intuitively rrasing reflectssing treatmlained that thriends and fa

a service to awith it.

atients, wou

hom were sehem, the prinsting it would

E departmenr treatment?”

gives patientunderlying example,

additional

basis for thed or A&E ight. Doctorsted the

ment options hey consideamily when a friend or

uld I

eeing the FFTnciple of the d be easy for

nt to ”

s

e

s,

r

T

r

Thevienotthephr

ThawopatrecserCon

ere were mews on wht ‘recomm

e right warase the q

at’s the kinould say totients, wo

commendrvice to mnsultant

3

mixed hether or mend’ way to question.

nd of thingo my uld I

d the my family?

31

as

g I

Th

13-0

Yowiin medoPa

ByimseseothPaenacbebe

It’sin recgoPa

It iGe

HoFFhagacatha

It win Sta

In wawono

Yoto Ge

he Friends and F

089420-01 | FINAL

ou can get ithout havina restauraeal there aon’t have totient

y referring to portance whriously. For rvice if nothihers, and participants hocourage highieve a highlieved there ing satisfied

’s importantthe same s

ecommend ood. tient

is just askineneral public

owever, concT question. Sd its origins thering feedn be more c

at might occu

was originathe NHS bu

akeholder

particular, soas asking patould thereforet want their f

ou wouldn’to hospital. eneral public

Family Test: Qua

v1 | PUBLIC

a really qung to think

ant. You’re gand the atmo think abo

friends and hich encouraexample, theng had gonerticularly frie

olding this vieher standard score. Thiswas little dif with it.

nt to me becsituation, sothey go the

ng whetherc discussion

cerns were aSome cited tin the privateback on a pomplex and ur when buy

ally about put I don’t th

ome patientstients to recoe be unwillinfriends and f

’t recomme

c discussion

alitative research

uick honestk too much going to re

mosphere. Yout it.

family, the qged participey suggestee wrong, but ends and famew suggesteds as it woulds was not a ufference betw

ecause I’d lisome of myhere ‘cause

r we are hagroup

also raised abthe fact that e sector andublic serviceserious than

ying a produc

private sechink it work

s and membommend poong to providefamily to bec

end that you

group

h

t answer froabout it. It

ecommend You know s

question assupants to answd that they mwould only r

mily, if it had ed that this, id be more d

universal viewween recomm

like to thinky family or fe I do think t

appy with th

bout using rethe question

d, thus, was ie: patient joun the more trct or using a

ctor brand ks very well

ers of the puor health or he a positive recome ill or ne

our friends a

om sometht’s similar to

d it if you enstraight aw

umed a levewer the quesmight be satirecommendbeen exceptn turn, couldifficult for se

w though, asmending a s

k that if somfriends, thathe treatme

the service.

ecommendan and net pronappropriate

urneys and eransactional service.

loyalty. It ll.

ublic felt thathospital treatesponse as

eed care.

and family

hing like thiso if you’re njoyed the

way, and

l of tion isfied with a

d a service totional.

d help ervices to s others service and

meone wasat I would

ment’s really

e.

ation for the omoter scoree for

experiences exchanges

is now used

t the questiotment. They they would

should go

is

s

y

e

ed

n

Yourecfriesho

Gendisc

It wprivloyin tthin

Stak

u wouldncommendends and ould go to

neral public cussion grou

was originvate sect

yalty. It is the NHS bnk it work

keholder

3

’t d that youfamily

o hospital

up

nally abouor brand now used

but I don’tks very we

32

ur

l.

ut

d t

ell.

Th

13-0

Thmorecavaof chbaSimthewaeitca

YoonGe

If iGe

5.1

Thexttheoffscatyp

I wthaSta

Otdacorecres

CawoPa

Irrepasco‘exthe

he Friends and F

089420-01 | FINAL

e concept ofore detail in Ccommendatioailable. Thisparticipants oice in most sed on dista

milarly, someey are admittas less meanher the hospre.

ou cannot cne? The receneral public

it was an eeneral public

1.2 Opinion

e FFT questitremely unlikese responsefer enough gale (achieve

pically used

would havehan the 5-poakeholder

hers, typicalta of the kindmplicated. Fcommend / nsponse to the

an’t they juould be a lotient discuss

espective of tients would ore. There w

xtremely likeleir experienc

Family Test: Qua

v1 | PUBLIC

f recommendChapter 8) bon would ons was seen afrom across

t areas aboutance and howe pointed outted to, and s

ningful. Rathpital as a who

choose a ccommend isc discussion

emergency,c discussion

ns of the res

ion uses a sikely (plus a ‘de options; soranularity or d by asking in net promo

e preferred oint scale a

ly patients ad generated For this groupnot recommee question.

ust ask us iflot simpler?sion group

this, particip want to sele

were two reay’ would imp

ce; secondly

alitative research

dation was aby some: parly make sen

as particularls all of the grt which A&E w quickly thet that patient

so felt that reher, they belieole or a spec

certain wardis nonsensegroup

, I’d like to group

sponse cod

ix-point Likerdon’t know’ oome staff and

differentiatioparticipants

oter scores) w

d to have usas it allows

nd the publiby the FFT f

p, there was end / no opin

f we would ?

pants from acect the ‘extresons for this

ply that there, patients in

h

also linked torticipants stase in a situaty problematoups arguedto visit; inste

e patient neets are not abcommendingeved the quecific service,

rd so how ce.

o go to any

des

rt scale fromoption). Thed stakeholdeon in the dat to give a likwould help to

sed the 11-ps more gran

c who were found the resa preferenc

nion scale or

d recommen

cross all groemely likely’ r: firstly, it wa

e was little roEngland wer

o choice (disated that asktion where aic for A&E, ad that there wead, it was aeded to be trble to chooseg a ward in testion shouldfor example

can you rec

hospital.

extremely liere were mixeers suggesteta and that akely score froo achieve th

point scalenularity.

naturally lessponse optio

ce for a threer even a yes

nd it or not

oups questioresponse coas felt that anoom for improre perceived

cussed in ing for a choice is

as a number was very littlea decision reated. e which wardthis context d refer to e cancer

commend

kely to ed views on

ed they do non 11-point

om 0-10, andis.

e rather

s familiar witons overly e-point / no

t. That

ned whetherde as a top

nswering ovement in d to be

e

d

ot

d

th

r

If it emgo

Gendisc

t was an mergency,

to any ho

neral public cussion grou

3

, I’d like toospital.

up

33

o


13-089420-01 | FINAL v1 | PUBLIC

culturally unlikely to say ‘extremely’, with responses tending toward a more reserved viewpoint. This led to a concern that this would have a negative impact on Trusts’ overall scores4 given the way the FFT score is calculated.

I would never pick extremely likely ---- there is always something they can improve on. Patient discussion group

5.1.3 The FFT question for vulnerable groups

Stakeholders and staff, particularly those working with vulnerable groups, thought that some patients would find the question difficult to understand. For example, staff working with older patients explained that some patients with dementia struggled to consider the entirety of their visit before answering the question, but might be able to communicate their feelings at the moment that they are being questioned. Some patient experience leads argued that an ‘easy read’ version of the question would be preferable for patients with learning disabilities. One in particular had tested the question with a patient with learning disabilities, and stated that a three-point scale using smiling and frowning faces to represent a positive or negative experience would have been more accessible.

5.1.4 Focussing on specific wards

When considering the FFT in an inpatient setting, there was some confusion about how patients would interpret the question and what they would be thinking about before deciding how to respond.

Patients and the public pointed out that they may visit several wards during their time in hospital making it difficult to know which one to rate. To address this, some Trusts chose to add ward names to the FFT question, or alternatively an indication as to what ward the FFT refers to (e.g. ‘the ward you spent most time on’). That said, stakeholders and staff agreed that individuals’ responses would not just reflect patient experiences of a particular ward, but could also reflect their broader experience of the care they have received at the hospital.

Experiences prior to admission may also influence how people respond to the FFT, such as the referral process or car parking. Indeed, staff remarked that the responses to the follow-up question provided evidence for just that.

5.1.5 The follow-up question was particularly valuable

While a single, closed question was easily understood by patients, staff and stakeholders talked about it being a ‘broad brush’ and a ‘blunt tool’ as it did not allow for detailed feedback or identifying why a particular response was

4 Please note that the FFT data does not support this view: in January 2014, 74% of inpatient respondents answered ‘extremely likely’.

The follow-up question provided the greatest value ---- without this the FFT data was too blunt.

Th

13-0

givalo

It iSta

FowitsetchTrumaas/ a

Thunfeebeimanbe

ThraCli

Thof argprothe

It’sPa

Whwea feitpopoInspathe

It fPa

he Friends and F

089420-01 | FINAL

ven. This ledone could be

is only a sinakeholder

r this reasonth all participttings also asoose the wousts. Many uain reason fok more genebout their ca

e qualitative derstand wh

edback abou made, allowproved. Thed largely an fore the FFT

he qualitativare.

nical busine

e views of pastaff, who stgued that a sovide the leveir feedback

’s useful as tient discuss

hile patients ere available free text respher making assible that thtential to con

stead, hospittients and p

emselves, th

feels like hotient discuss

Family Test: Qua

v1 | PUBLIC

d participante used to imp

ingle questi

n, the inclusiopants. Nearlysked a followrding of the use the followor the answeeral questionare or if they

data helpedhy they had but what patiewing greater e follow-up qimprovemen was introdu

ive feedbac

ess manager

articipants frrongly advocsingle closedvel of feedba

to influence

s it allows yosion group

and the pubto them, the

ponse was aa formal comhe multiple fenfuse. Howetals actively ublic as prefrough the PA

hospital is lision group

alitative research

s to questionprove service

tion and it is

on of an addy all Trusts ww-up questiofollow-up quw-up questior to the FFT cs about whahave any co

d staff to makbeen scoredents felt went

capacity to uestion was

nt of the quauced.

ck is really

rom the patiecated askingd question wack that they service imp

you to elabo

blic assumedey had little dlso thought t

mplaint or seneedback chaever, no-one engaging to ferential to pALs departm

istening to y

h

n how respones.

is not clear

ditional free tewho ask FFT on (97% for buestion so theon to ask for closed ques

at could be imomments the

ke sense of t in this way. t well and whdiagnose hoseen as a relitative data

y useful, and

ent and publg a free text f

would be insuwould like, e

provement.

orate on yo

d that formal detailed knowto save somending a thanannels availacomplainedgather feedatients takin

ment for exam

you.

nses to the F

r what it tel

ext question in inpatient

both). Trustsere is variatioan explanattion, while so

mproved, whey would like

the FFT ratin It provides

here improveow services ceal strength available at

d before th

ic groups refollow-up quufficient for thespecially in

our point.

complaints cwledge of thee patients thenk you letter.able to patied about beingback was seg the initiativ

mple.

FFT question

lls you.

was populaand A&E

s were free toon across ion of the ome Trusts hat was good

to make.

ng and ward level

ements couldcould be of the FFT ward level

he FFT [was]

eflected thoseestion. Theyhem to order for

channels ese. Leavinge trouble of It is nts had the g confused. een by ve

r

o

d

d

s]

e y

g

It’s youyou

Patigrou

useful asu to elabour point. ent discussiup

3

s it allows orate on

on

35


13-089420-01 | FINAL v1 | PUBLIC

5.1.6 Considerations when answering the FFT question

Although not intending to cognitively test the FFT, the research provides evidence about the different aspects of care that patients take into consideration when deciding upon their answer to the main FFT question. This evidence derives both from the patient and general public discussion groups, the reports of staff and the free text responses that were observed within the case studies.

Participants in the patient and general public discussion groups suggested that they would answer the FFT thinking about the way in which staff cared for them and about environmental (or ‘hotel’) factors, rather than thinking about the clinical care they received. These included issues such as the friendliness of staff, being kept informed, waiting times (particularly for A&E), cleanliness and car parking.

This was also observed in the case studies, where many free text responses focused on how friendly and caring staff had been. Frontline staff also highlighted that where areas for improvement had been identified, they tended to be around environmental factors such as broken toilets and noisy bins.

It seemed that patients would be unlikely to comment on any medical care that they had received unless it seemed particularly poor (in which case they may be more likely to complain formally). This was because they did not believe they had the requisite knowledge and understanding to make a judgement about their medical care and, instead, could only comment on their overall experience.

Stakeholders and staff also agreed that the FFT was primarily a tool for measuring patients’ views of ‘hotel’ factors and the ‘softer skills’ of staff such as being friendly and caring, rather than their clinical experience because these are the things patients talked about in their responses to the follow-up question. This, in turn, had an influence on who was most likely to be engaged with the FFT. It was felt that FFT in inpatient settings was very much linked to patients’ experiences on the ward, reflecting their interaction with ward staff such as nurses and ‘hotel’ factors. Those who seemed to be less linked to a specific ward (for example, consultants and doctors who worked across a number of wards) tended to view the FFT as commenting less on their own work and so, on the whole, believed FFT had less relevance to their role. This contrasts with doctors in A&E settings who, with greater responsibility for their departments, seemed to assume greater ownership of the way the FFT worked and the results it provided. See section 6.10 for more detail on this.

5.2 The scoring system

The FFT scoring system is adapted from the Net Promoter Score, a loyalty measure developed by Fred Reichheld, Bain and Company, and Satmatrix.


13-089420-01 | FINAL v1 | PUBLIC

The FFT score defines promoters as those who say they are ‘extremely likely’ to recommend a service, and detractors as those who give a neutral response ‘neither likely nor unlikely’ or a negative one ‘unlikely’, or ‘extremely unlikely’. The FFT score is a net score calculated by subtracting the detractors from the promoters. The following chart illustrates how a score would be calculated in a fictional Trust.


5.2.1 Reactions to the score

The way in which the FFT score is calculated was disliked across all the audiences included in the research. There were three key perceptions underlying negative views of the FFT score:

It was confusing and difficult to understand;

It was unfair; and,

It restricted choice.

These perceptions are explored in the following sections.

5.2.2 Participants found the scoring system confusing and difficult to understand

Patients, the public and staff all believed the FFT score was confusing and difficult to understand. This was reflected in the online feedback provided by Trusts. As figure 5.2 illustrates, almost all disagreed that the score is easy for patients (90%) or the public (91%) to understand, while many also disagreed that it is easy for staff to understand (79%).


48 – (6+10+16)100 - 2

Response SCORE


16

B Likely 18

C Neither / nor 6

32D Unlikely 10


F Don’t know 2

FFT score: = 16

5%

Only 5% of Trusts agreed that the scoring system was easy for the general public to understand. Online feedback form among Trusts

The scoring system was considered to be confusing and unfair and restrictive of choice.

Th

13-0

Fig

TrustucoDifit m

It iexproNH

Paalsthethereafict

WhGe

Fua pscokne

Lincoacstapethe

B2

Tw

he Friends and F

089420-01 | FINAL

gure 5.2 ---- U

usts feeding udies pointedlleagues on fficulty undermade it hard

is overcomxplain to staroviding. HS Trust

tients and thso acknowledey were usedey were moreaction given tional Trust.

What is that oeneral public

rthermore, mpercentage. ores were peew they were

nked to this, nfusion due customed to

ar rating systrcentage). M

ese simpler w

Easy for staff

Easy for patients

Easy for the geunders

Strongly ag

Base: All Trusts that resp014.

Thinking now awhat extent do

Family Test: Qua

v1 | PUBLIC

Understandi

back througd out that thethe frontline rstanding theto link the F

mplicated ataff how the

he public alsdged in the od to seeing ne comfortablwhen mode

out of? Is tc discussion

many of the s In some casercentages we not, they st

the range ofto of its unfa

o thinking abtems used onMany Trusts ways of pres

2%

2%

2%

4%

3%

to understand

to understand

eneral public tostand

gree Tend to agre

ponded (95). Fieldwork

about how theo you agree o

alitative research

ing of the F

gh the online e scoring systasked with

e score undeFT output to

and this mae score equ

o found the sonline feedbnumerical dale with. This rators prese

that a percgroup

staff we spokses this was while others till thought o

f the score (familiarity. Staout numbersn consumer in the online

senting the F

12%

%

%

7%

3%

3%

34%

33%

ee Neither / nor

dates: 12th December 2

e Friends andor disagree tha

h

FT score

form and sestem was difadministerin

ermined the patient expe

akes it extreuates to the

score difficuack from Tru

ata expressedwas evidentnted them w

centage?

ke to discussbecause staexplained thf them in this

rom -100 to aff and paties being out owebsites), ofeedback suFT results. T

41%

Tend to disagree

2013 - 2nd February

d Family Test sat the numeri

enior staff in fficult to explng and usingcredibility oferience.

emely diffice care they

lt to understusts. They ed as percent from the co

with the FFT s

sed the scoreaff assumed hat even thous way.

+100) also cents alike weof five (for exor out of a huuggested chThey were al

56%

58%

Strongly disagree

score is calcucal score is. .

the case ain to their

g the FFT. f the Test as

cult to y are

and, a fact xplained thatages, which

ommon score of a

e as if it werethat the

ugh they

caused re more

xample, the ndred (i.e. a

hanging to so

38%

1%

1%

Don't know

Source: Ipsos MORI

ulated, to . . .

at h

e

a

%

%

TheSconegoutstaboa

ChieOffic

e Net Promore produgatively stcomes…

ar rating inard repor

ef Operatingcer

3

moter uces skewed

…we use an our rts.

g

38

a


13-089420-01 | FINAL v1 | PUBLIC

uncomfortable with the possibility of very low or even negative scores, and did not understand what a negative score would mean.

I would have thought that zero is a terrible score, but it is actually half way. General public discussion group

Furthermore, patients and the public suggested it would be alarming to find out that you were going to a hospital with an FFT score of zero or below.

5.2.3 Many thought the scoring system was unfair

All were concerned about the ‘likely’ response option being – as they saw it – disregarded in the score calculation. Patients and the public believed that this meant such a response would be ‘wasted’ as it would not be included within the calculation, something they felt to be unfair on those completing the survey who they assumed would consider a ‘likely’ response to be a positive one.

This sense of unfairness was also a significant concern for staff and was also flagged by Trusts in the online feedback. They felt that not counting ‘likely’ as a positive response distorted the data; that the score therefore did not accurately reflect the positivity of the FFT feedback provided by patients. This, in turn, could potentially damage staff morale.

The NPS produces negatively skewed outcomes…we use a star rating in our board reports. Chief Operating Officer

By excluding ‘likely’ responses, this worked against the positive aspects that the FFT could bring. Some even felt the score was being used to portray them in a negative light.

Top 2 vs. bottom 2 would be much fairer, much more honest result. I really like that. All opinions should be valid. The score changes our perspective on the FFT ---- it turns us against it. Nurse

Patients and the public also expressed misgivings about how ‘neither / nor’ and ‘don’t know’ responses were being used in the calculation. There were suggestions that ‘neither / nor’ was not a negative response and, as a result, should not be grouped with other ‘detractor’ responses, such as ‘unlikely’ and ‘extremely unlikely’. The ‘neither / nor’ option was also viewed as being synonymous with the ‘don’t know’ option. Patients and the public therefore queried the logic behind ‘don’t know’ being removed from the calculation entirely, while ‘neither / nor’ was included as a detractor. Therefore, the way they were treated differently from one another by the calculation could be confusing. Some of those participants who found this confusing went as far as suggested removing one of them from the FFT question with the


13-089420-01 | FINAL v1 | PUBLIC

assumption that those who are unable to give an opinion would not take part in the survey.

Although there were misgivings about the fairness of the scoring across all groups included in the research, stakeholders and some senior staff tended to be a little less vocal in their criticism of the FFT score being unfair. They believed that, as long as the system was the same for all Trusts, there would be little impact of the unfairness, beyond a lack of comprehension among patients. A few responses from Trusts to the online feedback also expressed this view in that they themselves did not have an issue with how the NPS was calculated, but felt a different or clearer explanation was needed for patients and the public.

5.2.4 Participants thought the scoring system restricted choice

There was a perception among patients and the public that the single score restricted choice. Instead of presenting data for all FFT answer codes, the overall score was seen to oversimplify the information available to patients, preventing them from being able to select the information of particular salience to them.

When presented with FFT data for three fictional Trusts that gave them the responses for each answer option rather than only the overall score, patients and the public often adopted different selection criteria when deciding which hospital they would like to be treated at. There was no consensus among patients and the public about the best way to interpret and use this data. Some considered the proportion of people who had recommended each Trust by combining the ‘likely to recommend’ responses. Others preferred looking at the proportion who were ‘unlikely to recommend’ each Trust. Some found it difficult to make a decision as they found the volume of information overwhelming. Different people therefore approached the data in different ways. However, by only providing patients with a single score, this flexibility for patients to use the data in their own preferred way was denied.

5.2.5 Preferences for calculating the FFT score

Participants across the different audiences therefore identified aspects of the FFT score that they disliked. Reflecting these concerns, many Trusts adopted different scoring systems when communicating internally and presenting the results back to patients, the public and staff, or did not use a score at all (for more on the presentation of FFT data, see Chapter 8: Communicating the results). This meant that few participants were aware of the formal FFT score for their Trust, hospital, ward or department.

The research highlighted the key features of how participants thought a scoring calculation should work. As part of this, alternative ways of both calculating and presenting the FFT score were explored with patients and the public, as well as with Trusts in the online feedback form (see

Different people want different information when making a choice. A single FFT score restricts this.


13-089420-01 | FINAL v1 | PUBLIC

appendices for details on the scoring options discussed), to assess what would increase their confidence when it came to using this data. The key features of the scoring system were:

All audiences favoured calculations that were simple and easy to understand. For this reason, percentages tended to be favoured above a net score as they were a familiar and more easily understood concept, while net scores were considered more complicated. For example, people preferred a combined percentage for likely to recommend, possibly alongside a combined unlikely to recommend percentage.

Patients and the public wanted to know the number of people responding for each hospital, ward or A&E department. They explained that this is available on sites such as Amazon, where the total number of product reviews is made clear, and it would provide an indication of how much confidence they could have in the score.

The score needed to take account of all potential responses to the FFT (for example, not to exclude ‘likely’ responses). Participants from across all audiences argued for this, to ensure that no responses were ‘wasted’ or discounted, and for the score to be seen as transparent.

When looking at different potential calculations for the score, patients and the public pointed out that two hospitals with superficially similar overall scores may have a very different range of responses behind that score. They therefore favoured a flexible system which offered an overall score, but also the option of exploring a breakdown of the data behind the score. This had the added advantage of providing transparency, as people would not feel there was an attempt to conceal any findings.

All audiences for the research felt that these features would address their concerns around comprehension of the score, fairness and transparency.

Having considered perceptions of the FFT question and score, the report now goes on to explore how Trusts are collecting FFT data.


13-089420-01 | FINAL v1 | PUBLIC

6 Data collection


13-089420-01 | FINAL v1 | PUBLIC

6 Data collection This chapter discusses the different methods of collecting the Friends and Family Test (FFT). It explores the ways in which Trusts have striven to make data collection as effective as possible, including the use of external suppliers and how the implementation guidance could better support them. It then explores two key issues for data collection:

How to ensure that everyone has the opportunity to complete the FFT; and,

The particular issues that impact on implementation of the FFT in Accident and Emergency (A&E).

Key findings

Pen and paper methodologies were the most common method of collecting FFT data, both for inpatients and in A&E settings.

Trusts weighed up a number of different considerations when selecting a data collection methodology, including familiarity (depending on other methods already used), perceived impact on response rates, ease of use for both staff and patients, the ability to generate honest and representative feedback and the extent to which they wanted real time feedback.

After implementation, Trusts made adjustments to data collection in order to improve response rates. Data collection worked best when processes were made routine, when staff were engaged, when a particular individual took responsibility for the FFT on a ward or in A&E, and when there was a clear Trust-wide FFT strategy around monitoring and improving response rates.

The FFT was typically conducted at discharge, a busy time for staff and patients; consequently the FFT was unlikely to always be a high priority.

Reaching the full eligible population was a challenge and it was clear that not all eligible patients were being asked the FFT. This depended on logistical constraints such as how busy staff were, sensitivity (e.g. those who were going to move into a care home), capability (e.g. those who had dementia) and, on occasions, deliberate avoidance where a patient was particularly ‘difficult’ or agitated.


13-089420-01 | FINAL v1 | PUBLIC

6.1 Data collection methodology

A range of different methodologies, from pen and paper through to tablet devices and text messages, were being used to collect FFT responses.

Comments cards and postcards were the most commonly used data collection methods, both in A&E (70%) and in inpatient settings (64%). Other popular methods for A&E included open internet surveys (used by 33%), paper questionnaires (27%), texting (26%), and token systems (also 26%). Common methods in inpatient settings included paper questionnaires (42%) and an open internet survey (33%).

Figure 6.1 ---- The different data collection methods used by Trusts

70

33

27

26

26

19

18

13

11

3

1

10

64

33

42

7

3

4

24

6

11

4

12

9

Comments cards/postcards

Internet survey - open to anyone to complete

Paper questionnaire

Texting (SMS)

Token system

Kiosk - e.g. in the ward

Tablets (eg iPad or similar)

Telephone survey i.e. speaking to an interviewer

Internet survey - via a link in an email

Mobile phone app (not SMS survey)

Bedside terminal

Other

% A&E

% Inpatient Settings

Base: All Trusts that ask FFT in; A&E (90); Inpatient settings (95). Fieldwork dates: 12th

December 2013 - 2nd February 2014. Source: Ipsos MORI

Which of the following methods are used to collect the FFT feedback in your Trust in. . .?

Improvements to the guidance were suggested, including the

pros and cons of different data collection methodologies, when the FFT should be asked, potential follow-up questions that could be included, and how to enable a wider range of patients to take part in the FFT.

External suppliers were being used for little beyond providing FFT questionnaires and generating datasets. Decisions about using suppliers were based largely on practical considerations of the cost, quality and resource implications of using a supplier versus managing the survey in-house.

A&E was a particular challenge for data collection, with lower response rates than inpatient wards. This was driven by the high volume, rapid turnover and the type of patients attending A&E. Trusts had tackled this by increasing publicity, engaging all staff, and carefully choosing a data collection methodology (sometimes multiple methodologies).

Postcards and paper questionnaires were the most commonly used methodologies.


13-089420-01 | FINAL v1 | PUBLIC

The use of different data collection methods in A&E and inpatient settings reflected the practical considerations that apply to each. For example, bedside terminals are only practical where patients spend sufficient time in a bed; something that is reflected in the data (only one per cent of Trusts used a bedside terminal in an A&E setting compared to 12% of Trusts in an inpatient setting). In contrast, the token system was more widely used in A&E as it was seen as a quicker way of collecting data from more transient patients who were spending less time in the hospital (26%, compared with three per cent on wards).

The data shows that there was greater diversity of data collection methods used in A&E than in inpatient settings; A&E departments using an average of 2.5 modes, compared to 2.2 modes in inpatient settings. This reflects the finding that Trusts found it easier to make the FFT data collection work in inpatient wards than in A&E, as discussed later in this chapter. On inpatient wards, the first method adopted usually achieved the response rates required (though this did not always happen immediately). However, staff stated that meeting response rate targets has been more of a challenge in A&E and, as such, more methodologies have been employed in order to achieve this.

6.1.1 Primary data collection methods

Many Trusts were using multiple data collection methodologies, particularly in A&E, and so they were also asked what they considered to be the main methodology. While the two paper-based methods dominated among inpatient wards, A&E again saw a wider range of methodologies being used. As outlined in the online feedback, this again reflected the harder task, and subsequent trial and error, witnessed when attempting to establish how best to collect the FFT data in A&E.

Comments cards or postcards (38%) were the most commonly used main data collection method in A&E settings, followed by tokens (18%) and paper questionnaires (14%). A similar pattern emerged in inpatient settings with comments cards and postcards the most popular (52%) followed by paper questionnaires (28%). The case studies suggested that the majority of comments cards and postcards were completed by patients while still within the hospital.


13-089420-01 | FINAL v1 | PUBLIC

Figure 6.2 ---- Primary data collection methods

6.2 Choosing a data collection method

Choice of data collection methodology in each Trust was driven by a range of considerations as outlined in the case studies and online feedback. These included:

prior familiarity with a particular methodology among Trusts and staff;

the impact the method would have on response rates;

the ease of use for staff and patients;

the ability of the method to provide qualitative feedback;

the resources / cost required to implement the FFT; and

the likelihood of more honest and representative data delivered quickly.

No single method perfectly addressed each of these criteria, nor was ideally suitable to each and every setting or context. The challenges presented by inpatients and A&E were very different, as were those presented by different types of patient. Consequently, different Trusts adopted a solution based on their own priorities and experiences.

The section below explores the considerations that Trusts weighed up when deciding how to administer the FFT.

6.2.1 Familiarity with the methodology

As outlined in the case studies and in the online feedback, Trusts that were already collecting patient experience data before the introduction of the FFT found it easier to build on this established system. They either integrated it

38

18

14

13

8

2

1

1

2

2

52

1

28

3

1

8

2

2

1

1

Comments cards/postcards

Token system

Paper questionnaire

Texting (SMS)

Telephone survey ie speaking to an interviewer

Tablets (e.g. iPad or similar)

Kiosk - e.g. in the ward

Bedside terminal

Internet survey - open to anyone to complete

Other

We don't have a main method, they are all used equally

% A&E

% Inpatient Settings

Base: All Trusts that ask FFT in A&E (90); Inpatient settings (95). Fieldwork dates: 12th

December 2013 - 2nd February 2014. Source: Ipsos MORI

And which is the main method used to collect the FFT feedback in your Trust in. . .?

Th

13-0

intofambesathe

FosecoFF

WethifaiInf

Homebewememareschsu

6.2

Wipaan

Foto staprodusaco

PonocoNH

In metheon

he Friends and F

089420-01 | FINAL

o their curremiliar, approanefits from dme equipmeerefore reduc

r example, alected this mmmunicatingT.

We were alrehis method. airly tech saformation Ma

owever, this wethodology, e followed by

ere concerneethodology. ade in some sponses theyanged the mpplemented

2.2 Influenc

th a CQUIN rticularly keed which, if n

r instance, abe both the

aff were ableocess. This e to the fact me way, nurmplete it, fur

ostcards areow achievinomplete anHS Trust

contrast, nuressage had ae FFT via a tee case study

Family Test: Qua

v1 | PUBLIC

nt system, oach. Trusts

doing this, inent and suppcing training

a Trust condumethod becag with patien

eady doingWe though

avvy here danager

was not the ceither becau

y their local ped about meeThe online fTrusts wherey were receiv

methodology their existing

ce on the re

target set aren to adopt aecessary, ga

a key benefit high respon

e to have ownwas becausthat patients

rses and suprther boostin

re relativelyng good re

nd post thei

rses and supa different exext messagey Trust that u

alitative research

r adopted a perceived acluding the e

plier, as well costs.

ucting the FFuse they alre

nts in this way

g outpatienht it would due to remin

case for all Tuse they werepatient expereting responfeedback alse there was ving from theto somethin

g system wit

esponse rat

round FFT rea methodoloave them co

of the postcse rates achnership of anse postcardss completed

pport staff weng the likely r

y straightfoesponse rateir cards be

pport staff in xperience. We within 48 housed text me

h

similar but s range of praefficiencies oas making it

FT through teeady had any that could

nt remindersbe easier t

inders.

Trusts and soe not aware rience questnse rate targeso suggesteda pressure toeir chosen mg else compth additional

te

esponse rategy that enabntrol over thi

card and tokehieved and and influence yielded a la them while

ere availableresponse rat

orward to ates by enco

efore discha

Trusts admiWith this appours after dis

essaging des

separate, andactical and fof being ablet easier for st

ext messagen established

be easily ad

rs via text sto manage

ome adoptedthe FFT queions, or becaets using thed that chango boost the n

method. Sompletely, while

methodolog

s, Trusts webled a high reis aspect of t

en methodslso the extenover the dat

arge number still in the ho

e to encouragte.

administer aouraging paarge.

inistering theproach patiescharge. Nuscribed their

d thus financial e to reuse thtaff and

e had d system of dapted for th

so went withe. We’re

d a different stion could ause they eir existing es had beennumber of

me had others

gies.

re esponse ratethe FFT.

was thoughtnt to which ta collection of response

ospital. In thge patients to

and we arepatients to

e FFT via texnts were senurses at the

frustration a

e

e

th

n

e

t

es e o

e

t nt

at

Posrelastraadnowresencto cthedis

NHS

stcards aatively aightforwminister aw achiev

sponse racouragingcomplete

eir cards bscharge. S Trust

4

re

ard to and we aring good tes by g patient

e and posbefore

47

re

s st


13-089420-01 | FINAL v1 | PUBLIC

being pressured to increase the response rate, while having little control over it apart from promoting the survey more heavily to patients.

We have no control over the response rate by doing it through text message. We can only explain it to them and check we have the right phone number. Nurse

6.2.3 Ease of use

The number of staff and the time they spent dedicated to distributing and collecting the FFT was an important consideration for Trusts when selecting a methodology. Similarly, reducing the burden for patients was also a factor.

Trusts that used a paper-based method acknowledged that collection of FFT was labour intensive, both in the distribution and collection of the survey to patients and in the subsequent data analysis. It meant that nurses and ward staff were all usually involved in the promotion and dissemination of the FFT, although some wards did identify particular types of staff to distribute and collect the FFT such as ward clerks or volunteers.

A huge amount of resource is spent on gathering it. Maybe we should move away from paper, but that would impact on response rate which we’d be beaten up for and wouldn’t work for A&E. Director

Across the case studies and online feedback, the staff time involved in managing paper based methods was seen as a potential challenge for the future expansion of the FFT into outpatient services, because of the increase in potential numbers of responses. Trusts felt that the increased number of patients may make paper based methodologies unaffordable and suggested that alternative data collection methods (such as texting, tablets or kiosks) would be needed to ensure that the FFT was sustainable. Trusts in this position were concerned that they would not have the financial resources needed to implement a new system.

With the impending inclusion of daycase, all outpatient departments and paediatrics, we will need to find an electronic solution to meet all the demands of FFT but we will have no financial resources to undertake this. NHS Trust

However, Trusts using other data collection methods such as text messaging removed the burden of distribution and collection of the FFT (though not the promotion) from the hands of ward staff, thus taking up less of their time than a paper-based method.

Th

13-0

6.2

Trufroresim

Ththefeewhhotruthefolelarepkno

In thethe

Pawato Wa

Pawhsugwewathe

YotheonGe

Otvomobia

Whgu

he Friends and F

089420-01 | FINAL

2.4 Capturi

usts wanted m patients, a

sponse rate. portant than

e consensuse FFT after dedback. Conhich patients spital. In pathful accoun

e survey. Thlow-up quesaborating on percussions own problem

contrast, in Te patient hade question pr

atients are rant to upse

o their care ard manager

tients and mhen providingggested that

ere still in hosas, therefore,em once thei

ou might behe ward. Betnce you’re seneral public

her Trusts adlunteers to dore frank feedased or incom

When we firstuiding peop

Family Test: Qua

v1 | PUBLIC

ing honest

to ensure thalthough this Anonymity meeting res

s among patischarge woncerns were would answ

articular, nursnt if they themis concern w

stion. Nursesany issues tand a reluct

m with more f

Trusts whered returned horivately and t

reluctant toet nurses anif they are r

members of thg feedback at they may bspital due to , important toir treatment h

e too scareetter to do itsafely hom

c discussion

ddressed thidistribute anddback aboutmplete feedb

st started dople through

alitative research

responses

at they receis was balancwas therefor

sponse rate t

tients and staould ensure a

raised by frower the FFT ifses worried tmselves or awas particulas believed ththey had exptance to giveformal comp

e the FFT waome, nurses therefore mo

to give negand are wary

e critical.

he public alsabout the ca

be reluctant tthe fear of c

o these partihad been co

ed to answeit either righ

me. group

s problem bd collect FFTt staff and caback.

doing it ---- it wh it. How wo

h

ived accuratced against tre regarded targets.

aff was that banonymity anontline staff af it was compthat patients nother clinic

arly pronounchat patients mperienced due negative feplaints proce

s asked by tfelt it allowe

ore honestly.

gative commry of retribu

so felt that anre they had o give negat

consequencecipants that

ompleted.

er truthfullyht at the do

by employingT surveys. Thare, therefore

would be mwould you ge

te and honesthe need for at times as l

being askednd, thereforeabout the hopleted while may not alw

cian were adced when thmay avoid mue to fear of

eedback diredures.

telephone ord the patien

ments. Theyution or con

nonymity wareceived. Ttive feedbaces on their trthe FFT be a

y when youoor of the h

g non-uniformhis was felt te reducing t

me with theget a true re

st feedback a good ess

to completee, honest onesty with still in

ways give a ministering ere was a entioning or

ectly to staff:

r text once ts to answer

ey don’t nsequences

s important hey

ck while they reatment. It asked of

u’re still on hospital or

med staff or o prompt he risk of

e iPad eflection?

e

a

es

It’s hondalimpeothro

Nurs

great to nest answnger is it

mited amoople that ough text

rse

4

get an wer. The reaches a

ount of we serve

t.

49

a

e


13-089420-01 | FINAL v1 | PUBLIC

Now our ward clerk does it in normal clothing, and we get a balanced view. Nurse

6.2.5 Capturing a representative cross-section of patients

Senior staff, frontline staff and patients all recognised the danger of some methodologies biasing responses towards some demographic groups (such as younger people more familiar with using technology). They felt it was important to avoid this for two reasons: firstly, to allow all patients an equal opportunity to feedback; secondly, to avoid receiving unrepresentative feedback and ensure that the issues identified in the FFT data are the issues that truly needed to be addressed.

For example, text messaging was felt to bias responses towards younger patients, who were thought to be more comfortable with such technology. In contrast, both patients and staff alike believed that this approach would be less likely to capture elderly patients, as they were considered less likely to own a mobile phone or be comfortable using one.

It’s great to get an honest answer. The danger is it reaches a limited amount of people that we serve through text. Nurse

Some Trusts indicated in the online feedback that they made it a priority to be as inclusive as possible. They tested certain methodologies with different patient groups. As a result, they established that postcards were appropriate to allow elderly patients and those with learning difficulties to participate.

There is a large elderly population. We undertook a consultation exercise with inpatients prior to commencing the FFT and asked them which method they would prefer to use. They chose the postcard methodology. NHS Trust

6.2.6 Real time feedback

The FFT data often provided many of the case study Trusts with a better approximation of real time feedback than anything that had existed previously. However, different data collection approaches delivered different versions of real time feedback, providing faster, more visual or more immediately actionable data, depending on the preference.

Nurses on wards using the paper questionnaires or postcards often checked the responses and read them on a regular basis. Cards from the return boxes were often reviewed on a daily basis. This allowed positive feedback to be shared immediately, or for problems to be identified and addressed straight away. This did have implications for anonymity,

The FFT provides the closest yet approximation of real time feedback, though some methodologies are better at this than others.


13-089420-01 | FINAL v1 | PUBLIC

although it allowed ward managers to instantly identify potential improvements.

The postcard method enabled an interface with patients about their care and an opportunity for staff to see comments quickly and address problems or emerging themes. NHS Trust

On the other hand, this only ever provided a snapshot based on the completed FFT forms that were immediately to hand at any given time, rather than a broader look at all the feedback provided. Furthermore, collating the quantitative data via this method proved time-consuming and some staff felt it would be useful if scores could be produced more quickly, particularly for more senior staff, so that they could identify potential problems more quickly.

It will be helpful to be able to see results quicker ---- there is currently a ‘lag’ on getting the paper data. Patient Experience Lead

The token system was seen to be helpful in this sense. The transparent token boxes provided a quick visual display of the responses provided by patients, allowing staff to monitor their progress as feedback comes in.

We get fast feedback with tokens: staff are constantly looking at the see-through boxes to see how they and other wards are doing. Patient Experience Manager

Those with experience of the technology-based methods – such as text messaging or tablets – spoke highly of the speed with which FFT data was available to them. For example, patient feedback entered onto a tablet was available within the main database of responses as soon as the survey was completed. While this offered similar advantages to real time examination of paper responses, it was much more comprehensive, providing immediate access to all feedback collected, not only the feedback on the cards that happened to be available on a ward at any given moment. Given the relative lack of computers available to staff on wards – one or two per ward – the paper postcard feedback proved more accessible to frontline staff.

As noted, these considerations were all weighed up when Trusts made their decisions about which data collection methodologies to use. This chapter moves on to explore the actions that Trusts took to improve data collection and increase response rates.


13-089420-01 | FINAL v1 | PUBLIC

6.3 Making FFT data collection work

As Trusts have built their experience of conducting the FFT, they have begun to apply the lessons they have learned around ensuring that data collection works well and that the response rate is on target. This is important given that, with CQUIN payments linked to the FFT response rate5, there is a financial incentive to ensure the Test is completed by as many patients as possible.

There were a number of factors that influenced how effectively data collection worked in Trusts. It was important to have the collection of the FFT embedded into a ward’s routine, to have frontline staff engaged in the process, and to have specific members of staff within each ward or department responsible for collecting the FFT. Trusts had also developed organisation-wide approaches that aided the smooth running of the FFT. The section below explores these themes.

6.3.1 Making the FFT routine

Collection of the FFT worked best when it was routine and systematised. This was inherent in some methodologies such as text messaging or telephone interviews, as a system would be set up with a supplier to ensure patients were asked about their experience within two days of discharge.

However, it was more difficult for other methodologies that relied heavily on frontline staff. Making the distribution and collection of postcards routine was challenging for some and Trusts recognised that having a clearly defined and understood system was key to overcoming data collection difficulties. In particular, integrating FFT collection into the discharge process helped nurses in charge of administering it to remember to give it to the patient. For example, FFT postcards would be attached to discharge documentation, or incorporated within patients’ paperwork at A&E, to ensure they were not forgotten during busy periods.

Engaging staff in the process, so that the postcards are routinely given to patients on discharge was difficult. However, it is now becoming more a part of routine practice and response rates have increased considerably. NHS Trust

Creating these sorts of processes in order to achieve the response rate targets has helped to improve the number of responses Trusts receive and appears to be a factor in the overall national level increase in FFT response rates.

5 http://www.england.nhs.uk/wp-content/uploads/2013/02/cquin-guidance.pdf

Trusts have been primarily focused on meeting the CQUIN response rate targets.


13-089420-01 | FINAL v1 | PUBLIC

6.3.2 Engaging staff in the FFT

Encouraging ward managers to see the importance of gathering the FFT has been absolutely central to the successful collection of the FFT. Where this has been possible, it has resulted in a strong drive behind the data collection, with frontline staff feeling a sense of ownership over the FFT on their ward.

One important way of engaging staff has been to demonstrate the added value of the FFT, for example by sharing positive feedback about jobs well done, or demonstrating how improvements can be made as a result of feedback through the FFT. Thus, Trusts have found that the feedback gathered from the FFT has been instrumental in encouraging staff to collect the data.

The main positive was the buy-in from the wards once they were getting rich positive data back which they now display on the wards. NHS Trust

As for any new initiative, it was clear from the case study Trusts that some wards embraced the FFT more than others, and data collection was therefore working better in some wards than others. This depended on how engaged the staff on each ward were and whether any individual staff had taken responsibility and ownership for the FFT.

Trusts responding to the online survey suggested that maintaining staff engagement with the FFT would continue to be a challenge in the future. In particular, some suggest that there could be an on-going tension between the need to maintain enthusiasm for the FFT and the need to meet other priorities.

Keeping the momentum going is challenging. As we focus on one area to help them improve response rates, another area may be put under operational pressures. NHS Trust

6.3.3 Assigning responsibility

Wards typically assigned specific people the task of collecting the FFT, thereby establishing clear roles and responsibility. At other times, certain staff members happened to be particularly positive about the FFT and took it upon themselves to establish it in their wards as ‘champions’, without necessarily being assigned to it formally.

Those tasked with distributing and collecting the survey tended to be among the nursing and administrative staff. However, in A&E departments it was common for medical staff to be overseeing the FFT because they were more involved in the running of the department than a doctor or

Th

13-0

cowawaco

WewaareNH

Uswanusoco

BecleteafutAs

Papefeeintethemabe

WhpegaPa

6.3

Haof entheanto wa

he Friends and F

089420-01 | FINAL

nsultant wouard manager ard or departllection of th

We use the ward to distrre waiting foHS Trust

sing administas seen as a rses and so me suggestillect the surv

etter to havlerks, houseeam of paidture thoughsociate Chie

tients and thrson collecti

edback anoneraction wouey were beinay be misinteing asked fo

What would merson sayingates and gitient discuss

3.4 Introduc

aving a Trustthe FFT wascouraging d

e response ray issues withdifferent are

as on the age

Family Test: Qua

v1 | PUBLIC

uld be when r or matron wtment, with the survey, an

ward adminribute survefor their dis

trative staff sway to imprreducing theons of potenvey to patien

ve individuaekeeper etcd people. Th. ef Nurse

he public alsng their feed

nymously wauld suggest tg listened to

erpreted as cor their feedb

make me mng ‘‘I’m monive them thsion group

cing a Trust

t-wide appro also seen toata collectioates on a weh data collecas of the Truenda for all f

alitative research

working acrowas responsihe task of pr

nd circulating

n, housekeeeys and proscharge pa

such as housove response risk of the ntially payingnts, though b

als on wardtc. but thereThat probab

o discusseddback. Whileas important, that issues w

o. However, confrontationback face-to-

more honesnitoring thishe good an

t-wide strate

ach to promo be beneficon. FFT leadeekly or monction. They wust to talk to srontline staff

h

oss differentble for oversromoting FFTg the results.

eeper and nomote the T

apers.

sekeepers, wse rates by eFFT being ov

g people to pbudgets rarel

ds responsie’s no budg

ably could b

the benefitse they acknothey also be

were being tathere was a

n and not all -face.

st would beis hospital’’nd the bad.

egy

oting and imial for respons or senior nthly basis, re

would follow staff and prof.

t wards. Genseeing the FFT distribution

nursing stafTest whilst p

ward clerks oasing the prverlooked. Tpromote, distly permitted

sible such aget to havebe an optio

s of having aowledged thaelieved that faken seriouslso recognitiwere comfo

e an indepe’…I’d open .

mproving the nse rates an

nurses closeleacting if thethis up with

omote the FF

nerally the FT in their

n and

ff on the patients

or volunteersessure on There were tribute and this.

as ward e dedicatedon in the

dedicated at giving face-to-face sly and that on that this rtable with

pendent n the flood

distribution d ly monitoredey suspectedregular visits

FT, ensuring

d

d s it

Thethewagetdatnowwa

NHS

e main poe buy-in frards oncetting rich ta back ww display

ards. S Trust

5

ositive waom the they werpositive

which they on the

54

as

re

ey


13-089420-01 | FINAL v1 | PUBLIC

Every Monday we look at response rates, and there’s a pool of people we can call upon to check the FFT. As a result, we have seen slow and steady improvements in numbers we collect. Associate Chief Nurse

A similar example was seen in an A&E, which had been achieving a low response rate. The FFT response rate was included in the daily ‘situation report’ highlighting how things were running in A&E, which meant it had visibility in A&E, keeping it high on the agenda and enabling immediate action to be taken when rates dipped.

We put a section on FFT in the sitrep [situation report], so instead of emptying the boxes with the cards at the end of the month…we went through a phase in December where they were emptying them every day. Chief Nurse

Promotion and communication of the FFT in a Trust-wide manner was seen as key to its success. In particular, implementing a programme of ongoing communication across the Trust from Chief Executive to ward staff emphasised the importance of the FFT. This meant they were more likely to administer it (depending on methodology) or to promote it to patients, as well as to look at the findings. Other methods of achieving this Trust-wide drive included publicising the survey with posters, displaying the results, and getting staff to talk about the survey. For example, one case study Trust had publicised the FFT at the front reception for all patients and staff to see (see fig 6.3). The ultimate aim of all these interventions was to embed the FFT into the Trust culture and assist it in becoming an integrated part of the patient journey.

Posters in the wards, displaying results, embedding it in the culture, getting all staff to talk about it. It’s new and difficult ---- it’s a change to how we work. You need to get people to own it. Consultant

Promotion and communication was seen as key to the success of the FFT.

Th

13-0

Fig

Pleco

Thflexstaapdifarrsetpe

WaknHe

Fulocmeto

ThdoSta

Staim

he Friends and F

089420-01 | FINAL

gure 6.3 ---- P

ease refer to mmunicated

6.4 The

ere was a coxible. It wasakeholder intpropriate for

fferent wardsrangements, tting. For exn and paper

Wards are alnow their paead of Patien

rthermore, scal ownershipethod reducedetermine th

he flexibilityo somethingakeholder

akeholders aplemented q

Family Test: Qua

v1 | PUBLIC

Promotion o

the followingd to staff and

e need fo

onsensus thas clear from tterviews thatr A&E. Evens, each with t needed to w

xample, a Trur on elderly w

ll different. patients the nt Experience

enior staff anp of the FFT.ed the potenhe most appr

y helps Trusng with it an

also pointed quickly by bu

alitative research

of the FFT at

g chapter fod patients.

or flexibili

at the data cthe case studt what is appn within a Trutheir differenwork out howust might usewards.

You need te best.

e

nd stakehold. Allowing T

ntial for resistropriate met

sts to have nd it is more

out that flexiuilding on ex

h

the main re

r more inform

ity in data

ollection medies, online f

propriate for ist, those ove

nt patients anw the FFT coue text messa

to allow th

ders felt this frusts the freetance from Thod for their

e ownershipre meaningf

bility has alloxisting patien

eception

mation on ho

a collect

ethodology nfeedback aninpatients merseeing thend dischargeuld work bes

aging overall

hem flexibil

flexibility encedom to sele

Trusts, by allopatients and

p. It means gful to them

owed the FFnt feedback s

ow the FFT is

ion

eeded to bend ay not be FFT felt thate st in their but also use

lity as they

couraged ect their ownowing them d staff.

they can m.

T to be structures.

t

e

Youwartheypat

Headexpe

u need to rds flexibiy know thtients the

d of patient erience

5

allow lity as

heir best.

56


13-089420-01 | FINAL v1 | PUBLIC

The flexibility means Trusts can continue to use some of their old patient feedback systems rather than starting again. Stakeholder

This chapter now goes on to explore the additional challenges Trusts faced when implementing the FFT.

6.5 When should the FFT be asked?

It was clear that the administration of the FFT was not always understood by frontline staff. Some methods exacerbated this problem. The FFT guidance document states that ‘patients should be surveyed on the day of discharge

or within 48 hours of discharge’6. Where the distribution and collection of the FFT was dependent on frontline staff, most were aware that the question must be asked at or around discharge. However, there was evidence that in some cases the guidance had been misinterpreted; for example a nurse described having to distribute the FFT two days before a patient was discharged.

Once you know for definite they are going home you have to give them the FFT up to 2 days before discharge. It is done separately to the discharge process. Nurse

There were other cases, however, where nurses were not aware that it should only be administered at discharge, and instead the FFT was being circulated at various times throughout a patient’s stay, therefore not adhering to the guidance and capturing patients’ views at the end of a stay.

Furthermore, nurses often questioned whether discharge was in fact the most appropriate time to deliver the FFT, largely due to staff availability and timing. Discharge is invariably a very busy time, and the FFT was described by nurses as the last thing on a ‘very long list’ of priorities. This meant that distributing and collecting the FFT did not take precedence, particularly on very busy days. In addition, nursing staff could also simply forget to ask patients to complete the FFT. This inevitably had an impact on response rates, as the fewer patients given the opportunity to complete the FFT, the lower the response rate.

This is not the priority. I’d prefer to discharge my patient safely with medicines and prescription. If I forget one of these because of the survey that’s a big problem. Nurse

6 https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/213047/NHS-Friends-and-Family-Test-Implementation-Guidance-v2.pdf p16


13-089420-01 | FINAL v1 | PUBLIC

Patients and public views on the best time to be asked the FFT were mixed. Many felt there were advantages in receiving it once they had returned home (whether via text, email or letter). This would allow them sufficient time to think about their answer, as well as relax after their hospital visit.

I would prefer a considered answer ---- when I leave hospital the last thing I want to do is fill out a form. Patient discussion group

This also reduced the fear of receiving less good care if negative feedback had been given while still in hospital. Others felt that receiving the FFT at discharge would be more convenient and encourage them to answer it as the experience was fresh in their mind, in contrast to staff who often thought this time was too busy.

6.6 The eligible population

The implementation guidance7 specified that the following groups should be surveyed in inpatient settings and A&E:

Adult acute inpatients (who have stayed at least one night in hospital); and,

Adult patients who have attended A&E and left without being admitted to hospital or who are transferred to a Medical Assessment Unit and then discharged.

Trusts struggled to conform to these requirements, and this was particularly the case for data collection methodologies that relied on frontline staff to administer the FFT, such as postcards.

Where the FFT was not being administered to all eligible patients, the case studies suggested there were three key reasons why all eligible patients were not being asked to complete the FFT. Common across all case studies, these were:

Logistical constraints due to staff workload at busy times;

Sensitivity to patients’ needs and capabilities on the behalf of some patients; and,

In extreme cases, occasional deliberate avoidance of some types of patient.

Each of these factors helps explain why the FFT is not always asked of all patients. However, some patients were being asked the FFT more than

7 https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/213047/NHS-Friends-and-Family-Test-Implementation-Guidance-v2.pdf p14

Asking the FFT of the whole eligible population was not possible or practical, particularly when relying on frontline staff.

Th

13-0

onres

Th

6.6

DisFroengotimdegecofor

WedisHe

At thisbuFF

Thgivto Nu

Wias adconu

Laanit menavathatho

WecawiCo

he Friends and F

089420-01 | FINAL

ce during a sponse rate c

ese issues a

6.1 Logistic

scharge wasontline staff isuring that ping to get ho

mes such as partments. t home as somplete the F

r example).

We haven’t cischarge is ealthcare ass

times, therefs meant thatsy times. In T to individu

hought it waive out whe

o give it to aurse

th different lewithin wardsministered tonstraints coumber of fron

rger Trusts ad diverse po

more difficultcourage theailable, very an English. Sose not able

We've thoughan go in anithout the laommissioner

Family Test: Qua

v1 | PUBLIC

hospital visitcalculation.

are explored

cal constrain

s widely acknndicated tha

patients undeome safely. Christmas, oPatients, on oon as possiFFT (unless t

come up wa rush so it

sistant

fore, frontlinet the FFT was addition to tal patients, a

as a good ien people aabout 2 in 5

evels of buy-s, some will o all eligible uld be comptline staff to

also discusseopulation. Fot to engage wm to complefew respons

Some Truststo complete

ght about hond encouraglanguage c

alitative research

t, that is they

in the sectio

nts

nowledged tat the FFT woerstood how This issue w

or in particulathe other haible and wouhey could co

with a good it’s not alwa

e staff priorits not given tothis, staff coagain particu

idea ---- but are dischar5 patients.

-in among stalways be mpatients. It

pletely addreadminister t

ed the challeor example, with a significete the FFT. ses were act were consid the FFT in E

holding daysage people capability.

h

y were being

on below.

o be hectic fould never bto handle th

was exacerbaarly busy setand, simply wuld not want omplete it wh

d way of hanays the bes

tised other iso all eligible uld simply foularly when t

t it can be hrged. I pro

taff, across Tmore committ

is difficult to ssed where he Test.

enges that colanguage ancant proportAlthough tratually receivedering the usEnglish.

ys where coe to fill it in,

double-cou

for staff and e a higher p

heir medicatiated at particttings, such awant to leaveto take extrahile waiting f

anding it oust time.

ssues over thpatients, pa

orget to admthey were bu

hard to remobably only

Trusts and wted to ensurisee how theFFT relies on

ome with sernd cultural bation of patienanslations of ed from langse of volunte

ommunity vor help pe

nted in the

patients. riority than on and werecularly busy as A&E

e hospital anda time out to for discharge

ut yet;

he FFT and articularly at inister the

usier.

member to y remembe

wards as well ng the FFT is

ese n a large

rving a largearriers made

nts and the FFT weruages other

eers to help

volunteers eople

e

d

e

er

s

e e

e r

Disit’s tim

Hea

scharge isnot alwa

me. althcare assis

5

s a rush soays the be

stant

59

o est


13-089420-01 | FINAL v1 | PUBLIC

6.6.2 Sensitivity and capability

In many cases, patients were not being asked the FFT because, while eligible, they were deemed incapable of answering a survey. To illustrate, all case study Trusts described difficulties administering the FFT in wards with large numbers of elderly patients, many of whom suffered from dementia, where patients lacked the ability to complete a survey. The language ability of patients, sight and cognitive impairments, as well as the need for sensitivity when approaching terminally ill patients, were also barriers to distributing the FFT to all patients.

It’s difficult for many, particularly elderly with dementia, people for whom English is not their first language, people with sight or cognitive impairments. Patient Experience Lead

Nurses were managing these problems in a number of ways. Many were supervising patients more closely as they answered the FFT. Alternatively, nurses were encouraging friends or relatives to complete the FFT on a patient’s behalf – especially common for those patients with dementia, although frontline staff were unsure of what they should do in these cases and would like additional guidance.

Elderly care struggles due to the type of patient so they encourage relatives to complete it, or volunteers to do it with them. Patient Experience Manager

One case study Trust using the token methodology chose to add images of smiling and frowning faces to the answer scale, as they found this made it easier for patients to answer – particularly those with learning disabilities.

Th

13-0

Fig

6.6

In panarelin omoto

I’vanprodocoNu

Reto

Seenresthewecla

he Friends and F

089420-01 | FINAL

gure 6.4 ---- H

6.3 Delibera

extreme castients the optural inclinatationships worder to biasore likely to bavoid putting

ve gone to gnd the last trobably goion’t want thonfrontationurse

egardless of positively bia

nior staff alscouraging psults. Howeve perhaps jusere adamant aimed that it

Family Test: Qua

v1 | PUBLIC

Happy face

ately not as

ses, staff sompportunity to tion to avoid

were particulas the FFT datbe positive. g frontline st

give it to sothing I wan

oing to tell mhem to say n.

the justificatas the FFT d

so raised conpatients to resver, there wastifiable avothat they cowas not hap

alitative research

token colle

sking the FF

metimes delibcomplete thpatients whoarly poor. Thta collection Rather, thereaff in difficul

someone whnt to do is gme where toy bad things

ion, it was clata.

ncerns aboutspond in a pas little evideidance of ‘diuld effective

ppening to an

h

ection

FT

berately avoe FFT. In pao were agitathis sort of avin favour of

e was a senst situations.

who’s been rgive them ato put it. Ra

gs’’, you just

ear to staff t

t the possibiparticular wayence that thisfficult’ patien

ely police anyny significan

ided giving sart, this was dted, or where

voidance waspatients whose that it was

really aggra card bec

Rather than st don’t wan

that this had

lity of ‘gaminy in order to s was happents. Further,y possible gat degree.

some due to a e s rarely doneo seemed s acceptable

ravated cause he’s thinking ‘‘I

nt the

the potentia

ng’ – of staff bias the

ening, beyon senior staff aming and

e

e

I

l

d

Ratdosayjustcon

Nurs

ther than n’t want ty bad thint don’t wanfrontatio

rse

6

thinking them to ngs’’, you ant the on.

61

‘‘I


13-089420-01 | FINAL v1 | PUBLIC

There are ways to skew the results: ward clerks could tell people how to answer it, or force people to answer. This methodology encourages or risks gaming. I’m confident in our data though ---- I know the people involved and know we’re careful. Director of Nursing

6.6.4 Asking the FFT more than once

While all Trusts struggled to meet the requirement to ask all patients the FFT, in some cases patients were being given the opportunity to complete the Test more than once. In these Trusts, staff rarely described any precautions they took to ensure patients were not asked the FFT multiple times. On the contrary, a few nurses described administering the survey every day. Therefore, if a patient is there for more than one day, they are likely to be asked the FFT more than once.

We’ve decided to give iPads to patients no matter how many times they want to use it, as long as they’re with us. Nurse

Other Trusts were aware of the challenges that double-counting presented and tried to minimise the extent to which it happened, but most systems were not flawless. Clearly, this double-counting, while rare, has implications for the accuracy of the FFT response rate. If some patients are being avoided and others are being asked more than once, this suggests that the accuracy of the response rate figures produced by some Trusts ought to be treated with an element of caution.

Some may get asked more than once ---- we’ve gotten over 100% response rates in a week before. There’s no way of making it better. Ward Matron

6.7 Implementation and guidance

Staff explained that the early stages of introducing the FFT were characterised by a focus on response rates – on the proportion of patients completing the FFT – and on developing processes for collecting data, managing the survey and engaging staff. Although mostly requesting flexibility around data collection, there was also a call for guidance on best practice.

The Department of Health provided all Trusts with guidance on implementing the FFT8. Trusts requested updated guidance on the following issues:

8 http://www.england.nhs.uk/wp-content/uploads/2013/07/fft-imp-guid.pdf


13-089420-01 | FINAL v1 | PUBLIC

Data collection methodologies: in the online feedback and the case studies, Trusts suggested that there was a need for more information and guidance around different data collection methodologies, the advantages and disadvantages of each and case studies of where they had been used.

Timing: the timing of when the FFT should be asked needed clarification. Senior staff, stakeholders and patients pointed out that those answering the question while still in the hospital may be thinking of different things compared to those who answered it when they returned home.

Using a follow-up question: while all of the Trusts suggested that the follow-up question was useful this did not necessarily translate into a consensus that it should be mandated as part of the FFT. Some believed there should be flexibility here so that Trusts could collect follow-up data through separate patient experience surveys or comments cards. That said, it was recognised that this would make it difficult to match data from the FFT question with any qualitative feedback. Some stakeholders suggested that there should be a ‘question bank’ of potential follow-up questions that Trusts could use. This would ensure that any additional questions are properly validated.

Hard to reach groups: some Trusts suggested that it would be useful to have additional guidance around how to ensure a wider range of patients can take part in the FFT, including:

o patients who are not literate, have low levels of literacy, or have poor English;

o patients who are moving to a care home;

o patients who have experienced a traumatic incident;

o patients with learning disabilities;

o patients with dementia or a cognitive impairment; and

o other patients who may need support in completing the FFT survey.

When responses should not be presented: having clear information about when the results for the FFT should not be presented, for example at what level of response / response rate do the results become invalid to present, and what to do in these circumstances.

Getting the right balance between flexibility and prescriptive guidance: flexibility in choosing what methodology to adopt for the


13-089420-01 | FINAL v1 | PUBLIC

FFT allowed Trusts to select a system that worked best for them and their patients. However, the subsequent wide range of methodologies adopted inhibited the comparability of results.

6.8 Using suppliers

External suppliers were used by many Trusts to help with the collection, management and processing of the FFT, as figure 6.5 illustrates.

Figure 6.5 ---- Use of external suppliers for different tasks

A majority of Trusts (54%) used an external supplier to generate a dataset of the responses. Patient experience leads within the case studies explained that this work could be time consuming, particularly if staff had to count the results themselves rather than using an automated system such as a scanner (for paper responses) or relevant software. The other tasks were more commonly managed within the Trusts, with only one in five using an external supplier to collect the responses (18%), one in ten using them to interpret the data (11%) and only one per cent using them to communicate the results to staff and patients.

Decisions around whether or not to use external suppliers were largely based on practical considerations of the cost, quality and resource implications of using external suppliers versus managing the survey in house; some felt that NHS staff do not necessarily have the skills needed to design and implement data collection methodologies or (to a lesser extent) to analyse and use the results.

We are using them because they are cheap and I didn’t have the resources to manage. Patient experience lead

However, questions were raised by stakeholders about whether all Trusts were equally well equipped with the necessary skills or knowledge to commission high quality external suppliers. In these cases, they suggested

54%

18%

11%

1%

46%

81%

88%

97%

1%

1%

2%

Generating a dataset of theresponses

Collecting the responses e.g. askingthe Friends and Family Test

questions

Interpretation of the data and/ordeveloping recommendations

Communicating the results to staffand patients

We use an external supplier The Trust undertakes the task itself Don't know

Base: All Trusts who responded (95). Fieldwork dates: 12th December 2013 - 2nd February 2014. Source: Ipsos MORI

For each of the following tasks please tell me whether you use an external supplier or if the Trust itself undertakes the task?

54%

54% of Trusts use a supplier to generate the dataset but only 11% use suppliers to interpret the data. Online feedback form among Trusts


13-089420-01 | FINAL v1 | PUBLIC

that there could be a role for NHS England in either regulating this market by insisting on minimal standards, or providing Trusts with the information they need to make an informed decision; for example, more information on the relative advantages and disadvantages of different data collection methodologies used by the suppliers.

Much more guidance around using external suppliers is needed. At the moment there is some really bad work out there. People need more advice. If the data collection isn't done right then it can cause real problems. Stakeholder

Patients and the public did not have a great deal of knowledge around the use of external suppliers. However, they did spontaneously discuss whether data should be collected by an independent body rather than the Trusts themselves. As we often find, the public does not always trust the objectivity of official statistics. Patients and public felt the involvement of independent organisations could provide additional reassurance about the veracity of the results.

There has to be an independent body in charge. General public discussion group

In practice, Trusts using external suppliers were still collecting the FFT data themselves, but patients and the public were not aware of the detail of how the FFT is collected.

6.9 The A&E challenge

A major theme emerging from the research was the array of challenges that A&E poses to FFT data collection. In the online feedback, this led to four in five Trusts saying the set-up of the FFT had been difficult in A&E (73%), while just three in ten Trusts said the same of setting up the FFT in inpatient settings (29%).

A&E presented a major challenge to the collection of FFT data.

Th

13-0

Fig

Thstuthe

ThapanNH

RecamaFF

AsA&

6.9

Truemhig

I cdepeA&

BD

HT

he Friends and F

089420-01 | FINAL

gure 6.6 ---- E

e challengesudies, where e FFT in an e

he A&E settippropriate nd commenHS Trust

esponse rateusing many

andated by tT in A&E par

High vol

Different

a result, Tru&E. Each of t

9.1 Volume

usts have strmergency degh turnover o

can’t imagietail of Frieneople lined &E Senior Nu

Base: All Trusts that ask December 2013 - 2nd Fe

How easy or dTrust in…?

Family Test: Qua

v1 | PUBLIC

Ease of set-u

s of setting u the challeng

emergency s

ting does noenvironment upon the

s in A&E depTrusts to riskhe national Crticularly cha

ume and rap

t types of pa

usts have hathese issues

e and turnov

ruggled to copartments. A

or throughpu

ine staff wilends and Fad up. urse

2%1

52%

21%

A&E

Very easy F

FFT in A&E (90); Inpatbruary 2014.

difficult has it

alitative research

up of the FF

up the FFT inges had led etting.

ot always leent where peir experien

partments wk failing to mCQUIN. Twoallenging:

pid turnover

tients.

d to develops is discusse

ver

ope with the A&E is a bus

ut of patients

ill spend thaamily Test w

17%

8%

Fairly easy Neith

tient settings (95). Fieldw

been to set-u

h

FT

n A&E also emstaff to ques

lend itself topeople are nce.

ere lower thameet the respo key factors

of patients; a

p their own bd below.

number of esy and fast p.

hat much timwhen they’v

er / nor Fairly dif

13%

27%

Inpa

work dates: 12th

up the Friends

merged fromstion the valu

to being anable to sto

an in inpatieponse rate tas made admi

and,

bespoke solu

eligible patiepaced enviro

me discussve got six o

fficult Very difficu

11%

47%

2%

atient settings

s and Family T

m the case ue of asking

n op, reflect

nt settings, rgets inistering the

tions for

nts in onment with a

sing the or seven

ult

Source: Ipsos MORI

Test in your

e

a In adayplaherto d

Med

a ward thys and yo

an aheadre you hado everyt

dical directo

6

here are 2ou can for it, but

ave 4 hourthing. r

66

2-3

t rs


13-089420-01 | FINAL v1 | PUBLIC

In the online feedback, larger Trusts were most likely to say the FFT has been difficult to set up and collect in A&E, particularly using paper and postcard methods; it follows that the larger the Trust, the greater the volume of patients. This, in turn, has created greater pressure on staff, which meant they were less likely to hand out the FFT survey.

In the hectic A&E environment, patient visits are unscheduled. Patients are seen and discharged in a relatively short space of time. In particular, Trusts, senior and frontline staff highlighted the four hour waiting time target as an example of the pressure placed on staff in A&E compared to the relatively slower pace on inpatient wards.

It’s very difficult to collect in the Emergency Department. In a ward there are 2-3 days and you can plan ahead for it, but here you have 4 hours to do everything. It’s difficult to drive staff to do it. Executive Medical Director

The often hectic and pressurised environment in A&E has meant A&E staff have been less able to plan ahead, making it difficult to build in time to ask the FFT. Furthermore, staff talked about how there were typically many forms for patients and staff to handle during these brief and pressurised interactions. This meant that staff could easily forget to mention the survey and patients were also less likely to consider it a priority to complete the FFT feedback.

In such a fast paced environment, it often feels like a burden for staff to have to give out and, most importantly, patients to fill in an additional form. NHS Trust

6.9.2 Type of patients

The case studies and online feedback also highlighted differences in the types of patients. Patients visiting A&E were often characterised as having been through a traumatic experience. A&E staff suggested that distributing the FFT to patients at this sensitive time was inappropriate. Consequently, distributing the FFT to the whole A&E population was extremely difficult, with responses biased towards those with minor ailments.

Asking the FFT at the point of discharge was also an issue in A&E departments. With discharge rarely occurring at specified or planned times, the FFT had been more difficult to work into a routine. A&E patients were also perceived as often not being admitted for long enough to build up a relationship with the health professionals – unlike inpatients – which therefore resulted in them seemingly less included to want to give up time to comment on the service they had received in this setting. In addition, A&E

Th

13-0

pato

AftleasuNH

Widesofol

6.1

PriThwanuthedahopa

WeresthePa

he Friends and F

089420-01 | FINAL

tients were pask them to

fter long waave the dep

urvey. HS Trust

6.10 So

th little guidaveloping thelutions met tlowing proce

Publicisi

Engagin

Distribut

Employin

10.1 Greate

marily, Trustis involved m

aiting areas, mber of box

e FFT also emta collectionw many resprticular day s

We’ve gone tesponse ratehe importan

tient Experie

Family Test: Qua

v1 | PUBLIC

perceived aspause and r

aits and treepartment, n

lutions to

ance or preceir own besphe FFT guidaesses:

ng the FFT w

g medical st

ting the FFT

ng a differen

er publicity

ts focused omaking the Ft-shirts for re

xes for deposmphasised th to staff. Tra

ponses had bstaff could th

to great lente; lots of punce and proence Manage

alitative research

s particularlyreflect on the

eatment timnot hang a

o A&E dat

cedent, Trustoke approacance, some

wherever pos

taff in the pro

at admission

nt, or multiple

y

n publicisingFT more visi

eceptionists siting postcahe value of cansparent cobeen collecthen collect m

engths to wopublicity in Aocess to staer

h

y eager to geeir experienc

mes many paround and

ta collect

ts dealt with ches to data of the solutio

ssible;

ocess;

n to A&E, rat

e, methodolo

g the FFT to bble within A&(see fig 6.5)

ards. Those communicatiollection boxeted. If numbmore respons

ork out howA&E waiting

taff.

et home, makce, no matter

patients jusd fill in a sa

tion issue

these obstaccollection. W

ons adopted

her than disc

ogy(ies) if ne

both staff an&E through p, and increascharged witng the impoes helped st

bers looked loses.

w to improvng areas, ex

king it harder how briefly.

st want to atisfaction

es

cles by While not all

d included th

charge; and

ecessary.

nd patients. posters in sing the h overseeingrtance of FFtaff to see ow on a

ve A&E xplaining

r

e

g T

Welenhowres

Patiman

e’ve gonengths to ww to impr

sponse ra

ent experiennager

6

e to great work out rove A&E te.

nce

68


13-089420-01 | FINAL v1 | PUBLIC

Figure 6.7 ---- Receptionist with an FFT t-shirt

6.10.2 Engaging medical staff

Unlike inpatient wards, doctors in A&E were dedicated to one department, as opposed to working across a number of wards. As a result, doctors in A&E tended to be more engaged and involved in the FFT data collection process than elsewhere in the hospital. Thus, A&E could better rely on all staff to promote the FFT to patients than colleagues on inpatient wards.

6.10.3 Distributing the FFT at admission

Given the hectic nature of A&E, and the many forms patients and staff are required to handle, one solution was to hand out the FFT postcards upon admission. This approach allowed patients to complete the FFT at any time during their visit. However, it resulted in many patients completing the FFT during waiting periods, which meant that their responses could not reflect any aspect of the treatment they would receive. In addition, distributing the FFT at admission is not in line with the FFT guidance.

6.10.4 Use the most appropriate methodology

Many Trusts decided on a different data collection methodology for A&E than inpatient settings, often following a period of trial and error. While postcard or paper FFT data collection remained the most popular methodology in A&E, Trusts were increasingly looking to replace or supplement this with other approaches. In particular, text messaging and the token method were under consideration as alternatives to improve response rates. Each of these methods relies less heavily on frontline staff

It was important to engage medical staff and adopt the most appropriate methodology in order to make FFT work in A&E.


13-089420-01 | FINAL v1 | PUBLIC

to distribute and collect surveys. For example, the text messaging approach requires staff to collect mobile phone numbers at admission or discharge, but nothing else.

The new SMS texting system is better, as it removes the burden from staff. NHS Trust

The token system had the perceived advantage of producing good response rates, being accessible to patients and providing a quick visual appraisal of how a department is performing. Additionally, comments from the case studies and from the online form suggested that other Trusts using this approach in A&E reported fewer issues gathering the required number of responses. However, it had the disadvantage that they would not be able to collect the valuable feedback from the follow-up question.

We introduced the tokens system to provide a more accessible method for answering the question and help improve the number of responses achieved. NHS Trust


13-089420-01 | FINAL v1 | PUBLIC

7 Communicating the results

Th

13-0

7

In copuwabe

he Friends and F

089420-01 | FINAL

7 Cores

this chapter mmunicatedblic. We the

ays in which st be shared

Key findi

Internal althoughof auton

Many waother inflocal suralthough

The wayTrusts. Wpublicatpresentasystems

Many Trfollow-upoften in tas a resqualitativ

The FFT focused follow-uptargets.

Patients communand so tthemselv

Family Test: Qua

v1 | PUBLIC

mmusults

we discuss d the Friendsen present thdiscussion g

d with them.

ngs

communicath ward manaomy to explo

ards displayformation sucrveys and mh the FFT dat

y in which FFWhile the forion guidanceations were cs, bar charts

rusts also prep question. the form of “ult of patientve FFT feedb

was often fe on responsep feedback,

and the pubnications. Rehey also wanves and inte

alitative research

unica

the differents and Family he public andgroup partici

tions were dagers were tyore and shar

ed the FFT rch as other petrics such ata was not a

FT results wermal score pe was sometcommonplacand simple p

esented the It was usualyou said / wt feedback. back alongs

eatured in Boe rates and talongside a

blic wanted ieducing the nted the optrpreting resp

h

ating

t ways in whiTest (FFT) re

d patient perpants sugge

riven by patiypically empre their own w

esults on ‘wapatient experas pressure ulways up-to-

re presentedresentation mtimes used, ace (for examppercentage f

qualitative fely presentede did” to shoSome Trustside their mon

oard papersthe FFT scorbattery of ot

ntuitive and data to one ion of explorponses in the

g the

ich Trusts haesults to stafrspective, exested the res

ent experienpowered withward’s result

ard boards’ arience measulcers and fa-date.

d varied acromandated inalternative ple, star ratinfigures).

eedback fromd on ward boow the actions published anthly board p

, although thre rather thanther metrics

easy to undescore was to

ring the dataeir own way.

ave ff and to the xploring the sults could

nce leads, h a degree ts.

among sures from alls,

oss n the

ng

m the oards – ns taken all papers.

hey n the and

erstand oo simple

Thisjustit's othaboth

Patigrou

s informat from us from us t

her. We ole to read

hers are sa

ent discussiup

7

ation isn't to them, o each

ought to bd what aying. on

72

be

Th

13-0

Thinte

7.1

Thbobrifolboinc

Typatheof setheindacele

he Friends and F

089420-01 | FINAL

7.1 Sha

ere was a wernally and e

1.1 With fro

e FFT resultsth directly thefings on walow-up quesards. The ra

cluded the fo

Team brdiscusse

Quality a

Email co

Monthly

Posters

Staff new

pically, comtient experie

e wider Trustthe latest dant directly toe data on thedependently cessible dat

ectronic colle

They warespons‘drill dow

Ideally, tsuch as on which

Family Test: Qua

v1 | PUBLIC

aring the

ide range of externally.

ontline staff

s were sharehrough monthards and indstion on publange of waysollowing:

riefings and sed;

and safety da

ommunicatio

reports to w

inside the Tr

wsletters.

munication oence leads ints respondinata – responso ward manaeir wards, or within their w

tabases of Fection and sh

anted accesses the data w

wn’ into the fr

they wanted infection rat

h to base the

alitative research

e results

ways in whi

ed most freqhly, weekly oirectly via plic-facing was that FFT re

staff meeting

ashboards o

ns and Trust

wards, boards

rusts; and,

of the FFT ren the case stg to the onlinse rates, scogers. Ward mwere empow

ward. The laFT feedbackharing of FFT

s to informatiwas based oree text resp

to see the Ftes, thereforeeir opinions.

h

ch the FFT re

uently with fror even dailyacing scoresrd boards ansults were s

gs, where sc

on wards and

t intranet;

s and CCGs

sults to staff tudy Trusts. ne feedback

ores and follomanagers wewered to expatter option uk, so worked T.

on such as ton, and also ponses.

FFT results ale providing a

esults were s

rontline nursy face-to-faces and feedbnd on staff inhared with fr

cores and co

d in staff roo

;

was handleIn both thes

k form, a sumow-up commere encoura

plore and useusually requi

best when l

the number owanted to b

longside otha range of inf

shared

ing staff, e team ack from thenformation rontline staff

mments wer

ms;

d by the se Trusts, andmmary reportments – was

ged to sharee the data red easily inked to

of e able to

er data formation

e

re

d t

e

If yavestaaskwothescomigbla

Chie

you stoppeerage me

aff on the ked wheth

orld revolve friends aore, I suspght get onank looks.

ef Executive

7

ed an ember of ward andher their ved arounand familypect you ne or two

73

d

nd y

o


13-089420-01 | FINAL v1 | PUBLIC

Patient experience leads within Trusts extolled the virtues of displaying the FFT publicly on every ward. However, the case study visits suggested that this did not happen quite as intended. There was a large degree of variability in how and when this was done on the wards, both across different Trusts and within Trusts. Often, the data on display was out of date by a couple of months or missing entirely. Most attributed delays or absences to particularly busy periods – it is worth noting that our fieldwork period started shortly after the New Year holiday and staff cited the festive period as their busiest time. In this situation, keeping FFT communications up-to-date was not a key priority for frontline staff.

There were cases where FFT results were not publicly shared on wards at all. Sometimes, this was because Trusts had chosen to not display this information. In other cases, staff were supposed to display the data but did not do so. Senior staff felt it was a challenge to convince all ward managers of the benefits of the FFT and, consequently, the need to display up-to-date data. This again emphasised the importance of staff buy-in to the successful administration and use of the Test.

Most of the wards kind of embraced it and took it in their stride, others were a bit more challenging…if you stopped an average member of staff on the ward, and asked whether their world revolved around the friends and family score, I suspect you might get one or two blank looks. Chief Executive

In contrast to the purposeful communication with nurses and ward-based staff, clinical staff received little direct communication about the FFT results. Consultants and doctors occasionally saw results on ward boards but never saw the results otherwise. The exceptions to this were those who saw board papers as part of their executive roles and A&E doctors who, as discussed in chapter 6, were much more engaged with the FFT due to being attached to a specific department.

7.1.2 With the board

While board members had occasional exposure to the FFT results through ward visits or even direct personal experience as a patient, their primary interaction with the FFT data was in monthly board reports. As discussed in chapter 5, the FFT score and response rates were typically communicated to board-level staff as part of a dashboard alongside many other metrics – typically up to 30 or 40 metrics in total, as figure 7.1 illustrates.

FFT response rates and scores were included in board reports among other metrics.


13-089420-01 | FINAL v1 | PUBLIC

Figure 7.1 ---- Publicly available board report dashboard

The FFT information was, therefore, only a small part of the information that a board discusses in a given month. While these board reports and the summary dashboards were published online, some Trusts also appended the full set of free text responses from the follow-up questions. In cases where senior staff were particularly sceptical about how the FFT is calculated, board papers included a caveat advising caution when interpreting the score, further underlining the negative opinion that exists around this issue.

7.1.3 With patients and the public

FFT results were displayed in leaflets or posters in the public areas of hospitals (see figure 7.2). Results were also shared within wards on public information boards. Sometimes this comprised the score alone (or another quantitative FFT output where the formal score was disliked). Some Trusts showed responses from the follow-up question alongside this.


13-089420-01 | FINAL v1 | PUBLIC

Figure 7.2 ---- Poster advertising FFT results

As the two examples below show, the FFT data is often communicated alongside other key metrics. This further highlights the way in which the data was almost always used in conjunction with other data – triangulated – to gain a fuller picture of how a ward or a Trust was performing. Figure 7.3 shows a ward board with the FFT score represented as a simple percentage total of those responding likely or extremely likely. Figure 7.4 shows the FFT data represented as a bar chart containing the percentage of responses for each response category – extremely unlikely to extremely likely.


13-089420-01 | FINAL v1 | PUBLIC

Figure 7.3 ---- Ward board using a combined percentage alongside other metrics

Figure 7.4 ---- Ward board bar showing the detailed FFT results

The FFT publication guidance document advises that “wherever the results are calculated and presented to the public, the methodology outlined above


13-089420-01 | FINAL v1 | PUBLIC

must be used to calculate the Friends and Family Test score”9. However, our case study visits suggested that the mandated calculation for the FFT score was not always used when communicating the results. That is, scores calculated by other methods than the one defined in the guidance were presented instead. The score calculation is shown in the figure below.


The two examples above were from Trusts which adopted their own approach to displaying the FFT results. Other Trusts used standardised displays of FFT results produced by suppliers, which sometimes also adopted an alternative score to the FFT score.

As explained in chapter 5, the dislike of the FFT score explained why many of the Trusts used an alternative method for presenting the FFT results. That said, not all Trusts were using alternatives to the FFT score to present their results. There were also examples of Trusts following the publication guidance to the letter, as figure 7.5 below makes clear.

9 Page 4: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/214941/Friends-and-Family-Test-Publication-Guidance-v2-FOR-PUBLIC_E2_80_A6.pdf


48 – (6+10+16)100 - 2

Response SCORE


16

B Likely 18

C Neither / nor 6

32D Unlikely 10


F Don’t know 2

FFT score: = 16

The FFT score was not always used when communicating the results.


13-089420-01 | FINAL v1 | PUBLIC

Figure 7.6 ---- Presentation of the FFT score in line with the guidance

It seems that this variability in the way results were communicated was a function of the autonomy afforded to ward managers within Trusts as well as the broad antipathy towards the FFT score across Trusts and suppliers. In addition, there existed a distinct preference for doing as much as possible in-house. Trusts often used suppliers to collect their FFT data but communicated the results themselves to patients and the public – almost all (97%) of those that completed the online feedback form said that they did not use suppliers for communicating with the public (though some did use suppliers to produce the displays used in wards).

7.1.4 Presenting the feedback from the follow-up question

Among those Trusts that were most comfortable with the FFT – and with using patient experience feedback more broadly – there was a particular focus on communicating the follow-up question, as well as the score, to patients and the public. As discussed elsewhere in the report, the qualitative follow-up feedback was often considered the richest and most powerful component of the FFT, both from the case studies and the online feedback. Communicating the qualitative data back to patients was felt to be an important element of being transparent and assuring patients that their voice was being listened to.


13-089420-01 | FINAL v1 | PUBLIC

Trusts communicating other patient experience measures usually went a step further and merged them with the qualitative element of the FFT, as well as the score (or other representations). This allowed a comprehensive presentation of patient feedback matched with the actions that staff had taken as a result, as figure 7.6 illustrates.

Figure 7.7 ---- ‘‘You said, we did’’ display

7.2 What do patients and the public want?

Given the very low awareness of the FFT among the public, even among recent hospital patients, once the FFT had been explained there was a keen desire that more be done to publicise it. Some of the Trusts providing feedback through the online form were also keen for the FFT to be promoted at a national level, and felt that the results nationally were currently too difficult for patients and the public to access. While people themselves freely admitted they might not use it themselves, they felt it was important to publicise FFT to those who might choose to do so. Patients and the public also demanded transparency about what happens to the feedback they provide. When asked to complete a survey, patients want to


13-089420-01 | FINAL v1 | PUBLIC

know what happens to the feedback they provide. They want to know that they are not wasting their time completing the FFT and that their voices will be heard, not just by staff but by fellow patients.

It could go in a waste paper basket…this information isn't just from us to them, it's from us to each other. We ought to be able to read what others are saying. Patient discussion group

A range of different formats for communicating the results of the FFT were explored during the patient and public discussion groups. It became clear that there was a preference for communications that are intuitive, easy to understand and contain supporting information, such as benchmarking and qualitative feedback. This allowed people to form their own judgment on the data, based on their own preferences for interpreting the data.

7.2.1 Intuitive and easy to understand

Of the range of results presentations discussed with patients and the public, the star rating system garnered the most enthusiastic responses. This sort of display was familiar from websites such as Trip Advisor and Amazon. This system therefore met the key criteria of being easily understood and intuitive.

I like this, it’s clear and straightforward. If you’re watching a film you look at the rating, it’s what you’re used to. Patient discussion group

However, the star rating did not meet with universal approval. There were also concerns about institutions such as hospitals adopting an approach like this, with participants suggesting that the star system does not convey the seriousness of the situation that patients find themselves in. From this perspective, adopting a rating system – while fine for rating a hotel – is inappropriate for use in a healthcare setting. Those who raised concerns about this approach tended to disapprove of a single FFT score, demanding more detail and context, not just on the FFT results but also other metrics and benchmarks.

I like the combination of your hospital vs. national average; if that were incorporated here, that works. Better than the star rating ---- that’s a bit too ‘Mickey Mouse’. Patient discussion group

At the other end of the scale from the star rating system was the traffic light display (see appendix for further detail on the displays shown in the discussion groups). This was especially disliked for sending the wrong message about care – patients and the public felt it implied that they would be rushed in and out of hospital rather than treated with care and attention.

Patients and the public wanted FFT data to be displayed in a way that was simple and easy to understand.


13-089420-01 | FINAL v1 | PUBLIC

The idea of a red light for some hospitals was felt to be ‘scary’ for patients – there were concerns about how knowing they were going to a ‘red light’ hospital would make patients feel. There was also a pronounced scepticism about whether simple displays of ‘negative’ results would be actually be published, something that the traffic light system shared with the green tick/red exclamation mark presentation (the way in which the FFT was originally presented on the NHS Choices website).

It was also felt that these presentations were overly simplistic and patronising. Furthermore, as tested in the discussion groups, these displays did not contain the contextualising numerical data that participants felt was important to them.

I don’t like that ---- it’s patronising. I look at that and think it’s a race to get you in and out. Patient discussion group

7.2.2 The need for supporting information

When presented with different scoring options, participants became sceptical about manipulation of the data. The different scoring systems presented to patients and the public resulted in differing scores that altered how each hospital appeared to be performing. A hospital that seemed to have a high score on one calculation had a lower score on another calculation. This meant that patients and the public often did not trust the creation of a single score (please see chapter 5 for more on how patients and the public reacted to the FFT scoring system).

When you see them all, I thought this was a better hospital than the other one. Yet it came out with a lower rating. Patients discussion group

It became clear that people employ different strategies for deciding whether a set of responses to the FFT question would be good or bad. Some wanted to see just a single score (typically contextualised with numbers answering the FFT and comparisons to other hospitals), preferring to keep things as simple as possible. For others, though there was a need to weigh positive against negative scores. There was a general desire to see the qualitative comments. Finally, there were some who wanted the broader contextual information provided by a battery of other useful metrics, from mortality and infection rates to staffing levels and waiting times. Patients and the public also possessed an intuitive understanding of how the sample size can support (or undermine) confidence in the data. Too few responses made them question the validity of a metric and so they insisted that this supporting information be made available to inform their decision making.

As well as being given an FFT score, patients and the public also wanted the results for each response, the number of responses it was based on, and access to the free text responses, ideally among unrelated metrics such as infection rates.


13-089420-01 | FINAL v1 | PUBLIC

Just give us the raw information and we'll use our brains to work it out. Patient discussion group

As a result, the NHS Choices mock-up of how data might be presented in reality met with firm and widespread approval. Participants engaged immediately with this presentation, asking questions about certain metrics and discussing among themselves what it might be telling them. They liked being able to choose from a range of measures. This would, they felt, provide a rounded view of how a hospital is performing, rather than relying solely on the public view offered by the FFT. In this way, their response mirrored the triangulation approach adopted by hospital staff when they considered the FFT alongside other targets and metrics.

If you’ve got more time to research you can delve into it… it’s all there. General public discussion group

This ability to drill down into the data, to click and look at more detailed results behind a particular metric, was especially appealing. Participants believed this would allow people to adopt their own preferred strategy for assessing a hospital.

I love it, there’s something for everybody. You could look at the stars, there are numbers…lots to look at. Whatever you are interested in you could use. Patient discussion group

Some Trusts responding through the online feedback form also supported this approach of providing additional information to patients and the public, to enable them to use the data.

While the default suggestion for the data presentation was online, participants did flag up what they felt to be obvious drawbacks – the elderly or others who were perceived as less likely to be able to access information online. Thus, participants suggested that scores be published in GP surgeries, hospitals and newspapers.

Publicise it more…we should get more information about our hospitals. General public discussion group


13-089420-01 | FINAL v1 | PUBLIC

8 Using the data


13-089420-01 | FINAL v1 | PUBLIC

8 Using the data This chapter will cover the key ways in which the Friends and Family Test (FFT) data has been used within Trusts – for example, in feeding back to staff, improving services, comparing performance against other Trusts and informing patient choice.

8.1 Who uses the data?

The FFT data is primarily used at two levels – ward and board. Board-level staff such as CEOs and medical directors, and frontline staff (especially ward managers but also nurses more generally) used the FFT far more than anyone else. While commissioners were aware of the FFT and could talk at length about it, they did not use the data a great deal, beyond ensuring that CQUIN targets were being met. Doctors without executive roles had little contact with the FFT data.

Key Findings

Most Trusts used the Friends and Family Test (FFT) to give feedback to frontline staff, thus often boosting their morale, and to improve services. The ‘real time’ nature of the data was thought to be particularly helpful in this regard, as were the free text responses to the follow-up question. Although only a small proportion of free text responses could be used to improve services, many examples were given of occasions where changes had been made as a result of the FFT. These issues were often known to staff but patient comments provided added impetus and evidence for making changes.

FFT scores were also used by senior and board-level staff to monitor performance over time and make comparisons with other Trusts. However, doubts about data validity meant there was a reluctance to put too much faith in these.

The data were not considered by staff to be useful for facilitating patient choice. That said, patients themselves expressed an interest in having ready access to the scores to help them in their decision making.

Likewise there was also uncertainty as to how FFT data could be used to commission effectively; other metrics were generally considered more useful in this regard.

The FFT was used at ward and board level, rather than from board to ward.


13-089420-01 | FINAL v1 | PUBLIC

As Figure 8.1 shows, the online feedback suggested that Trust boards were the most common users (92% used it at least a fair amount), followed by ward managers and nursing staff (84% and 80% respectively).

Figure 8.1 ---- Audiences using the FFT data

Among healthcare professionals, doctors and consultants were the least frequent users of FFT data; only one in five said they used it a great deal or a fair amount (19%). They had little interest in the FFT data; they were rarely involved in the implementation of the FFT and so did not have a stake in it in the same way that nursing staff did. That said, others suggested that FFT data was not circulated as effectively as it could have been and it was this lack of access that prevented them from making use of it.

I haven’t seen the results […] the data isn’t circulated. I would like to see it ---- it would add another layer to the feedback that we already have. Consultant

Furthermore, when prompted, there was recognition that this sort of direct patient feedback could be increasingly important to doctors, particularly during the revalidation process.

8.2 Service improvement, not comparison

There was a broad consensus that the value of the FFT data lay in service improvement and improving patient experience. It was primarily being used to help staff identify areas of improvement based on patient feedback and to communicate to staff about their performance. There was little evidence that it has been used by patients to help them choose where to go for treatment.

Nine in ten Trusts responding to the online feedback form said that the FFT was being used as a tool to give feedback to staff about their department or ward (91%). Slightly fewer (85%) said that it was being used to improve

Source: Ipsos MORI

57%

49%

38%

38%

25%

7%

4%

3%

2%

35%

35%

42%

36%

48%

24%

15%

12%

18%

7%

13%

16%

6%

19%

40%

52%

25%

22%

1%

1%

1%

14%

12%

8%

3%

1%

2%

3%

20%

6%

15%

18%

52%

40% 15%

The Trust Board

Ward managers

Nursing staff

Commissioners

Departmental heads

Allied Health Professionals

Doctors/Consultants

Patients

Other staff

A great deal A fair amount Just a little Not at all Don't know Not applicable

Base: All Trusts that responded (95). Fieldwork dates: 12th December 2013 - 2nd February 2014.

To what extent, if at all, are the following groups in your Trust using the Friends and Family Test data?

92%

of Trust Boards use the FFT data at least a fair amount

Trusts Boards make most use of the FFT data.


13-089420-01 | FINAL v1 | PUBLIC

patient experience within their Trust, and seven in ten said that it was being used to deliver service improvements (72%).

Figure 8.2 ---- Ways in which the FFT data is used

Senior staff gave examples of monthly FFT scores having been used in conjunction with other data to identify individual wards requiring improvement due to scores declining over several months. They had more confidence in using the data in this way; by only using their own FFT data they felt they were comparing like with like. The consistency of the methodology used offered them greater reassurance about the reliability of the data.

If I'm frank, the benchmark itself is interesting but maybe of secondary importance contrasted with how ours is changing, or not, over time. Because at least you're sure that you've got a consistent way of measuring […] if the score changes over time, can anyone explain why? Chief Executive

The online feedback suggested that around three in four Trusts had used the FFT data to compare themselves with other Trusts (76%). While senior staff acknowledged during the case study visits that they had used the FFT to compare the performance of their Trust with others, they also expressed doubts about whether they were right to do so. These staff, as well as some Trusts completing the online feedback form, felt that differences in data collection and variations in response rate meant that the FFT scores were not truly comparable.

It has to be consistent ---- you have to specify what Trusts do if you’re going to compare hospitals to one another. Finance Director

56%

34%

31%

19%

7%

2%

35%

52%

45%

53%

33%

14%

7%

1%

6%

12%

16%

14%

4%

8%

5%

18%

22%

1%

2%

3%

4%

5%

8%

1%

7%

5%

7%

17%

19%

1%

4%

21%

It is being used as a tool to givefeedback to staff about their

department or ward

It is being used to improve patientexperience

It is being used to compareourselves against other trusts

It is being used to deliver serviceimprovements

It is helping to inform patients aboutthe way services are delivered

It is helping patients to choose whereto receive their treatment and care

Strongly agree Tend to agree Neither / nor Tend to disagree Strongly disagree Too early to say Don't know

Base: All Trusts that responded (95). Fieldwork dates: 12th December 2013 - 2nd February 2014. Source: Ipsos MORI

Thinking now about how the FFT has been used in your Trust, to what extent do you agree or disagree with each of the following statements?

The FFT was particularly useful for feeding back to staff, improving patient experience and service improvement.


13-089420-01 | FINAL v1 | PUBLIC

Some Trusts completing the online feedback form thought that the publication of results nationally encouraged comparisons, and were concerned about the impact of publishing results that were based on small numbers or low response rates.

For specific wards where promoters are low and the number of passive responses are high can make national publication of results without the means to explain the results demoralising. NHS Trust

Similarly, stakeholders tended to focus on the methodological barriers to comparing data across different Trusts. This led them to suggest that FFT data should not be used to compare the performance of different Trusts against each other. To guard against this perceived inappropriate use of the data, they urged what they believed to be the real strengths of the FFT be more clearly defined and articulated to Trusts.

NHS England should not get hung up on comparability ---- that is not the main strength of the FFT, which is more about the qualitative, conversation aspect: it is a line of sight method, an indication of how things look. Stakeholder

8.2.1 Comparability is not just about methodology

However, the issue of comparability was about more than just data collection methodologies. As senior staff and stakeholders pointed out, even supposed ‘peer group’ Trusts have to deal with different challenges, different local populations and have different specialities. Consequently, many questioned whether any Trust was truly comparable to another.

Even if the methodological challenges are looked at there would still be further issues with comparability ---- there are big differences between Trusts. Stakeholder

This issue was also raised by commissioners. They felt that comparisons between Trusts were rarely useful, being more interested in internally-focused time series data to help assess performance over a period of months or years. Commissioners also pointed out that different hospitals will have different relationships with their patients which will be reflected in how those patients respond to the FFT.

Comparison is not useful ---- there are differential ways of interpreting the FFT in a local general hospital compared to in a specialist cancer care centre. Each has a different patient relationship. Commissioner

Th

13-0

Thpawealsanchsucinf

Asfeechpepa

8.3

Paof othdo

Ameleof pri

I lihaGe

Whpumorefto weThtheUK

Hotheun

10 h

he Friends and F

089420-01 | FINAL

8.3 The

ere were dotient choice.

ere unsure ofso had mixedd nurses all oice – they sch as in A&Eluential on p

can be seeedback agreoose where r cent strongtients were u

3.1 Patients

rticipants in views on pathers were no so effective

mong those wement of concare they recor to a visit.

ike to haveaving optioeneral public

hen making ablic said the

outh or onlinefer back to pwork with the

ere trusted tois reflected b

eir long-standK10.

owever, not eemselves abenthusiastic

http://www.ipsos

Family Test: Qua

v1 | PUBLIC

e FFT and

ubts about h. Patients thf how importd views on thexpressed s

suggested thE, while also atients than

n in figure 8.eed that the F

to receive thgly agreed. Ausing FFT da

s and publi

the patient atient choice

ot enthusiastiely.

who were enntrol over theceived by re

e a sense oons: it helpsc discussion

a choice aboey would relye – provided

personal expee advice givo tell them whboth the expding position

everyone waout the treatquestioned

s-mori.com/Ass

alitative research

d patient

how much themselves weant it would he notion of cscepticism ahat choice wbelieving thathe FFT data

.2, only one FFT data washeir treatmenA similarly smata a great d

c; mixed vie

and public d– some wereic and felt th

thusiastic abir treatment.

esearching h

of having sos you to feegroup

out where to y on recomm by other receriences of ien to them bhere they sh

pertise and kn as one of th

s keen on thment they rewhether the

ets/Docs/Polls/F

h

choice

e FFT is beinere largely ube in their dechoice itself.bout how muas not workaat other factoa.

in six Trustss being usednt and care (mall proportioeal or a fair a

ews on cho

iscussion gre keen on beat they would

bout choice, It allowed tospitals, con

ome choiceel like you h

go for treatmendations –cent patientsndividual ho

by their GP: hould do for tnowledge exhe most trust

e idea of maeceived. Thoy would have

Feb2013_Trust_

ng used to fanaware of thecision-mak Senior stafuch patients able in manyors would be

providing ond by patients16%); of theon of Trusts amount (15%

oice

roups expreseing able to cd lack the kn

it offered pathem to choonsultants or s

e, that perchave some

ment, patientwhether via

s. Similarly, ospitals. Othhealthcare pthe best possxpected of ated professio

aking a choicse who weree all the info

_Charts.pdf

acilitate he FFT and ing. They ff, doctors exercise

y settings, e more

nline s to help themse only two said that

%).

ssed a rangechoose, whilnowledge to

atients an ose the type surgeons

ception of e control.

ts and the word of some would ers preferredrofessionals sible care. GP but alsoons in the

ce for e rmation they

m

e e

d

o

y

Thecurusepatabhea

I’d havchoto fcon

Gendisc

e FFT did nrrently seeed to infotients’ chout their althcare.

like a senving somoice…it hfeel you hntrol. neral public cussion grou

8

not em to be rm oices

nse of e

helps you have som

up

89

e

Th

13-0

necohathe

It cdeansuPa

Foinaif tsucwhpafampave

Monoheuschpeim

Thfreunresex

- H- I- IhaGe

Thaloof thecogivsco

he Friends and F

089420-01 | FINAL

eded to enancerned aboving to make

em was a so

could haveeath rates…nxious. If yourgery that w

tient discuss

r these particappropriate. hey knew thach choices w

ho felt that mrticularly A&

mily membertients tickingry satisfied w

ore broadly, r the public talthcare. Oning it for this apter 7). Onrformance dportant to pa

at said, patiee text respoderstand whsponses couperience the

Has there b’d want to k’d want to save been theneral public

ere was a prongside the Fwhat to expee FFT with semments allo

ven and appores or rating

Family Test: Qua

v1 | PUBLIC

ble them to mout their abilie these choicurce of anxie

e a negativ…if they’re rou need hewould just sion group

cipants, the They suggeat others wewere, in pracany patients

&E services –r or friend beg ‘unlikely’ orwith their car

due to low letended to usnce it was ex purpose, esn balance, hodata (such asatients in hel

ents and thenses from th

hy a hospital uld help provey should exp

been an MRknow abousee what phere; I’d looc discussion

reference forFFT score: thect from a hoervices such wed people ly their own jgs.

alitative research

make an infoity to interpreces at what wety that they

ve effect. Mreally bad teart surgeryt make you

wording of tested that there able to ch

ctice, rare. Ts would be re– as they woue unwell. Thur ‘very unlikere.

evels of awase it to help txplained to tspecially aloowever, clinis waiting timeping them m

public werehe FFT follow

had been ravide reassurapect.

RSA outbreaut waiting tpeople say ok online agroup

r looking at the words of ‘ospital. Patieas Trip Advto get a cleajudgement o

h

ormed choiceet the data cwas often a vdid not need

Most hospitathat wouldy and the stfeel more a

he FFT quesey would onlhoose that seThis was endeluctant to ‘reuld not wish tus there wasly’ with this i

reness of thethem make dhem, there wngside othercal data (sues) were con

make a choic

e interested iw-up questionated in a parance about t

ak? times.

y about the at people’s

he response‘real people’

ents and the isor and Amar sense of won how much

e; some wercorrectly. In very stressfud.

tals are judgd make youstats are baanxious.

stion was tholy ‘recommeervice. Theydorsed by staecommend’ to recomme

s some concn mind, even

e FFT, neithedecisions abwas some apr metrics (asch as infectinsidered to be than the F

n being ablen. This wouldrticular way. he kind of ca

place, peos experienc

e to the follow would give public direcazon, where

why scores hh weight to g

re also addition,

ul time for

dged on u more ad for heart

ought to be nd’ a service

y felt that akeholders a hospital –nd that a ern about n if they were

er patients out their

ppetite for discussed ion rates) an

be more FT score.

e to view the d help them These are and

ople who ces.

w-up questioa better ideatly compared

e free text have been give certain

rt

e

e

n d

on a d

I’d basan speand

Patigrou

make mysed on loonline fo

eaking tod family. ent discussiup

9

y choice ooking at orum and

my friend

on

90

ds

Th

13-0

I’mopHopromyfamPa

8.3

BepawaawFF

ThpewoHe

Raderecthathequsuthe

If ymeGe

Stachmasuc

I’monanmiDir

he Friends and F

089420-01 | FINAL

m quite savperation onospital’ androbably 25,y informed

amily as wetient discuss

3.2 Trusts a

eyond this thotients would

as available twareness am

T data, as w

hey may findeople wouldould they kealthcare Ass

ather, staff sucisions basecommendatioat these influeir decisions estioned whpported by sey might not

you’d neveember of steneral public

aff believed toice at all, F

aking procesch as mortal

m not sure hn personal end relativesinority amorector

Family Test: Qua

v1 | PUBLIC

vvy on comn my knee Id then wha, 30, people

d choice onell. sion group

nd staff do

ough, there w be unlikely tto them. It wong patients

well as other m

nd the FFT scld look for itknow it wassistant

uggested thaed on the adons of close ences are w on. With littether or not

some patientimmediately

er been befstaff what it c discussion

that, for thosFT data wou

ss alongside ity rates and

how patienexperiences. Yes someong the info

alitative research

mputers […]I’d type in G

at would prole’s experien that and p

ubted patie

was a perceto use FFT d

was assumeds and the pumetrics that

score helpfuit, or woulds there?

at patients wvice of GPs,friends and

what patients tle knowledgpatients wouts and the puy trust a new

fore, you mt was. group

se few patienuld be one sm

the traditiond infection ra

nts make che. You ask ye people wormed midd

h

] so say, foGoogle, ‘Knrobably popence of thatprobably (m

ents were u

ption amongdata to informd that there wblic about wmight help th

ul but I’m nd know how

ould be far m personal exrelatives. Tand the pube of or expould trust it. Tublic themseand unprove

might distrus

nts who weremall componnal influencestes.

choices ---- I wyour GP, yo

will look thinddle classes

or example,Knee operatp up is a fo

at hospital. my own) fri

using FFT

g Trusts and m choice evewere low leve

where and hohem make c

not sure howw to look fo

more likely toxperience anhere was anblic have alwosure to the FThis perspecelves, who suen source of

st it. I’d as

e aware that nent in the des and other i

would say you ask youngs up, buts.

e, I need antions X

orum, with I’d make iends and

staff that en when it els of w to access hoices.

w many or it. How

o make nd the direct assumption

ways based FFT, staff ctive is uggested f information

sk a

they had a ecision-nformation

it's mostly ur friends t it’s a smal

n

.

all

I’mpeoit…kno

Hea

Forisn’proanythewasig

Direand

m not sure ople wou

…how wouow it was

althcare assis

r our patie’t really a

ovider. Thywhere e

em to go uant to travnificant d

ector of Nursd Quality

9

how manuld look fould they

there? stant

ents therea choice o

ere isn’t lse for unless the

vel a distance.

ing

91

ny or

e of

ey

Th

13-0

8.3

Staof req

Thsetem

Dischpathesptrewefurlacevvie

FoansigwatheDir

Cothebe

AsweCoresTrusaw Coas triapecoco

he Friends and F

089420-01 | FINAL

3.3 Does ‘re

aff, patients achoice. Choquired, and l

ere were patting, where

mergency wo

stance to traoice. This wtients were a

ere because ecialist Trustatment for c

eek. As suchrther to be trecked the tranident outside

ew also expre

or our patienywhere elsgnificant dias having a

heir feedbacrector of Nur

8.4 Use

ommissionerse response rayond that, d

can be seeere being useommissionerssponse ratesusts. They ww this thems

ommissionersa standalon

angulated wirformance. mmissionersmplaints, for

Family Test: Qua

v1 | PUBLIC

eal choice’

and memberoice was felt ack of willing

rticular concstaff and pa

ould be impra

vel, particulawas typically acute non-elit was closets regardingancer patienh, it would noeated in a hinsport or finae urban areaessed amon

ents there isse for them

distance; thea knee op Ick, but it’s rsing and Qu

e of the F

s were knowates of their escribing it a

n in Figure 8ed a great ds described s and scoreswere aware oselves.

s were uncee measure. th other metThe FFT was

s looked at –r example. A

alitative research

exist?

rs of the pubto be constr

gness to go

cerns about htients alike feactical and u

arly when oumentioned bective patien

est. Similar v patients wit

nts may requot be feasiblegher-perform

ancial meansas where theg staff in spe

sn’t really am to go unlehey are real

I might loodifferent fo

uality

FFT data i

wledgeable aproviders. Has one perfo

8.1, almost theal or a fair ahow they typ

s in monthly oof the qualita

rtain about hThey stated

trics in order s rarely the o

– they also loAs a result, c

h

blic alike all qrained by sebeyond one

how choice celt that choounlikely.

tside Londoby staff at Trunts; these paviews were exh certain con

uire several ve for them toming hospitas to do so. Thre was less lecialist ward

a choice of ess they wally tied into

ok at differeor a cancer

n commi

bout the FFTHowever, theormance met

hree in four Tamount by cpically receivor quarterly qtive follow-up

how the FFT that the scoto provide a

only patient eoked at annu

commissione

questioned thtting, type of’s local hosp

could functioosing where t

n, was also fusts where thatients were sxpressed bynditions. To visits to a hoso travel a greal – especiallyhis was partiocal choice

ds or treatme

f provider. Tant to traveo their localent hospitalr centre.

ssioning

T and lookedey used the dtric among m

Trusts said thcommissioneved details oquality reporp data but ra

scores couldore would nea reliable picexperience mual reports a

ers focused o

he relevancef treatment

pital.

on in an A&Eto go in an

felt to limit he majority osimply takeny staff at

illustrate, spital in one

eat deal y if they cularly available, a

ent centres.

There isn’t el a

al centre. If als and get

d closely at data for little

many.

hat FFT data rs (74%).

of FFT rts from arely, if ever,

d be useful eed to be cture of Trustmeasure thatand on the

e

of n

I

t

Cothewitcurres

Thebe comonlconme

CCG

ommissione FFT in coth other drrently foc

sponse ra

e FFT alonused in

mmissionly use it inntext of o

easures. G Represent

9

ners used onjunctionata, and

cus on thete CQUIN

ne wouldn

ing. We’dn the other

tative

92

n

e N.

n’t

d

Th

13-0

resCQ

It iThit iCC

Seexbetha

Coin nomeDir

In of areproquitse

IncreaCo

Almqu

ThhaThalo

Hobeexa

8.5

Staonmo

he Friends and F

089420-01 | FINAL

sponse rate QUIN.

is used in the FFT alonein the conteCG Represen

nior staff beperience datlieved that c

an other qua

ommissionecomplianc

ot scores. I easure. rector of Nur

contrast, thethe FFT datae gathering aoviders shouestion to ideelf be incent

centive payeasons for nommissioner

8.5 Usi

most all suggestion was t

e value in thd been giveerefore, it pr

one. It also a

owever, many done to getample, a mo

5.1 Boosting

aff at all leveline form, me

orale and pri

Family Test: Qua

v1 | PUBLIC

information a

the CQUIN e wouldn’t text of othentative

lieved that cta, nor did th

commissionelity indicator

ers have noce for CQUIdon’t think

rsing

ere was a dea, in order to and responduld be using entify and adtivised along

ayments shonon-recomm

ing data

gested that the most use

e follow-up qn and immedrovided a muacted as a va

y staff, staket the most ouore detailed t

g morale a

ls in the casentioned thede among fr

alitative research

and whether

---- responsebe used in

er measures

ommissionehey place a hers viewed thrs such as m

ot asked foUIN ---- they ak they see it

sire among cbuild a bette

ding to patienthe qualitativdress patienside the resp

ould be aromendation

from the

the feedbackful output fro

question wasdiately identuch more nuaaluable mora

eholders andut of this invathematic ana

nd pride

e studies, as use of posit

rontline staff,

h

this met the

e rates are fn commissios.

rs had little ihigh value on

he FFT as beortality and i

or the data.are interesteit as a parti

commissioneer understannt feedback. ve feedbacknt concerns, ponse rate.

ound the aband tackle

follow-up

k collected from the FFT.

s that it shedified potentiaanced picturale booster fo

d commissionaluable data alysis of the r

s well as Trustive follow-up inspire hea

e requiremen

financially oning. We’d

nterest in pan it. As a resing much lesinfection rate

. They are ited in respoticularly go

ers to make nding of how They talked

k from the FFsomething t

ability to idee these issu

p questio

rom the follow

d light on whyal service imre than the For frontline s

ners felt that source throuresponses p

sts feeding bp comments lthy competi

nts of the

y rewarded.’d only use

atient sult, it was ss important es.

interested onse rates, ood

greater use w providers d about how T follow-up hat could

entify the ues.

n

w-up

y low scoresprovements

FFT score didstaff.

more could ugh, for rovided.

back via the to boost tion between

d.

. d

n

Incshoabrearectac

Com

centive paould be aility to ide

asons for ncommendckle these

mmissioner

9

ayments around theentify the non-

dation ane issues.

93

e

d

Th

13-0

waStapaNucoanfee

It’sneapNu

Suapinfaccares

ThbethesaanNu

It tFFtheimmenustato

8.5

Neidefindtimexresof folho

he Friends and F

089420-01 | FINAL

ards and indiaff, especiallrticularly citi

urses said thamments for td Clinical an

edback could

’s nice to seegative stufppreciate uurse

ch commentpreciative paluence uponted as a strore, as well assponse rates

he nice thingeen very kin

he perceptioay the FFT hnd caring. urse

herefore folloT were share

e FFT tendedproving. Somembers of stamber of the

aff by email, identify and

5.2 Service

egative commentify where d out why pa

me’ where poperience leasponding to responses wlow-up commw the data c

Family Test: Qua

v1 | PUBLIC

ividuals and y nurses, peng negative at they werethis reason, nd Nursing led be to their

ee that we’ff in the meus.

ts, especiallyatients, were

n their mood ong motivatos encouragin

s.

ng is when ynd, and coon is that n

has reassure

owed that in ed more widd to be higheme examplesaff (usually p‘best’ commwhile other Tpraise staff

improveme

ments were aservice impratients gave

ossible, was hads, ward mathe online fe

was sometimments could can be used.

alitative research

generate a serceived thatmedia cove greatly appand several eads also emteams.

’re doing goedia, it’s go

y those in whe widely recoand the pridr for staff to ng their ‘buy

you hear thouldn’t do enurses arenred me that

those Trustsely and in an

er and respos of good prapatient exper

ments from eaTrusts used anamed in po

ent

also welcomrovement necertain resphighly valuedanagers andeedback formes higher fobe less deta

h

stronger focut positive feerage as havreciative of pward manag

mphasised ho

good things ood to see t

hich individuognised to hade they took maintain hig

y-in’ to the FF

hem say thaenough for n’t caring ant patients d

s where free n effective mnse rates beactice in thisrience leads)ach ward anda ‘wall of famositive comm

ed, as they ceded to be f

ponses, and ad by staff – in senior nursi

m commenter A&E than foailed for A&E

us on patienedback was ing an impacpatients’ posgers, Nursingow valuable

s ---- you see that some

ual nurses weave a strongin their work

gh standardsFT – thus imp

hat the nursethem. In th

anymore, budo think we

text commemanner, enthuetter – or at les regard inclu) identifying d passing th

me’ on their sments.

could be usefocused. Thact on issuen particular ing staff. Sod that, whileor an individE, having imp

nt experiencerare, ct on moralesitive g Directors positive

e so much people

ere named b positive . In turn, this of patient

proving

ses have he press, ut I would

e’re kind

ents from the usiasm for east uded a small

hem on to all staff intranet

ed directly toe ability to s in ‘real patient

ome Trusts the volume ual ward, the

plications for

e.

.

by

o

e r

It’s wethinmutheseeap

Nurs

Exathain refeehig

nice to se’re doingngs ---- youuch negate media, ie that sompreciate

rse

amples ofat had beeresponse edback weghlighted.

9

ee that good

u see so tive stuff iit’s good tme peopleus.

f changesen made to FFT ere

94

n to e

s


13-089420-01 | FINAL v1 | PUBLIC

Staff and some of those providing feedback through the online form acknowledged that the majority of patient comments were very generic, simply stating for example that ‘there were no problems’, ‘everything was ok’ or ‘I would not want to come here again’. However, some – in particular those most closely involved in the FFT – mentioned useful ‘nuggets’ of detail emerging in some comments, which could be used to identify specific problems and make timely service improvements. A selection of examples given were:

New heaters being installed in an A&E waiting area after a number of patients said that it was cold.

The introduction of quieter equipment, earplugs and ‘silent bins’ on wards where a lot of patients reported disturbance of their sleep by excessive noise levels during the night.

The introduction of a ‘Care Companions’ project through which volunteers provide a befriending service during visiting hours to patients who have no relatives.

The introduction of a Discharge Co-ordinator role – one nurse taking responsibility for the discharge of all patients on a certain ward in any given day, to ensure the process runs as smoothly as possible.

Addressing and improving the attitudes of receptionists in a particular ward or department.

Investigating why the shower in a particular room was not running water that was hot enough.

Where comments were collected and read by staff very regularly – for example on a daily or weekly basis – issues such as a broken shower or an impolite receptionist could be identified, communicated to the appropriate staff and addressed immediately. Even where this practice was not common, staff in some Trusts said that the regularity of feedback collection allowed them to identify common topics emerging among comments, and push for changes and improvements to be made sooner than they otherwise might have been.

If A&E get 50 free text comments in a month that say the receptionist was rude, or the chairs are disgusting, that’s really helpful for prioritising what they need to do to improve. If they put a new set of chairs in, they might see their score go up ---- those sorts of things. Director of Patient Experience

Staff tended to be aware of the issues raised through the free text question, but having the evidence provided added impetus for making improvements

Comments allowed identification of common topics so staff could push for improvements sooner than they might have otherwise.


13-089420-01 | FINAL v1 | PUBLIC

and made it a more immediate problem to solve. In addition, there were cases of staff using free text responses to provide evidence highlighting issues that they themselves had identified, but needed more ammunition to persuade senior colleagues to act.

8.5.3 A need for more thematic analysis

That said, staff explained that very little formal or systematic analysis was being carried out on the follow-up question data. Patient experience leads in particular felt that this would be beneficial in helping to give a clearer picture of what matters most to patients and ensuring that important issues are not overlooked. However, lack of time, skills and resources prevented the possibility of rigorously analysing follow-up responses in the majority of Trusts.

We are focused on numbers not quality; we’re worried about the benchmark for the CQUIN payment…We don’t report the number of free text comments and what percentage were positive or negative. It would be a laborious process and cost money. Finance Director

Stakeholders cautioned that, at present, there is inconsistency between Trusts over the extent to which the FFT is being used to improve patient experience – a view supported by our case study visits. They suggested that training staff on how to get the most from the free text comments may help in this regard.

A lot more can be done in this area. In particular, there needs to be more coaching around FFT ---- helping people learn from the suggestions and improve their work. I believe that this isn’t really being done in a systematic way. Stakeholder

Therefore, while the free text responses were widely seen to be the most useful element of FFT, there was a sense that, as of yet, the data is not being used to its full potential.

There was some tentative interest in thematic analysis of the free text questions although resource and capability were a concern.


13-089420-01 | FINAL v1 | PUBLIC

9 Conclusions


13-089420-01 | FINAL v1 | PUBLIC

9 Conclusions The key themes emerging from the research were:

There was real enthusiasm for the collection of patient experience data.

The FFT was felt to offer the best yet approximation of real time patient feedback.

Triangulation of the FFT data with other measures is necessary to achieve an understanding of the whole picture.

There were differing opinions about the appropriateness of the question wording.

The free text follow-up question was believed to be the most valuable aspect of the FFT.

The calculation of the FFT score was disliked and confusing.

The FFT was primarily being used by Trusts as an internal tool to drive service improvement.

The FFT data was not perceived to be strictly comparable between Trusts.

Clinical staff had less awareness and exposure to the FFT, compared to other staff groups.

A&E posed a challenge to the FFT.

There was little consistency between Trusts in administering the FFT.

Exactly how the FFT was administered to patients was inconsistent within Trusts. There were issues with the practical definition of the eligible population.

Focusing on the response rate was understandable but could lead to perverse outcomes.

Commissioners paid little heed to the FFT beyond the response rate CQUIN.

Presentation and communication of FFT data was inconsistent.


13-089420-01 | FINAL v1 | PUBLIC

In its current form, the FFT’s relevance to patient choice was questioned. It would need a clearer metric and increased public awareness to be able to inform patients’ choices.

There were questions about the sustainability of the FFT.


13-089420-01 | FINAL v1 | PUBLIC

10 Appendices


13-089420-01 | FINAL v1 | PUBLIC

10 Appendices 10.1 Online feedback form

Information for those who are responsible for collecting and collating feedback from colleagues and submitting the response on behalf of the Trust:

Thank you for taking the time to provide your feedback as part of

the Friends and Family Test (FFT) Review. It is extremely important

for NHS England to hear the experiences of those working with FFT

so that areas for improvement can be identified.

NHS England would like each Trust to submit one response. To

help facilitate this we have provided you with a Word version of the

feedback form, which can be downloaded here.

Please share the Word document with colleagues, and ask them for

their input. Once you’ve collated all comments in this way we would

ask you to complete this online questionnaire, providing all relevant

feedback from the Trust in one response by 31st January 2014.

Each link to the survey is unique and once a response has been

submitted the survey link will close, so please make sure the

response submitted is on behalf of the whole Trust.

NHS England have commissioned the independent research

organisation Ipsos MORI to conduct the survey. All responses are

confidential and no individual Trust will be identifiable in the results.

If you have any questions about the research then please contact

David Jeans at Ipsos MORI on 020 7347 3349 or at

[email protected].

The survey contains a mix of closed questions (which ask you to

select an answer or answers from a list) and open free-text

questions that allow you to give a more detailed response.

Please note that there is a limit on the number of characters that

can be entered for all the open free-text questions. For all open

free-text questions except Q8b there is a limit of 4,000 characters.

For Q8b there is a limit of 1,000 characters.


13-089420-01 | FINAL v1 | PUBLIC

Section 1: Information about the teams / individuals involved in FFT ASK ALL Q1. Please tell us where responsibility lies for the Friends and Family Test within your Trust. In your response please name all relevant teams and the job roles of any individuals with responsibility for the Test. OPEN QUESTION ASK ALL Q2. In which of the following settings does your Trust currently collect Friends and Family Test feedback: SINGLE CODE FOR EACH STATEMENT

a) A&E b) Inpatient settings c) Maternity d) Other (please specify)

Yes No Don’t know SHOW TEXT FOR ANY RESPONDENTS WHO SELECT CODE 1 FOR Q2c. The main emphasis of this feedback form is to understand your experiences of the Friends and Family Test in A&E and inpatient settings. Therefore in your responses please only think about these settings unless otherwise stated. There is a question towards the end of the form to capture any initial experiences of setting up Friends and Family Test in maternity services and questions on future challenges ASK Q3a OF THOSE WHO CODE 1 AT Q2a. ASK Q3b OF THOSE WHO CODE 1 AT Q2b. Q3. Which of the following methods are used to collect the Friends and Family Test feedback in your Trust in…? a) A&E b) Inpatient settings MULTICODE OK Bedside terminal Comments cards / postcards Internet survey – via a link in an email Internet survey – open to anyone to complete Kiosk – e.g. in the ward Mobile phone app (not SMS survey) Paper questionnaire Tablets (e.g. iPad or similar) Telephone survey – i.e. speaking to an interviewer Texting (SMS) Token system Other (please specify)


13-089420-01 | FINAL v1 | PUBLIC

ASK SEPARATELY FOR A&E AND INPATIENTS SETTINGS – ROUTED FROM Q2. Q3c. Why did you choose this method/these methods for A&E? Q3d. Why did you choose this method/these methods for inpatient settings? OPEN QUESTION ASK ALL WHO CODE MORE THAN ONE OPTION AT Q3a OR Q3b. Q4. And which is the main method used to collect the Friends and Family Test feedback in your Trust in…? By main method we mean the one that delivers the greatest number of completed responses. a) A&E b) Inpatient settings SINGLE CODE FOR EACH STATEMENT PLEASE ONLY PULL THROUGH THE RESPONSES SELECTED FOR EACH STATEMENT AT Q3a FOR Q4a OR Q3b FOR Q4b. IF OTHER IS SELECTED THEN PLEASE DISPLAY THE WRITTEN OPTION AS AN ANSWER OPTION. Bedside terminal Comments cards / postcards Internet survey – via a link in an email Internet survey – open to anyone to complete Kiosk – e.g. in the ward Mobile phone app (not SMS survey) Paper questionnaire Tablets (e.g. iPad or similar) Telephone survey – i.e. speaking to an interviewer Texting (SMS) Token system Other (please specify) We don’t have a main method, they are all used equally ASK Q5a OF THOSE WHO CODE 1 AT Q2a. ASK Q5b OF THOSE WHO CODE 1 AT Q2b. Q5. In your Trust do you ask a follow-up question to the Friends and Family Test question in…? By follow-up question we mean an open-ended question for patients to provide supporting comments. SINGLE CODE FOR EACH STATEMENT a) A&E b) Inpatient settings Yes No Don’t know


13-089420-01 | FINAL v1 | PUBLIC

ASK Q5c OF THOSE WHO CODE 1 AT Q5a. ASK Q5d OF THOSE WHO CODE 1 AT Q5b. Q5c. What follow-up question do you ask in A&E? Q5d. What follow-up question do you ask in inpatient settings? OPEN QUESTION ASK ALL Q6. Do you ask any further questions at the same time as the Friends and Family Test, and if so, what are they? Please include in your response any demographic information you may collect from the respondent (e.g. gender, age, etc.) OPEN QUESTION We do not ask any further questions SINGLE CODE ONLY ASK ALL Q7. We would like to know whether you are using any external suppliers to administer or deliver any aspect of the Friends and Family Test. For each of the following tasks please tell us whether you use an external supplier or if the Trust itself undertakes the task? SINGLE CODE FOR EACH STATEMENT

a) Collecting the responses – e.g. asking the Friends and Family Test question(s)

b) Generating a dataset of the responses c) Interpretation of the data and / or developing recommendations d) Communicating the results to staff and patients

We use an external supplier The Trust undertakes the task itself Don’t know ASK IF CODE 1 AT ANY OF Q7a, Q7b, Q7c AND OR Q7d. Q7e. Thinking about the external supplier(s) who:

INCLUDE IF CODE 1 AT Q7a: collects the responses INCLUDE IF CODE 1 AT Q7b: analyses the responses and

provides a detailed breakdown INCLUDE IF CODE 1 AT Q7c: interprets the data and / or

provides recommendations INCLUDE IF CODE 1 AT Q7d: communicates the results to staff

and patients

please can you tell us which external supplier(s) you use [TEXT SUB IF CODE 1 MORE THAN ONCE AT Q7a AND/OR Q7b AND/OR Q7c AND OR Q7d: for each task]? OPEN QUESTION


13-089420-01 | FINAL v1 | PUBLIC

Section 1b: Overall view of FFT ASK ALL Q8a. Thinking generally about the Friends and Family Test, how well, if at all, is it working in your Trust? SINGLE CODE Very well Fairly well Not very well Not at all well Too early to say Don’t know ASK ALL WHO CODE 1-4 AT Q8a. Q8b. Please briefly state your reasons for your response to the previous question. Please note that there are opportunities throughout this feedback form where you will be able to provide more detailed feedback on specific elements of the Friends and Family Test. As such we are only looking for a brief explanation here. OPEN QUESTION – LIMIT CHARACTERS TO 1000 Section 2: Setting-up the Friends and Family Test ASK Q9a OF THOSE WHO CODE 1 AT Q2a. ASK Q9b OF THOSE WHO CODE 1 AT Q2b. The Friends and Family Test is being rolled out to additional settings in the future. NHS England is keen to learn from your experiences of setting up the Friends and Family Test in A&E and inpatient settings to help this roll-out. Q9. How easy or difficult has it been to set-up the Friends and Family Test in your Trust in…?

a) A&E b) Inpatient settings

SINGLE CODE FOR EACH Very easy Fairly easy Neither easy nor difficult Fairly difficult Very difficult Don’t know


13-089420-01 | FINAL v1 | PUBLIC

ASK Q10a OF THOSE WHO CODE 1 TO 5 AT Q9a. ASK Q10b OF THOSE WHO CODE 1 TO 5 AT Q9b. Q10. Please can you tell us why it has been [TEXT SUB IF CODE 1 OR 2 AT Q9a or b: easy; TEXT SUB IF CODE 3 AT Q9a or b: neither easy nor difficult; TEXT SUB IF CODE 4 OR 5 AT Q9a or b: difficult] to set-up the Friends and Family Test in your Trust in…?

a) A&E b) Inpatient settings

In your response please describe what you think went well, and what the challenges were. OPEN QUESTION ASK ALL Q11. What changes would you suggest are made to the implementation guidance for the Friends and Family Test, if any? In your response please think about any changes you would make now to the current guidance on A&E and inpatients, but also more widely about the future implementation guidance on outpatients and services in other settings. OPEN QUESTION Section 3: Using the Friends and Family Test ASK ALL The next set of questions cover how the results of the Friends and Family Test are being communicated and used. Q12. What do you think about how the Friends and Family Test results have been published nationally and what changes, if any, would you make? OPEN QUESTION


13-089420-01 | FINAL v1 | PUBLIC

ASK ALL Q13. Thinking now about how the Friends and Family Test score is calculated, to what extent do you agree or disagree that the numerical score is…

a) easy for staff to understand? b) easy for patients to understand? c) easy for the general public to understand?

SINGLE CODE FOR EACH Strongly agree Tend to agree Neither agree nor disagree Tend to disagree Strongly disagree Don’t know ASK ALL Q14. What improvements, if any, do you think could be made to how the Friends and Family Test score is calculated? OPEN QUESTION ASK ALL Q15. In what ways, if any, is your Trust communicating the results of the Friends and Family Test to…? a) Staff b) Patients c) The general public OPEN QUESTION FOR EACH STATEMENT


13-089420-01 | FINAL v1 | PUBLIC

ASK ALL Q16. To what extent, if at all, are the following groups in your Trust using the Friends and Family Test data? SINGLE CODE ONLY FOR EACH STATEMENT a) Doctors / Consultants b) Nursing staff c) Allied Health Professionals d) The Trust Board e) Departmental heads f) Ward managers g) Patients h) Commissioners i) Other staff A great deal A fair amount Just a little Not at all Don’t know Not applicable – ONLY SHOW THIS CODE FOR STATEMENT i. ASK ALL Q17. Thinking about how the Friends and Family Test has been used in your Trust, to what extent do you agree or disagree with each of the following statements?

a) It is being used to improve patient experience b) It is being used to deliver service improvements c) It is being used as a tool to give feedback to staff about their department or ward d) It is helping to inform patients about the way services are delivered e) It is helping patients to choose where to receive their treatment and care f) It is being used to compare ourselves against other Trusts

SINGLE CODE FOR EACH STATEMENT Strongly agree Tend to agree Neither agree nor disagree Tend to disagree Strongly disagree Too early to say Don’t know ASK ALL Q18. Please tell us more about how the Friends and Family Test is used within your Trust, giving particular examples of the benefits, any drawbacks and any examples where the Friends and Family Test has made a difference to patients and/or staff. OPEN QUESTION


13-089420-01 | FINAL v1 | PUBLIC

ASK ALL WHO CODE 1 AT Q2a AND Q2b. Q19. What are the main differences, if any, in how the results of the Friends and Family Test are used in A&E and inpatient settings? OPEN QUESTION ASK ALL Q20a. What difference, if any, has the Friends and Family Test made to the emphasis on patient experience in your Trust? SINGLE CODE ONLY The emphasis on patient experience has increased a lot The emphasis on patient experience has increased a little The emphasis on patient experience has stayed about the same The emphasis on patient experience has decreased a little The emphasis on patient experience has decreased a lot Too early to say Don’t know ASK ALL CODES 1-5 AT Q20a. Q20b. Why do you say that [TEXT SUB FROM Q20a]? OPEN QUESTION Section 3b: Maternity FFT ASK ALL WHO CODE 1 AT Q2c. Whilst the main emphasis in this feedback form is on understanding your experiences of the Friends and Family Test in A&E and inpatient settings we are also keen to understand how you have found setting up the Friends and Family Test in maternity wards. Q21. What have been your initial experiences in setting up the Friends and Family Test for maternity services in your Trust? In your response please mention any challenges or benefits you’ve encountered so far. OPEN QUESTION


13-089420-01 | FINAL v1 | PUBLIC

Section 4: Summarising your experiences of the Friends and Family Test ASK ALL Q22. Thinking generally now about the next 12 months, what do you think will be the biggest challenges for the Friends and Family Test, and why? OPEN QUESTION ASK ALL Q23. What should be done to overcome these challenges? OPEN QUESTION ASK ALL Q24. Is there anything else you would like to add about the Friends and Family Test? OPEN QUESTION

THANK AND CLOSE


13-089420-01 | FINAL v1 | PUBLIC

10.2 Guidance for case study Trusts

Thank you for agreeing to participate in the review of the Friends and Family Test (FFT). We are inviting all Trusts in England to provide feedback and share their experiences, and we also wish to speak to a smaller group of Trusts in more detail.

We have selected a small sample of 10 Trusts from across the country, varying in size and in the way they collect FFT. The aim is to explore in greater detail how and where the FFT is used within each of these Trusts. Understanding how the FFT works in more detail is essential to the review and to inform any improvement areas in the future.

This document is intended to provide guidance on how the case studies will work and what we will be asking you to do. If you have any questions please contact Rachel Burkitt at Ipsos MORI on 0207 347 3487 or at [email protected]. If you wish to speak to someone at NHS England, please contact Laurence Bruce on 020 7972 5097 or at [email protected].

What is a case study?

Two researchers from Ipsos MORI would spend two to three days within the Trust during January 2014 conducting interviews and discussion groups with key members of staff and the public, seeking feedback on their experiences of the FFT, and learning how you implement and use the results from FFT.

With your permission, the Ipsos MORI researchers would also conduct on-site observation of how the Test is conducted in your Trust, including collecting examples of FFT materials from your Trust, where appropriate.

All responses are confidential and individual responses will not be shared directly with NHS England.

How can you help?

We will, of course, schedule the visit for a time that is convenient for you. At the same time, we would like to discuss the best approach to making appointments to interview members of staff. We will also be asking for your help to recruit a small number of recent patients to take part in a discussion group about the FFT. More details on what this would involve are given below. The groups would take place at a separate venue, and booking the venues will be arranged by Ipsos MORI.

Who would we like to speak to?

In order to get a complete picture of how the FFT is used and perceived within your Trust we would like to speak to as many people as possible.

The table below outlines who we would like to speak to, as well as the approach we would use for these discussions. Whilst we understand that not all of the following job titles may be relevant to your Trust we hope that similar roles may exist. We would like your help in identifying people who fulfil these roles (or similar), and in facilitating the interviews.


13-089420-01 | FINAL v1 | PUBLIC

Who? How many would we like to speak to?

How would we like to speak to them?

Chief Executive Officer 1

One interview per person, each lasting 30-45 minutes

Finance Director 1

Director of Patient Experience

1

Director of Nursing 1

Ward Managers 1

Clinical Director 1

Medical Director of A & E 1

Nursing Lead within A & E 1

Doctors 2-3

Nurses 6-8 A single discussion group of 60 minutes

If you think there are other people within your Trust who work with FFT and should be involved we would also be keen to speak to them. In most cases we would seek to conduct one-to-one interviews, as outlined above. However, if there are difficulties arranging suitable times, we would be happy to interview some staff in pairs, especially those who work closely together on matters relating to the FFT.

We will be taking a different approach for nursing staff. We feel that it is best to speak to nurses of a similar grade in a group setting, to get a better understanding of how they talk to one another about the FFT, their perceptions and use of the Test, and encourage them to share stories and opinions together.

In order to ensure that this work is as inclusive as possible, we could adopt other approaches if convening a discussion group of nurses is impractical. These approaches could include spending some time within staff rooms or canteens during break periods so staff can contribute their views as and when suits them.

We could also request that we can add suggestion boxes to such public areas, asking staff to complete a brief open-ended questionnaire asking for their thoughts on the FFT. However, we realise that this may not be easy to arrange and set up. We would agree the exact approach – deciding what is practical and achievable within the timeframe – in consultation with you and the case study Trusts.

What will we be talking about?

We are keen to talk to staff about a range of issues relating to the FFT in your Trust, including:

Implementing FFT and the challenges staff may have encountered the methods used to ask the question, and any suppliers you may

work with Perceptions of the FFT at all levels of your Trust


13-089420-01 | FINAL v1 | PUBLIC

How / whether the FFT is used to drive service improvement the FFT score, and perceptions of the method used to calculate this The way the FFT is perceived and used by patients Suggestions for improving the FFT and lessons that can be applied

to wider roll-out of the FFT

What about patients and the public?

We will also be speaking to a range of patients and the public about the FFT. We are keen to speak to three different types of public and patients.

General public within your area who have not attended A&E / hospital since the FFT was introduced. Our own team of specialist recruiters would recruit 8-10 people to attend a 90 minute discussion group in which the FFT would be discussed.

Recent patients who have been asked to complete the FFT. We would appreciate your assistance in recruiting this group: we would invite patients (via a short recruitment question which we will provide) to take part when they are being asked to complete the FFT. Ipsos MORI would then collect details from your Trust and their specialist recruiters would arrange for 8-10 patients to attend a 90 minute discussion group. The recruitment question would include consent for Ipsos MORI to follow up with those agreeing to take part.

Highly engaged patients who are involved with local patient bodies such as Healthwatch and PALS. We would appreciate your guidance on which organisations might prove most fruitful to recruit 2-3 highly engaged patients. Ipsos MORI would then approach these people and make arrangements with them for an interview. Ipsos MORI will arrange venues for the discussion groups – these will not be held on-site at your Trust. Public and patients would be asked to discuss the following issues:

Awareness and knowledge of the FFT; Reaction to the FFT – whether from experience (i.e. patients) or

when presented by moderators (i.e. the general public); Views on the methodologies used within the Trust to ask the FFT

question(s); Feedback on the presentation of FFT scores, including stimulus

material presented by moderators based on current scoring system;

What happens next? We are keen to speak to you about this in more detail over the telephone. We can provide more detail if required, and answer any questions you may have. A researcher from Ipsos MORI will be in touch in the next few days to talk about next steps. Thank you for taking the time to help us.


13-089420-01 | FINAL v1 | PUBLIC

10.3 Discussion guide for case study staff interviews

Introduction Introduce self and project: Qualitative review of the Friends and Family Test (FFT), part of a larger process being undertaken by NHS England, getting structured feedback from all Trusts, this is part of a case study focus on 10 carefully selected Trusts. Talking to a wide range of staff and public / patients so we hear from lots of different people, which will help NHS England to understand how the FFT is working and how it could work better in the future. Explain Ipsos MORI role – independent, etc Level of attribution and anonymity Ask for consent to record Please tell me briefly about your role in this Trust… Note to participants: We are now going to discuss a variety of aspects of the FFT – how it is administered in your Trust, how it was set up, what the data is used for and what patients think of it. You may have more to say on some issues than others, or feel more informed on some than others. This is why we are making sure we speak to a wide range of people across your Trust, so if you know less about some areas, that’s absolutely fine and we can skip it.

Notes Timings -TBC

Top of mind responses to FFT ASK ALL I’d like to start by talking about your immediate thoughts on the FFT… What do you think FFT is for? What do you use it for?

Probe: Who else uses it; what for?

How is it perceived within your Trust, do you think? How does the FFT fit with other measures of patient experience?

PROBE: What other measures? Is there overlap? E.g. seamless fit vs starting again?

What questions have you been asked about FFT by other members of staff?

PROBE: Any issues? For who? How often? More now / less now compared to during set up...

Question wording, if needed: “How likely are you to recommend our ward / department to friends and family if they needed similar care or treatment?”


13-089420-01 | FINAL v1 | PUBLIC

Some basic facts about FFT in your Trust ASK SENIOR MEMBERS OF STAFF: CEO or PATIENT EXPERIENCE REP I'd now like to get a sense of how the FFT works in this Trust – as you know, there are different ways of approaching how the FFT is administered, how it is asked and how it is reported... Who is responsible for the FFT in your Trust?

PROBE: Job titles/Roles/Structure of FFT roles What about you? What is your involvement with the FFT? Please can you explain to me exactly how the FFT operates in your Trust?

PROBE: A&E, Inpatient or Maternity departments / Mode(s) used / who administers it / when administered.

Information given to staff, patients Collecting and processing data

Is there a follow-up question(s)?

IF YES: Wording and why / Development of question IF NO: Why not?

Can you show me any materials from the FFT?

PROBE: Questionnaires, in-house reporting, materials for showing results to patients, etc.

Do you use a supplier to help with the FFT? IF YES: Which supplier; what they do; why this supplier;

describe the supplier relationship IF NO: Why?

Setting up the FFT ASK SENIOR MEMBERS OF STAFF: CEO, PATIENT EXPERIENCE REP, DIRECTORS I would now like to discuss how the FFT was set up within your Trust… How easy or difficult has the set-up been?

PROBE: What went well; problems encountered; ease of using the guidance on implementation; use of guidance now; reporting eligible populations for the RR denominator

Were there any differences in how the FFT was implemented in A&E, maternity and inpatient services?

PROBE: Collating data from different settings / Collating data from different modes/comparing these data / challenges faced / differences between wards


13-089420-01 | FINAL v1 | PUBLIC

Setting up the FFT ASK OTHER STAFF How did you first find out about the FFT?

PROBE: how it was introduced/information given/quality of information

How were you involved in the set-up of FFT?

PROBE: What went well/problems encountered/ease of using the guidance/use of guidance now.

Response Rates ASK ALL I’d now like to talk about response rates to the FFT. What do you know about response rates in your Trust?

PROBE: Emphasis placed on response rates internally, importance of response rates to the Trust.

NOTE FOR MODERATORS: For CEOs and PE Lead, as this question as “How much emphasis do you put on response rates? Why is that?”

Nationally, response rates have doubled since the FFT was first introduced. Do you know if something similar happened in your Trust?

PROBE: What has helped/been tried?

What measures have you taken to ensure that as many patients as possible are given the opportunity to answer the FFT?

PROBE: Ensuring all eligible patients are asked; keeping track of who should be asked, ensuring patients are only asked once, etc

What would you like to see done differently with the FFT to get more patients to complete it, if anything? MODERATOR NOTE: If “gaming” is mentioned, ask what they know and how they know it. Do you know whether the response rate differs between A&E and inpatient settings?

Why?

How much does the response rate differ between wards? PROBE: Why? What are these wards doing

differently? Effect on the data?


13-089420-01 | FINAL v1 | PUBLIC

Using the FFT ASK ALL I’d now like to talk about how the data from the FFT is used… What do you think of the data that has been produced so far?

Why do you say that?

Please tell me a bit more about the open-ended question(s)...

PROBE: processing the data/Use off open-ends/confidence in open-ends

How is the FFT used in your Trust? PROBE: Who: the Board / departmental heads / ward

managers / clinical staff etc. Sharing with staff; examples? Sharing with patients; examples? Use of data: Improving patient experience/ service

improvement/ informing staff about performance: any examples?

How is the data used? Who analyses the data? What data is used alongside FFT data?

Use of open ended responses/usefulness of open ended responses within the Trust

Differences across setting / services / wards

How if FFT used outside your Trust? PROBE: Who: patient choice / general public / commissioners

/ other Trusts etc. How data is reported to the national portal;

challenges Comparisons across Trusts/regions etc, how? Should FFT data be used this way? Why / why not?

How do you use the FFT?

PROBE: Frequency; level of detail; what is it used for

Who is FFT data most useful to? PROBE: You / Senior staff / Frontline staff / Patients /

Public / Commissioners - Why?


13-089420-01 | FINAL v1 | PUBLIC

For CCGs ONLY ASK COMMISSIONERS I'd now like to ask some questions about commissioning and the FFT... What measures of patient experience, if any, do you use to support your commissioning?

PROBE: Why these measures? Which are best? Why?

What about the FFT? How can this be used to inform commissioning?

PROBE: any examples of how it has been used, either by you or colleagues?

Is there a local CQUIN scheme? Tell me more about that... If no scheme, why not?

What would you change about the FFT to make it more useful for commissioning?

PROBE: How would this change be helpful? NOTE ON COMMISSIONING: CQUIN – Commissioning For Quality and Innovation payment scheme: The CQUIN payment framework enables commissioners to reward excellence, by linking a proportion of English healthcare providers' income to the achievement of local quality improvement goals. Since the first year of the CQUIN framework (2009/10), many CQUIN schemes have been developed and agreed. This is a developmental process and commissioners are encouraged to share schemes (and any supporting information on the process used) to aid transparency and support improvement in schemes.

1. There is a national CQUIN on response rates (15%+) but not the FFT score.

2. In some areas, local CQUINs for good scores have been put

in place by the CCG.

3. In future there will be a national CQUIN to do with the score, but the nature of this has not been finalised (e.g. it could be to do with consistency, or improvement, or relative score, rather than with absolute score).


13-089420-01 | FINAL v1 | PUBLIC

Scoring the FFT ASK ALL We are also interested in what you think of the way the FFT is scored… What’s a good score, do you think? And what’s a bad score?

PROBE: Examples/ what does this depend on?

Do you know what the latest score is for this Trust? PROBE: Are you happy with this score? Why/why

not?

What do you know about the way the score is calculated? PROBE: Ease of understanding for you/public/

patients/other staff

Do you use the numerical score within your Trust, or do you use a different measure? PROBE: IF DIFFERENT MEASURE, please tell me more about what measure you use…

Why do you use this measure? What advantages does that have over the numerical

score? And disadvantages? How useful is the score within your Trust? Who is it most useful for? (i.e. frontline staff,

management, patients, etc)

What do you think about how the score is calculated? What changes would you make to how the score is calculated?

PROBE: Impact of those changes

Scoring calculation: Proportion of respondents who would be extremely likely to recommend (response category: “extremely likely”) MINUS Proportion of respondents who would not recommend (response categories: “neither likely nor unlikely”, “unlikely” & “extremely unlikely”) This gives a score of between -100 and +100 and must be presented as a numerical score and not a percentage.

The future ASK ALL If you were to read an article about the FFT in, for example, the HSJ five years from now, what do you think that article will say?

PROBE: Challenges faced by the FFT Main lessons learned Success stories for the FFT


13-089420-01 | FINAL v1 | PUBLIC

Wrap up ASK ALL Has the FFT worked as you thought it would? If you were going to implement the FFT again, what would you do differently?

PROBE around mode, o/e question, suppliers, specific settings

What is the most important thing to change about the FFT now?

What effect will this have?

Is there anything else you would like to say about the FFT? THANK AND CLOSE


13-089420-01 | FINAL v1 | PUBLIC

10.4 Discussion guide for case study frontline staff interviews

Introduction Introduce self and project: Qualitative review of the Friends and Family Test (FFT), part of a larger process being undertaken by NHS England, getting structured feedback from all Trusts, this is part of a case study focus on 10 carefully selected Trusts. Talking to a wide range of staff and public / patients so we hear from lots of different people, which will help NHS England to understand how the FFT is working and how it could work better in the future. Explain Ipsos MORI role – independent, etc Level of attribution and anonymity Ask for consent to record Please tell me briefly about your role in this Trust… Note to participants: We are now going to discuss a variety of aspects of the FFT – how it is administered in your Trust, how it was set up, what the data is used for and what patients think of it. You may have more to say on some issues than others, or feel more informed on some than others. This is why we are making sure we speak to a wide range of people across your Trust, so if you know less about some areas, that’s absolutely fine and we can skip it.

Notes Timings - TBC

Initial thoughts on the FFT ASK ALL What three words would you use to describe the FFT? How useful do you think the FFT is? How is it perceived within your ward / department, do you think?

PROBE: Immediate peers, nurses, clinical staff,

patients, Trust leadership, etc

Question wording, if needed: “How likely are you to recommend our ward / department to friends and family if they needed similar care or treatment?”


13-089420-01 | FINAL v1 | PUBLIC

The FFT in your ward / department and the patient perspective Your involvement with the FFT ASK ALL I’d like to start by talking about your involvement with the FFT. How have you personally been involved in the FFT on your ward / department?

PROBE: Frequency of interaction, tasks / responsibilities,

how FFT fits in with other responsibilities

And who oversees the FFT in your ward/department? Asking the FFT ASK WARD MANAGER IN DETAIL BUT DO COVER WITH OTHERS Please talk me through the how a patient is asked the FFT in your ward / department? Tell me how it works in your ward / department…

PROBE: Wording, mode, timing, etc

Please tell me about any follow-up questions you ask

PROBE: Wording, typical responses, etc

NURSES ONLY How do you think patients find completing the FFT?

PROBE: Questions patients ask, how they feel about

being asked the FFT Who is most likely to take part

The set-up of the FFT ASK WARD MANAGER IN DETAIL BUT DO COVER WITH OTHERS Going back to when FFT was being set up, how did you first find out about the FFT?

Context for interviewer key here – presence of supplier versus in-house arrangements


13-089420-01 | FINAL v1 | PUBLIC

What information were you given?

PROBE: Guidance, instructions, etc

Were you collecting any other patient feedback prior to the FFT?

PROBE: What sort of feedback was being collected, how

it was being collected, how it fits with the FFT now

Please tell me how was it set up in your department?

PROBE: Challenges, successes

How do you think the set-up of the FFT in your department / ward could have been better? What changes, if any, have you made to the way the FFT is implemented in your ward / department since it was set up?

PROBE: Mode, supplier, etc Why were the changes made?

Response rates ASK WARD MANAGER IN DETAIL BUT DO COVER WITH OTHERS I’d now like to talk about response rates to the FFT. What do you know about response rates in your Trust?

PROBE: Emphasis placed on response rates internally,

importance of response rates to the Trust Nationally, response rates have doubled since the FFT was first introduced. Do you know if something similar happened in your Trust?

IF YES: What helped increase response rates here?

IF NO: What, if anything, has been tried to improve response rates?

What would you like to see done differently with the FFT to get more patients to complete it, if anything? MODERATOR NOTE: If “gaming” is mentioned, ask what they know and how they know it.

MODERATOR NOTE: Currently, hospitals get paid a national quality premium (CQUIN) for achieving a response rate of over 15%.


13-089420-01 | FINAL v1 | PUBLIC

Scoring the FFT ASK ALL We are also interested in what you think of the way the FFT is scored… What’s a good score, do you think? And what’s a bad score?

What does this depend on, if anything? PROBE: Ward, department, service

Do you know what the latest score is for this ward/department?

Are you happy with this score? Why/why not?

What do you know about the way the score is calculated?

o PROBE: o Ease of understanding

And what do you say to patients about the scoring?

What changes would you make to the way the FFT is scored?

PROBE: expected impact of these changes

Scoring calculation: Proportion of respondents who would be extremely likely to recommend (response category: “extremely likely”) MINUS Proportion of respondents who would not recommend (response categories: “neither likely nor unlikely”, “unlikely” & “extremely unlikely”) This gives a score of between -100 and +100 and must be presented as a numerical score and not a percentage. MODERATOR NOTE: It might be less misleading to think of the FFT score as an average of individual scores, with possible individual scores of -100, 0, and +100. This makes it easier to see that “Likely” responses gain a middle-ground score (of zero). There is a belief among some staff that “Likely” responses simply do not count.


13-089420-01 | FINAL v1 | PUBLIC

Using the FFT data ASK ALL I’d now like to talk about how the outputs from the FFT are used … Are FFT outputs shared in your ward/department? Where have you seen it? IF IT IS SHARED: How is it presented to you?

PROBE: How often do you see it? Score, rankings/comparisons, time series, free

text comments, coded/synthesised presentation of text, such as word clouds

Ask for examples… I’d now like to talk about what you think of the FFT outputs… How are the outputs useful to your job?

PROBE: What store do staff place in the FFT data; can

they use it to compare across wards or other hospitals; FFT’s value as an “early warning system” for potential problems

How much do you talk about the FFT scores within your ward / department?

PROBE: Who talks about it? Is it mentioned in staff

meetings for example? Where else? Please tell me how the answers to the follow-up questions are used…

IF THEY USE THEM PROBE: Examples of typical answers / topics, what are

they most useful, what are they least useful for? To what extent can the open ended responses

be used to address levels of service or standards of care?

IF THEY DON’T USE THEM: Why, what could they be used for

What do you think of the way the data is shared or communicated?

I’d also like to talk about how the FFT outputs – the scores, the follow-up responses, anything else – are shared with patients.

Will be interesting to note how the supplier relationship (where relevant) impacts on this section. MODERATOR: Note supplier before starting interviews


13-089420-01 | FINAL v1 | PUBLIC

Please tell me how this is done… PROBE: what outputs are shared; why; how is this

done (i.e. in hospital, on a website, etc); how often is this updated.

Impact of the FFT ASK ALL Now I’d like you to consider the effect the FFT has had on your ward / in your department… What has changed for you as a result of FFT?

PROBE: Positive or negative, examples

What impact has the FFT had on how you think about patient experience in this ward / department? What could be done to the FFT to make it more helpful for you to improve patient experience?

The future ASK WARD MANAGER ONLY If you were to read an article about the FFT in a newspaper five years from now, what do you think that article will say? ASK ALL What is the most important thing to change about the FFT now? What effect will this have? If you were going to implement the FFT again in your ward / department, would you do it in the same way?

Why/why not? What improvements would you make to how

you do the FFT, if any? MODERATOR NOTE: Try and get a sense of

whether or not participants think that FFT is here to stay…


13-089420-01 | FINAL v1 | PUBLIC

Wrap up ASK ALL What else you would like to say about the FFT? What is the most important message that we should take back to NHS England about the FFT? THANK AND CLOSE


13-089420-01 | FINAL v1 | PUBLIC

10.5 Discussion guide for patient and public groups

Introduction Introduce self and project: Review of how patients can feedback to Trusts about their experiences – this is part of a case study focusing on 10 carefully selected hospitals. We are talking to a wide range of staff as well as public and patients so we hear from lots of different people. All intended to help NHS England to better understand how patient feedback works and how this could function better in the future. Explain Ipsos MORI role – independent, etc Explain how these participants were selected – live locally and have either been patients recently or haven’t been patients in the past 9-10 months. Level of attribution and anonymity Ask for consent to record Get participants to introduce themselves Warm up Please tell me one good thing about your last visit to a hospital…

Notes 5 mins

Patient feedback ASK ALL In what ways do you think hospitals gather feedback from patients who have used its services, if any?

PROBE: personal experience of giving feedback

Why do you think the NHS collects this information? And how do you think patient feedback information might be useful to you?

PROBE: what information would you look for? Quality of care? Finances? Patient feedback? Etc. Where would you go to get this information?

What, if anything, have you heard about something called the ‘Friends and Family Test’?

PROBE: Explain it in your own words; what is it for

10 mins


13-089420-01 | FINAL v1 | PUBLIC

The question wording INTRODUCE FRIENDS AND FAMILY TEST:

The Friends and Family Test (FFT) is a simple, single question survey that asks patients to what extent they would recommend the service they have received at a hospital department to family or friends who needed similar care or treatment. All data collected is used to calculate a score that is then communicated to staff, patients and the public. SHOWCARD FFT QUESTION, MODERATOR TO ALSO READ OUT QUESTION ASK ALL What do you think this question means?

PROBE: What do you think the hospital wants to find out by asking this question? Why do you think they are asking this question? What would they use this information for?

Who might be interested in the answers to this sort of question?

PROBE: professional audience; patients; others

What sorts of things would you be thinking about when answering this question?

PROBE: Overall experience vs specific aspects of your experience; which aspects are most important; expectations beforehand?

How, if at all, would this differ for A&E compared to inpatient services? Why?

A follow-up question is also asked of patients in this hospital… SHOWCARD FOLLOW-UP QUESTION, MODERATOR TO ALSO READ OUT QUESTION(S) What sort of things do you think you might talk about when answering a question like this?

PROBE: Quality of care, hospital environment, the way staff treated you, etc; What do you think this question is asking about? Why do you think this is of interest to the hospital?

ASK PATIENTS ONLY What do you recall about this question, if anything?

PROBE: Overall reactions; did they understand what was being asked; did they answer it; why / why not

Question wording, if needed: “How likely are you to recommend our ward / department to friends and family if they needed similar care or treatment?” MODERATOR NOTE: have examples of follow- up question(s) for each Trust

10 mins


13-089420-01 | FINAL v1 | PUBLIC

What answer did you give? Why did you give this answer?

PROBE: i.e. if you answered “extremely likely”, why that answer? If you answered the question again about the same time you were at the hospital, would you give the same answer again?

If not, why not? IF THERE IS A FOLLOW-UP QUESTION This follow-up question is also asked of patients in this hospital… READ OUT FOLLOW-UP QUESTION What sort of things did you talk about when answering a question like this?

PROBE: Quality of care, hospital environment, the way staff treated you, etc; What do you think this question is asking about? Why do you think this is of interest to the hospital?

Which part of the question did you find most relevant to you – the main part or the follow-up?

PROBE: which one best allowed you to give the feedback you wanted? Why?

For which of these questions would you be most interested in seeing how other patients have responded? Why?

NEW QUESTION – FFT AND COMPLAINTS What do you know about the complaints procedure in hospitals? If you wished to complain about some aspect of your stay in a hospital, to what extent would the FFT help with that?

What makes you say that? IF RELEVANT: What about the follow-up question?

Methodology I’d now like to talk in more detail about how and when the Friends and Family Test is asked… ASK PATIENTS ONLY How were you asked to fill in the FFT at your recent visit to hospital?

PROBE: method used, timing; pros and cons; why?

5 mins


13-089420-01 | FINAL v1 | PUBLIC

ASK ALL What do you think are the best ways hospitals could ask patients this question? Any others? NOTE RESPONSES ON FLIPCHART Are there any others that could be used?

PROBE: Pros / cons

Of these different possible methods, what would your preferred method be of being asked this question?

PROBE: Pros and cons of each method; When should patients be asked to complete the

question (i.e. in hospital or after leaving)? Why?

Which way would make you most likely to respond to the question?

PROBE: why? And which way would make you least likely to do so?

Which way would make you answer most honestly, do you think? Why?

PROBE: with staff nearby; in private; anonymously; etc.

Choice How would you find out information about a hospital or other NHS service?

PROBE: online, speak to friends or family, think about past experience, etc

How likely would you be to look for information? PROBE: Why would you look for information?

What sort of things would you want information on?

PROBE: Hospital performance, specific services (inpatient, A&E, GP, maternity, etc)

How important, if at all, is being able to choose what hospital you are treated at?

PROBE: Why is it important (or not)? What other things matter in this sort of decision?

PROBE: Is it important to have information on the hospital(s) that are best overall ?

PROBE : What about information on the worst overall ? PROBE : Would you choose based on the hospital that

achieved the highest number of ‘extremely unlikely’ ?

5 mins


13-089420-01 | FINAL v1 | PUBLIC

The score ASK ALL I’d now like to turn to the way your answers to this question can be turned into a score for hospitals or for individual wards… At the moment all answers to the FFT are collected to produce a single score. This is calculated as follows: SHOWCARD SCORING SYSTEM Have any of you seen these sorts of scores before?

IF SO: Did you know what it was for? What did you think of the score you have seen?

What do you think about how the score is calculated? PROBE: Top of mind responses; what information is it

providing; is the score calculation clear, etc. Can you think of different ways to provide a score?

Tell me more about that…how would this be helpful to you or others like you?

Do you think you would look at these scores yourselves? What for?

PROBE: comparing different Trusts, hospitals; wards, specific services, etc.

How useful would it be for each of these things? Pros and cons for each?

Scoring calculation: Proportion of respondents who would be extremely likely to recommend (response category: “extremely likely”) MINUS Proportion of respondents who would not recommend (response categories: “neither likely nor unlikely”, “unlikely” & “extremely unlikely”) This gives a score of between -100 and +100 and must be presented as a numerical score and not a percentage.

10 mins

Comparing scores ASK ALL Here are the Friends and Family scores for three different fictional hospitals… SHOWCARD COMPARING SCORES Which hospital would you say is doing best?

PROBE: What makes you say that? Which is doing worst?

Would you be confident in using hospital (A, B, C)? Why?

FLIP CHART RESPONSES AS A QUICK VOTING EXERCISE Now here are the Friends and Family Test scores for each of these fictional hospitals…

15 mins


13-089420-01 | FINAL v1 | PUBLIC

SHOWCARD What do you think of these hospitals now?

PROBE: What are “good” and “bad” scores? Why are these good and bad?

Has the score being shown this way changed your views on the hospitals? How?

FLIP CHART RESPONSES AS A QUICK VOTING EXERCISE – what has changed and why? How useful a measure would FFT be to you when choosing a hospital?

PROBE: What other information would be helpful?

If you were going to hospital for, let’s say, a knee operation, and you wanted to look at the FFT scores, what information would you really want to know?

PROBE: enough to have the FFT score for the whole hospital?; what about specific departments, wards, consultant teams, etc?

How useful would the score be for each of these? Why?

Presentation of scores ASK ALL Here are some alternative ways of scoring the same responses to the same hospitals… SHOWCARD ALTERNATIVE SCORING SET – MODERATOR, there are two sets of alternative scores, present one set at a time… Which ones, if any, do you prefer to the scoring approach we discussed earlier?

PROBE: why?; pros and cons; what does one approach do that others do not, etc.

Let’s now think about how the FFT scores could be presented. Here are some of the different ways that the scores we discussed above might be presented… SHOWCARD PRESENTATION SET…present a range of options to participants of how the FFT score could be presented MODERATOR NOTE: Go through each alternative in turn, probing about pros and cons, and also mention that one is an example of how the FFT score might appear alongside other measures

20 mins


13-089420-01 | FINAL v1 | PUBLIC

Which ones do you really like? Which ones don’t you like? PROBE: reasons for liking / disliking

Which ones do you think should be used to present hospital FFT scores?

PROBE: which are most / least intuitive to understand (and why); which formats are most / least informative about Trust performance

MODERATOR NOTE: Try and establish a top 3 of best ways to present the score. Where would you expect to find this information?

PROBE: internet / posters in hospital / leaflets etc. Where would you want to find this information?

Wrap up Having heard the discussion today, what do you think about the Friends and Family Test? How would you describe it to your own friends and family?

PROBE: What about the score?

What would make you look at the FFT scores? If you needed to go to hospital for an operation, would you look at the FFT scores? Why? What is the most important message that we should take back to NHS England about the FFT?

Thank and close Hand out incentives

10 mins

Th

13-0

1

10

10

he Friends and F

089420-01 | FINAL

10.6 Stimu

0.6.1 The FF

0.6.2 The sc

Family Test: Qua

v1 | PUBLIC

ulus mate

FT question

coring syste

alitative research

erial for p

em

h

patient annd public groups

1335

Th

13-0

10

he Friends and F

089420-01 | FINAL

0.6.3 Comp

Family Test: Qua

v1 | PUBLIC

paring score

alitative research

es

h

1336

Th

13-0

10

he Friends and F

089420-01 | FINAL

0.6.4 Altern

Family Test: Qua

v1 | PUBLIC

ative scorin

alitative research

ng set

h

1337

Th

13-0

he Friends and F

089420-01 | FINAL

Family Test: Qua

v1 | PUBLIC

alitative researchh

1338

Th

13-0

10

he Friends and F

089420-01 | FINAL

0.6.5 Presen

Family Test: Qua

v1 | PUBLIC

nting the FF

alitative research

FT score set

h

1339

Th

13-0

he Friends and F

089420-01 | FINAL

Family Test: Qua

v1 | PUBLIC

alitative researchh

1440

Th

13-0

he Friends and F

089420-01 | FINAL

Family Test: Qua

v1 | PUBLIC

alitative researchh

1441

13-089420-01 | FINAL v1 | PUBLIC

[Report Title]: [Report subtitle 142

[Job number] | [version] | [confidentiality status]

For more information Ipsos MORI 79-81 Borough Road London SE1 1FY

t: +44 (0)20 7347 3000 f: +44 (0)20 7347 3800

www.ipsos-mori.com www.twitter.com/IpsosMORI

About Ipsos MORI’s Social Research Institute The Social Research Institute works closely with national government, local public services and the not-for-profit sector. Its 200 research staff focus on public service and policy issues. Each has expertise in a particular part of the public sector, ensuring we have a detailed understanding of specific sectors and policy challenges. This, combined with our methodological and communications expertise, ensures that our research makes a difference for decision makers and communities.

Kate Duxbury Research Director Ipsos MORI Social Research Institute [email protected] Paul Carroll Research Manager Ipsos MORI Social Research Institute [email protected]

Documents

Ju ly 2014 The F riends and Family Test - Ipsos · Jo 13-0 Ju T Qu b number ; Title 89420-01 | FINAL ly 2014 he F alitative of document : D v1 | PUBLIC rien researc raft status ds