Measuring effectiveness and impact in workplace counselling and EAP settings … · 2014-05-26 · Measuring effectiveness and impact in workplace counselling and EAP settings: Current

www.bacpworkplace.org.uk

BACP Workplace is a Division of BACP

Company limited by guarantee 2175320. Registered in England & Wales. Registered Charity 298361

Measuring up?

Measuring effectiveness and impact in workplace counselling and EAP settings: Current practice, attitudes and needs

Barry McInnes – October 2012

Measuring effectiveness and impact in workplace counselling and EAP settings Page 2

The contrasting experiences of two respondents to the survey, in answer to the question ‘What feedback, if any, do you receive from the measure data you submit?’

‘[I] Can look at my own data whenever I want to. Can look at colleagues data. Invited to compare own against service data. I do not rely on the data alone to understand the effectiveness or efficiency of the service. I use an evaluation questionnaire also to give a more balanced and meaningful perspective’

‘[I receive] Only the feedback I can evaluate from comparison of measures -‐ usually at the beginning and end of contract. This is a source of frustration as I would like feedback in how my practice compares to other practitioners. ‘


Contents

4. Service based responses……………………………………………………………………………. 32 Sector base of non-‐EAP services …………………………………………………………… 32 Services provided (all services) …………………………………………………………….. 33 Current use of measures ……………………………………………………………………… 34 What factors motivate services to evaluate? ……………………………………….. 35 Areas of outcome or impact measured by services……………………………….. 36 Measures used to assess impact or outcome of interventions……………… 38 Services’ views of key measures……………………………………………………………. 39 Systems used by services to collect, store and analyse measurement

data………………………………………………………………………………………………………. 40

How useful do services find their systems of data collection?................ 41 Length of use of measures……………………………………………………………………. 42 Frequency of data collection………………………………………………………………… 43 What model of data collection do services employ? ……………………………. 43 What other areas of service quality do services measure? ………………….. 44 To what purpose is the data generated primarily put? ………………………… 46 What obstacles or challenges do services face in using measures?......... 48 Services’ attitudes to using measures…………………………………………………. 49

Executive summary……………………………………………………………………….…..……… 5 1. Introduction and background to the study………………………………………………… 8 Background…………………………………………………………………………………………… 8 Survey development and process…………………………………………………………. 9 How the report is structured…………………………………………………………………. 10 Acknowledgements………………………………………………………………………………. 10 2. Key respondent role and demographic data……………………………………………… 11 How respondents heard about the survey……………………………………………. 11 Professional affiliations………………………………………………………………………… 11 The professional roles occupied by respondents…………………………………… 12 Professional role for survey responses………………………………………………….. 13 Gender, age and theoretical orientation profiles………………………………….. 13 3. Practitioner based responses…………………………………………..……………………….. 16 EAP practitioners and the number of EAPs they provide a service for…. 16

The sector base of internally based practitioners………………………………….. 17 The proportion of practitioners that use outcome measures………………… 17 Measures used to assess impact or outcome of interventions………………. 18 Practitioners views of key measures……………………………………………………… 19 Length of use of measures……………………………………………………………………. 23 Feedback received by practitioners from their measurement data……….. 24 The impact of using measures on practise…………………………………………… 25 What obstacles or challenges do practitioners face in using measures? 27 Practitioners’ attitudes to using measures……………………………………………. 29


5. Needs for further support and guidance……………………………………………………. 51 What needs do services and practitioners have in implementing and

using measures? ………………………………………………………………………..…………. 51

6. Conclusion and recommendation ………………………………………………….…………. 53 Appendix 1. Practitioners views of key measures (graphs and data tables)…. 55 Appendix 2. Qualitative feedback from practitioners on the measures they

use……………………………………………………………………………………………………………….. 59

Contact details 64


Executive summary Background

1. This study was commissioned by the British Association for Counselling and Psychotherapy (BACP) to further its understanding of how the workplace counselling and Employee Assistance Programme (EAP) sector is responding to the challenge of measuring its quality and effectiveness, and to inform BACP’s strategic approach in this area of its activity

2. 155 people in all responded to the survey, 132 from a clinical or practitioner role and 23 from non-‐clinical or service based roles, including five EAPs, 10 internally based services and eight other services (p 11 -‐ 15)

3. The typical survey respondent is female (76%), aged between 56 – 65 (46%) and describes their theoretical orientation as integrative (p 11 -‐ 15). Their most common professional affiliations are BACP individual member (88%) and BACP Workplace individual member (60%)

Practitioners

4. Of the practitioners in an internal service role, those in NHS (34%), higher or further education (25%) and the private sector (21%) were most strongly represented (p 17)

5. 75% of the 132 practitioners indicated that they use measures of outcome in their practitioner role. (p 17) A significantly higher proportion of EAP practitioners than internal practitioners indicate that they use measures of outcome (86.4% compared with 68.8%) (p 18)

6. CORE is by far the most commonly used measure, indicated as being used by 64% of respondents. Also in common use are GAD-‐7 (26%), PHQ-‐9 (21%) and EAP case closure reports (27%) (p 18)

a) Practitioners appear to find each of the most commonly used measures broadly acceptable, easy to score and determine relative severity, and also a pre and post-‐intervention outcome. Some issues were noted with the Outcome Rating Scale in terms of its perceived ability to capture data relevant to the clients' presenting issues, acceptability to clients and clients being clear about the meaning of the questions or statements (p 19 – 21)

7. 64% of practitioners indicated they have been using measures for more than five years. Only five percent have been using them for less than a year (p 23)

8. Comparison of practitioners by service type (i.e. EAP v. Internal) shows that practitioners within internal service roles receive a higher level of feedback than those in EAP roles. Across the two practitioner groups, more than four-‐fifths (82.1%) who indicated they received no feedback were from an EAP practitioner role (p 24)

9. Respondents indicated a wide range of positive effects to their use of measures, including the identification of clients at risk (76% of respondents), determining the client’s level of distress or need at assessment (71%), and validating the client’s sense of progress (69%). One-‐fifth (20%) said that using measures has little or no impact on how they practice and just 6% feel that measures represent an unhelpful intrusion into the therapeutic frame (p 25)

10. The most common obstacle or challenge to using measures, cited by nearly half (49%) of respondents, was time to administer measures, followed by the cost of a system to collect or analyse the data (35%). A perceived lack of expertise exists in knowing how to use measures optimally with clients (17%) , and how to understand or analyse the resulting data (21%) (p 27)

11. Internal service practitioners highlighted more obstacles or challenges than their EAP counterparts, with an average of 2.4 obstacles per respondent compared with an average of 1.2 for EAP practitioners. For every obstacle or challenge the proportions from internal services were higher, in some cases by a factor greater than four (p 28)


12. A very sizeable majority (73%) of respondents are either generally, largely or highly positive about the value of measures, contrasted with those who are either generally sceptical (11%) or largely or highly negative (1%) (p 29)

13. There appear to be associations between length of use of measures, feedback, and service type, and practitioners’ attitudes to measures. The differences, however, while notable, are not statistically significant (p 29-‐31)

Service based responses

14. There were 23 responses from non-‐clinical roles – 5 EAP, 10 internal service and 8 others 15. Of the respondents who were based in internal or other (non-‐EAP) services, the highest

proportion (28%) identified their sector as higher or further education. 22% of services indicated that they have contracts with organisations in more than one sector, the same proportion as those operating in the private sector (p 32)

16. The most common services provided are time limited counselling/ therapy, offered by 19 of the 23 responding services (83%), followed by consultation for managers or supervisors (74%), psycho-‐educational programmes for staff (65%), coaching and mentoring (61%), and mediation (39%). There are few differences between the EAPs and internal services in this sample in terms of the services offered (p 33)

17. Two-‐thirds of services (65%) use measures. The proportions of EAPs and internal services that currently evaluate outcomes is equal at 60% (p 34)

18. The highest rated motivating factor for using measures was service development and improvement, rated by 86% of services as critically important, followed by the need to provide evidence of service impact to key stakeholders, rated as critically important by 80% of services (p 35)

19. Notable differences exist between the three EAPs and six internally provided services that responded. All of the internal services indicated that monitoring the progress/recovery of clients was critically important, whereas this was highlighted as critically important by one EAP and moderately important by the remaining two (p 36)

20. The two most commonly evaluated outcomes are improved psychological health/wellbeing, cited by 93% of services, and reductions in levels of stress (80%). The profile of outcomes measured by EAPs and internally provided services appears very similar (p 37)

21. The position of most commonly used measure is equally split between the standardised CORE measures (the CORE-‐OM or CORE 10), and services’ own bespoke measures, each of which was recorded by 54% (n = 7) of services (p 38)

22. The most common systems for collecting, storing and analysing data were in-‐house solutions, including bespoke databases (53%), spreadsheet analysis of data entered from paper forms (27%), and simple ‘paper and pencil’ analysis of paper based data (20%) (p 40)

23. More than two-‐thirds (67%) of services have been using measures for three or more years, with no difference between EAPs and internal services (p 42)

24. The great majority of services (79%) collect outcome data routinely for all of their clients. Whereas all five of the internal services collect data routinely for all clients, this was the case for only one of the three EAPs. The remaining two collect data on a periodic basis with samples of clients (p 43)

25. In terms of the other outcomes that services monitor, all monitor levels of client satisfaction, and 73% monitor length of wait to an assessment or first contact session. Only just over half (53%) monitor client drop out or unplanned endings, and less than half of services (47%) appear to profile clients’ levels of risk routinely (p 44)

26. The vast majority of services (87%) use their measurement data to serve the needs of managers or external stakeholders in some way. Internal services are more likely to enable practitioners to have access to their own data to support their development, to use data to benchmark


performance against external benchmarks or other services, and in individual practitioner performance assessment/appraisal (p 46 -‐ 47)

27. Services perceive the time to administer measures (71%), and attitudinal or ideological standpoint (43%) as the most significant obstacles (p 48). This is in marked contrast to the views expressed by practitioners, of whom only 49% saw time to administer measures as a major factor, and only 19% said that their own attitudinal or ideological standpoint was a barrier

28. The overwhelming majority of services (87%) were either generally, or highly or largely positive about the use of measures of outcome. Internally based services appear to be slightly more favourably disposed than EAPs to using measures of outcome, with 90% of the internal services indicating that they are generally, largely or highly positive (p 49)

Needs for support and guidance

29. A wide range of needs for further guidance were expressed by respondents, with no option recorded by less than 34% of respondents and no more than 47%. This range of options, from selecting appropriate measures to using evaluation data for service development purposes, encompasses the spectrum of potential uses from early beginnings in measurement to more sophisticated purposes (p 51)

30. When the needs of practitioners and services are contrasted, higher proportions of services than practitioners express a need for guidance on which aspects of quality, effectiveness or impact to measure (43%), on choosing appropriate measures to determine outcomes/impact (67%), and on managing evaluation data systematically (62%) (p 51)


1. Introduction and background to the study Background This study was commissioned by the British Association for Counselling and Psychotherapy (BACP) to provide an understanding of how the workplace counselling and Employee Assistance Programme (EAP) sector is engaging with the challenge of measuring and monitoring its quality and effectiveness. The results of the study will be used to inform BACP’s strategic approach in this area of its activity, working alongside the staff and Executive of its specialist division, BACP Workplace. The project and programme for this study, developed by the BACP Workplace Executive, seeks to better understand current practice and attitudes among both workplace counselling practitioners and services, and also their perceptions of their needs for further guidance and support. Specifically, the survey set out to address a number of key questions:

1. What proportion of workplace counselling and EAP practitioners and service providers use measures of outcome or impact within their services, and for how long have they been using them?

2. What measures and measurement systems are currently in use in the sector, and how do practitioners and services perceive their value?

3. What feedback is provided to practitioners by services from the measurement data that they and their clients generate, and what is the impact of using measures on therapeutic practice?

4. What challenges and obstacles do practitioners and services face in using measures, and what are their overall attitudes to their use?

5. What are the range of services provided by workplace counselling and EAPs, and what areas of outcome or impact, and wider service quality, do they currently evaluate?

6. From a service perspective, what are the key factors that drive their evaluation effort? 7. What models of measurement do services employ, and is this routine for all clients? 8. To what use is the measurement data generated by practitioners and clients put by services? 9. What needs for further support and guidance do both practitioners and services have?

The study consisted of a detailed survey of the workplace counselling and EAP community in the UK, including members of the BACP Workplace division, BACP members whose membership data indicates they have a role or interest in workplace counselling, and members of the EAPA (Employee Assistance Professionals Association) UK, the BACP Workplace Trauma Network , the Association for University and College Counselling (AUCC) and the BACP Workplace LinkedIn forum. BACP Workplace is the specialist workplace division of the BACP, and exists to promote best practice in workplace counselling and other employee support interventions. It provides a community for all professionals with an interest in employee support and psychological health and wellbeing at work. BACP Workplace is the professional home for workplace counsellors and psychotherapists, providing a range of services including regional and special interest networks and a quarterly journal. Barry McInnes (the report author) is an independent therapist, coach, and consultant in service improvement and outcome measurement in psychological therapy. Former roles include Head of Service for the Royal College of Nursing counselling service, and Director with CORE IMS Ltd, providing consultancy, support and training in service improvement and outcome measurement to services and commissioners. He is a member (and former Chair) of BACP Workplace, former member of the BACP Research Committee and a Visiting Lecturer at Brighton University, School of Applied Social Sciences.


Survey development and process The development of the survey structure was led by the report author, working closely with the BACP Workplace Executive and Professor John McLeod, Professor of Counselling at the University of Abertay Dundee, who has acted as a consultant to the process. Questions D1 – D4 (see section 2. Key respondent data: roles and other demographics) collected basic information about how respondents had heard about the survey (D1), their professional affiliations (D2), professional roles (D3), and from which specific role – primarily clinical/practitioner or non-‐clinical/organisational (e.g. manager, case manager, OH) they were responding from (D4). Depending on which role they responded from at D4, respondents were then taken to the relevant sections of the survey questionnaire which asked further detailed questions about their use of measurement tools in their practise or service. The concluding sections of the survey collected further details from all respondents including basic demographic details, theoretical orientation and needs for further guidance. Questions D1 – D4 were required responses and collected data from all survey respondents. All other questions were optional. Information about the survey and an invitation to respond were sent to:

BACP Workplace members (email bulletin and contact networks)

BACP members whose membership data records they practise, or otherwise have a special interest in, the workplace sector (email bulletin and contact networks)

BACP Workplace Trauma Network members (contact networks)

Association for University and College Counselling (AUCC) staff counsellors and services (contact networks)

Employee Assistance Professionals Association (EAPA) UK members (contact networks)

Members of the BACP Workplace LinkedIn forum

The data collection period ran between 19 March 2012 – 16 May 2012, and each target group was sent a reminder in addition to the original invitation to respond to the survey. In all, 155 responses were received by the closing date -‐ 132 from practitioner roles and 23 from service roles. NOTE: The sample on which the data in this report is based is a convenience sample of those who responded to the invitation to complete the survey. As such, it should not be seen as being representative of the wider workplace counselling and EAP community from which it is drawn. Any findings, however interesting, cannot be reliably said to generalise to that wider population. That notwithstanding, a few sections of the report present data which have been tested for statistical significance (using a chi-‐square test), on the basis that ‘if this were a representative sample of the wider population, then this finding could also be said to be true for that population.’ The appropriate caveats are noted where such data are presented.


How the report is structured The report contains the following key main sections: Executive summary (p 5) Section 1. This section – the introduction and background to the study (p 8) Section 2. Contains key data from all survey respondents – how they heard about the survey, their roles and professional affiliations, and other demographic data including their theoretical orientation (p 11) Section 3. Contains the data for respondents who responded to the survey from one of the three practitioner roles about their experience of using measures in their practise and their attitudes towards measures (p 16) Section 4. Contains data from respondents who responded to the survey from one of the three service based roles about aspects of their service provision, experience of using measures and their attitudes towards using measures in their services (p 32) Section 5. Focuses on the views expressed by all survey respondents about their needs for further support and guidance in terms of implementing and using evaluation within their practice or service (p 51) Appendix 1. Contains the detailed responses of practitioners to each of the seven factors of utility of the most commonly used measures (p 55) Appendix 2. Contains the additional comments, in their original, unedited form, that were offered by practitioners on the measures that they use. Only comments that could be linked to a specific measure have been included. (p 59) Acknowledgements The author would like to record his thanks to the many people without whose support the survey and this report would not have been possible. To friends and colleagues on the BACP Workplace Executive, Andrew Kinder of the EAPA, Mary Dailey of the AUCC, Nicola Dooley, Co-‐Chair of the BACP Workplace Trauma Network, to numerous BACP staff including Claire Andrews, Carl Boucher, Karen Duffin, Richard Lallo, and Chelsea Shelley, and also to anyone whose contribution the author may have inadvertently overlooked. Special thanks to Rick Hughes, BACP Workplace Lead Advisor, and Professor John McLeod, Professor of Counselling at the University of Abertay Dundee, for his part in contributing over and above the call of duty. Finally, to all those who took the time and trouble to respond to the survey and make their views known – thank you.


2. Key respondent data: roles and other demographics NOTE: Each section heading is immediately followed by the relevant survey question itself. How respondents heard about the survey (D1. Please tell us how you heard about the survey) From the 155 responses recorded a total of 84% (n = 130) indicated that they had heard about the survey from either a BACP Workplace or wider BACP communication. The remaining 16% were through EAPA, AUCC or other communications, or through the BACP Workplace LinkedIn forum (see Figure 1).

Figure 1. How respondents heard about the survey

Professional affiliations (D2. About your professional affiliations) Respondents were asked to indicate their affiliations to key professional organisations, with the choice of selecting more than one option. A total of 313 separate responses were recorded, giving an average of just over two affiliations per respondent (see Figure 2).

Figure 2. Professional affiliations of respondents. People may select more than one checkbox, so percentages may add up to more than 100%.


BACP individual membership was the highest recorded affiliation, indicated by 88% of respondents. BACP Workplace individual membership, and BACP organisational membership were recorded by 60% and 16% respectively. 11% recorded affiliation to other bodies, including the UK Council for Psychotherapy, British Psychological Society, Chartered Institute of Personnel and Development, Institute of Career Guidance, as well as a range of other BACP divisions. The professional roles occupied by respondents (D3. About the professional roles you occupy) Question D3 asked respondents to indicate, from a specified list, which of six professional roles they occupy. Illustrations were provided to guide their selection e.g. the category EAP clinical included counsellors working within an EAP or externally provided service, as well as those acting as affiliate counsellors for EAPs. The choices were: 1. EAP clinical (e.g. EAP or external service counsellor, affiliate) 2. EAP non-‐clinical (e.g. EAP or external service manager, case manager, support staff) 3. Internal clinical (e.g. counsellor/practitioner for internally provided service) 4. Internal non-‐clinical (e.g. manager or other role for internally provided service) 5. Other clinical (e.g. any other counsellor or practitioner role) 6. Other non-‐clinical (e.g. manager, HR, Occupational Health) Respondents were able to record multiple options to indicate their full range of roles. A total of 229 responses were recorded, indicating an average of 1.5 roles per respondent.

Figure 3. Professional roles of respondents. People may select more than one checkbox, so percentages may add up to more than 100%.

Clinical roles were those most commonly selected, with 48% of respondents indicating they have an EAP clinical role. No non-‐clinical (i.e. service based) role was recorded by more than 8% of respondents (see Figure 3).


Professional role for survey responses (D4. From which role are you responding to this survey?) Respondents were asked to indicate from which one of the roles indicated at D3 they were responding to the survey. Of the 155 responses recorded, the highest proportions were from those in EAP clinical and internal service clinical roles (38% and 31% respectively). Overall, respondents in clinical roles form 85% of the total responses (see Figure 4).

Figure 4. The roles from which people responded to the survey.

The count of respondents by category is given in Table 1, showing that 132 responded from clinical roles and 23 from non-‐clinical (i.e. organisational) roles.

Response to survey by role n %

EAP non-‐clinical 5 3.2%

Other non-‐clinical 8 5.2%

Internal non-‐clinical 10 6.5%

Total non-‐clinical 23 14.9%

EAP clinical 59 38.0%

Other clinical 25 16.1%

Internal clinical 48 31.0%

Total clinical 132 85.1% Table 1. Breakdown of respondents by clinical and non-‐clinical roles


Gender, age and theoretical orientation profiles Respondents were asked to state their gender, age, and their main theoretical counselling orientation.

D5. Gender The gender breakdown of the sample is shown in Figure 5. Of the 153 respondents that indicated their gender, the great majority were female (76%).

Figure 5. Respondents by gender D6. Age 154 respondents give details of their age. The distribution of their ages is adjacent (Figure 6). The majority of respondents were in the 46 – 55 and 56 – 65 age bands (28% and 46% respectively). Only one percent of respondents (n = 2) were aged under 35.

Figure 6. Respondents by age

D7. Theoretical orientation 151 of 155 potential respondents indicated their main theoretical orientation. Some chose more than one option, and this question yielded 234 responses in all, an average of 1.55 per respondent. The chart adjacent (Figure 7) shows two breakdowns of each theoretical orientation.

Figure 7. Respondents by theoretical orientation. People may select more than one checkbox, so percentages may add up to more than 100%.


Firstly, (in blue) each orientation as a proportion of the total number selected by all respondents, and second (in red), as a proportion of the respondents who indicated one single orientation (n = 100). An integrative orientation was by far the most commonly selected, whether overall (61%), or as a unique orientation (51%). It is interesting to note the relative proportions for CBT as an orientation. Whereas CBT was selected by 21% of respondents overall, it is indicated as a unique approach by only four percent. Further analysis was carried out to explore differences between female and male respondents in terms of their expressed theoretical orientations (excluding those from the Other category). From the total responses (see Table 2), males were more likely to indicate CBT as an orientation than females (29% compared with 18%), and also humanistic and integrative orientations. A slightly higher proportion of female respondents indicated person centred and psychodynamic orientations.

Orientation

Female (n = 116)

Male (n = 34)

n % n %

CBT 21 18% 10 29% Humanistic 24 21% 8 24%

Person centred 36 31% 9 26% Psychodynamic 23 20% 6 18%

Integrative 70 60% 22 65%

Total 174 55 Table 2. The number and percentage (as a proportion of total responses) of responses by theoretical orientation.


3. Practitioner based responses The following sections highlight the responses from those in clinical/practitioner roles (i.e. EAP clinical, Internal clinical and Other clinical) to questions about their use of and experience with using measures of outcome, their opinions about the appropriateness and usefulness of those measures, the impact of using measures upon their practise, the feedback they receive from their collection of measurement data, and their overall attitudes to using measures in their practise. A total of 132 respondents (see question D4 above) responded from a clinical role. Each section heading is immediately followed by the relevant survey question itself. In some cases, where sample size has been sufficient to enable comparison, the data has been cross-‐tabulated with another variable (e.g. service type and use of measures are compared below to establish any differences between the proportions of EAP and internally based practitioners that are currently using measures). Where sizeable differences were detected, a chi-‐square test for significance was run to determine whether any differences were likely to reach the level of statistical significance. NOTE: As noted earlier, the sample on which the data in this report is based is a convenience sample, and, as such, should not be seen as representative of the wider workplace counselling and EAP community. Any findings, however interesting, cannot be said to generalise to that wider population. EAP practitioners and the number of EAPs they provide a service for (EAP1. If you are an EAP affiliate, please tell us how many EAP's you routinely provide a service for) Practitioners that responded from an EAP clinical role were asked to indicate how many EAPs they routinely provide a service for. A total of 56 people responded to the question (see Figure 8).

Figure 8. EAP practitioners and the number of EAPs they provide a service to Respondents that provide a service for only one EAP were in a clear minority at 13%. Just over a third (34%, n = 19) provide a service for five or more EAP providers.


The sector base of internally based practitioners (IP1. In what sector is your work located?) Internally based practitioners were asked in what sector their work is located. Respondents were able to select more than one option, and this question attracted a total of 97 responses. The largest proportion indicated that their work is located within the NHS (34%), followed by higher or further education (25%), and the private sector (21%). Figure 9 refers.

Figure 9. Sector location of internally based practitioners. People may select more than one checkbox, so percentages may add up to more than 100%.

The proportion of practitioners that use measures of outcome or impact (AP1. In your role as practitioner do you use measures that assess the outcome or impact of your interventions, e.g. in improved psychological health, reduced levels of stress, improved absence etc?) Respondents were asked the question “In your role as practitioner do you use measures that assess the outcome or impact of your interventions, e.g. in improved psychological health, reduced levels of stress, improved absence etc?” All practitioners responded to this question (n =132).

Figure 10. The proportion of practitioners using measures


Three quarters of respondents (n = 99) indicated that they use measures of outcome or impact as part of their role, with a further 14% (n = 18) indicating that they were not presently using measures but that their use was under consideration (see Figure 10).

Service type Total

EAP clinical Internal

clinical

Using

measures

Yes n 51 33 84

% 86.4% 68.8% 78.5%

Not at present, but

under consideration

n 3 11 14

% 5.1% 22.9% 13.1%

No n 5 4 9

% 8.5% 8.3% 8.4%

Total n 59 48 107

% 100.0% 100.0% 100.0% Table 3. Use of outcome measures for EAP and internal service practitioners. EAP practitioners were more likely to be using outcome measures in their roles. This result is statistically significant ( χ2 =7.488, p=0.024)

A significantly higher proportion of EAP practitioners (see Table 3) indicate that they use measures of outcome than internal practitioners (86.4% compared with 68.8%). This is perhaps not surprising given that 87% of EAP practitioners indicated that they were providing services to at least two or more EAP providers. The result is statistically significant, meaning that if the current sample were representative of the wider population from which it is drawn, the result could be generalised to that population. It is also interesting to note the relatively high proportion of internally based practitioners (22.9%) who indicated that the use of measures was under consideration.

Measures used to assess impact or outcome of interventions (AP2. Please tell us what measure or measures you use to assess this impact) Practitioners were asked to specify what measures of outcome they employ to determine outcome or impact. A total of 249 responses were recorded for this question from 98 respondents, giving an average of 2.5 measures per respondent. CORE is by far the most commonly used measure, indicated by 64% of respondents. Also in common use are GAD-‐7, PHQ-‐9 and EAP case closure reports (see Figure 11). The latter is to be expected given the high proportion of responses from practitioners in a clinical EAP role. The high prevalence of use of GAD-‐7 and PHQ-‐9 may merit further exploration to understand whether, for example, this is a reflection of the influence of the Improving Access to Psychological Therapies (IAPT) programme, or their perceived usefulness as brief measures. The measures noted under Other, where details were given, show them to be mostly bespoke outcome and satisfaction measures used by EAPs and internal services (one exception was the Depression, Anxiety and Stress Scale, DASS-‐21).


Figure 11. Measures used by practitioners to assess the impact of their interventions. People may select more than one checkbox, so percentages may add up to more than 100%.

There are small, but not statistically significant, differences between patterns of use between EAP and internal service practitioners across the more common measures. For measures with a combined count of 10 or more across the two practitioner groups, Table 4 shows the proportions from each group using those measures.

EAP practitioners n = 46

Internal Practitioners n = 27

n using measure

% using measure

n using measure

% using measure

CORE 37 79% 19 70%

BDI 6 13% 4 15%

GAD-‐7 14 30% 6 22%

PHQ-‐9 11 23% 5 19%

ORS 13 28% 1 4%

Table 4. Patterns of use across EAP and internally based practitioners

The proportions of EAP and internal service practitioners using the more commonly recorded measures is broadly similar, with the exception of the Outcome Rating Scale (ORS). Thirteen EAP practitioners indicated they were using the ORS, in contrast with one from an internal service. The reasons for this are not clear, neither is it known how many EAPs actually use the ORS, but it may be that ORS is favoured among some EAPs as an ultra brief (four item) measure of outcome.

Practitioners views of key measures MP1 & 2. How do you rate the usefulness of this measure, according to the following criteria:

b) The measure captures data relevant to the clients' presenting issues c) It is acceptable to clients


d) Clients are clear about the meaning of the questions or statements e) It is appropriate to use with clients whose first language is not English f) The measure is acceptable to practitioners g) It is easy to score and determine relative severity h) It is easy to determine the pre and post-‐intervention outcome

At question AP2. practitioners were asked to say what measures of outcome they use to determine the impact or outcome of their interventions. Building on the responses they gave, they were then asked to give their views on the usefulness of these, according to a range of seven factors (a – g above), each of which was rated on a five point scale from Strongly agree to Strongly disagree. Practitioners were given the option of giving feedback on up to two measures of their choice. From their responses the most commonly indicated measures were selected for comparison across the seven factors. Standard measures (e.g. GAD-‐7, PHQ-‐9) that were selected less than seven times were excluded, as were non-‐standard measures such as EAP case closure reports and service bespoke measures. There four standard measures remaining were:

• Outcome Rating Scale (n =8) • PHQ-‐9 (n = 9) • GAD-‐7 (n = 7) • CORE-‐OM and CORE-‐10 (n = 60)

The sections which follow briefly summarise the responses from practitioners for each of the seven factors as well as their more expansive qualitative feedback. The charts on which these summaries are based, which show the distribution of ratings across the seven factors (each accompanied by its own data table) are shown at Appendix 1. It should be clearly borne in mind that any differences in practitioner perceptions between measures (with the exception of the CORE-‐OM and CORE 10) are based on a count of less than 10. a) The measure captures data relevant to the clients' presenting issues

With the exception of the ORS, 85% or more of respondents either agree or strongly agree across all measures with the statement ‘The measure captures data relevant to the clients' presenting issues’.

The likely reason that 26% of responses for the ORS either disagree or strongly disagree may lie in its construction as an ultra brief measure of only four items. Whereas other measures focus on specific problems or condition-‐specific statements (e.g. ‘I have felt OK about myself’), the ORS asks clients to rate themselves on a 10 point scale across four broad domains of wellbeing or function:

Individually (Personal well-‐being) Interpersonally (Family, close relationships) Socially (Work, school, friendships) Overall (General sense of well-‐being) b) It is acceptable to clients

Again with the exception of the ORS, there was a high level of agreement across measures with the statement ‘It is acceptable to clients.’ Across PHQ-‐9, GAD-‐7 and CORE, 89% or more of respondents agreed or strongly agree with the statement. The reason why 38% of ORS responses disagree with the statement is hard to understand. The measure is the briefest of all four measures and hence might be expected to be the most acceptable. Based as the responses are on only three people, this may simply be a statistical quirk, or otherwise may suggest that clients experience the lack of specificity of the questions as problematic. Some support for


this view is provided by the responses to the subsequent statement ‘Clients are clear about the meaning of the questions or statements.’ c) Clients are clear about the meaning of the questions or statements

Across all measures, with the exception of the ORS, there is a high level of agreement (78% or higher agree or strongly agree) with the statement ‘Clients are clear about the meaning of the questions or statements.’ 26% of ORS responses either disagree or strongly disagree, which may relate to the issue of the acceptability of the ORS to clients highlighted above.

d) It is appropriate to use with clients whose first language is not English

This question produced the lowest levels of agreement and the broadest distribution of scores of all the questions. If the responses were based on the use of English language versions of the measures with clients whose first language is not English the results would be easier to interpret. Each measure is available in a range of languages other than English, however, in which case it might be logical that respondents disagree with the statement. e) The measure is acceptable to practitioners

All measures appear to display a high level of acceptability to practitioners. For each, 75% of more of responses agree or strongly agree with the proposition. The proportions of respondents for the ORS and PHQ-‐9 that disagree or strongly disagree with the statement (25% and 22% respectively) are unlikely to be statistically significant. f) It is easy to score and determine relative severity

To what extent do practitioners perceive their chosen (or in some cases imposed) measures as easy to score, and from that to determine the relative level of severity of clients? Across all measures 85% or more of responses either agree or strongly agree with this statement. All four measures therefore appear to be perceived as easy to score and gain a sense of the relative level of severity of a client. In the case of CORE, this could perhaps be seen as a little surprising, given that CORE respondents included both users of the ten item CORE-‐10, but also the longer 34 item CORE-‐OM. g) It is easy to determine the pre and post-‐intervention outcome Overall, most respondents seem to find it relatively easy to use their chosen measures to gain a pre and post-‐intervention measure of outcome. On all measures, except GAD-‐7, 75% or more of responses either agree or strongly agree with the proposition ‘It is easy to determine the pre and post-‐intervention outcome.’ Additional qualitative feedback In addition to the standard response options for rating each factor, respondents were able to provide additional comments on the measures they use and many chose to do so. A selection appears below and the full range of unedited responses (where it was possible to link a comment to a specific measure) are contained in Appendix 2. The following comments give a flavour of the range of views of practitioners, on their strength of feeling generally:

‘An intrusion into clinical work’ (CORE)


‘Despite practitioner resistance initially, the clinical value is now embedded in the work and agreed by all concerned’ (CORE)

‘It is quick to use and provides a good measure of the client's progress’ (ORS)

‘Too simplistic. Intrusive to have to complete & record every session’ (ORS)

and on how they are used by practitioners:

‘I have used CORE-‐OM with more than 500 clients and find it reliably useful in assessing levels of distress at start of therapy. I use the measure to form the basis of some therapeutic conversations’.

‘The questionnaire's focus on the symptoms makes it very acceptable to the vast majority of clients. I also find it particularly helpful with clients who are struggling to communicate the pressures behind the symptoms. Both PHQ9 and GAD7 are used at the same time’ (PHQ-‐9 comment)

and on their wider utility:

‘It is widely used by medical practitioners with whom I work, so they can relate to it, which is extremely useful for reporting outcomes in a simple and quick way’. (GAD-‐7)

‘Gives a clear numerical score clients and practitioners are interested in before and after case is completed. Is helpful for clients to take to GP's if, for example, depression score is severe. Gives useful data to agencies seeking to provide evidence interventions are effective and is not as intrusive as CORE.’ (DASS-‐21)

Conclusion The CORE measures are by far the most commonly represented instruments in this part of the survey data, with more than six times the volume of data for CORE than any other measure. Given the lower volumes of data for the ORS, GAD-‐7 and PHQ-‐9, a good degree of caution therefore needs to be exercised in interpreting the data. Practitioners appear to find each of the measures broadly acceptable, easy to score and determine relative severity, and also a pre and post-‐intervention outcome. Any differences between the individual measures are unlikely to be statistically significant. In retrospect, the question of whether the measures are appropriate to use with clients whose first language is not English was premised on the assumption that an English language version of the measure would be used. As all four measures are available in alternative language translations the responses to this question may have been based on this knowledge and therefore have confounded the results. If there is a substantive point of difference between the responses for different measures, it may lie in the way that the construction of the ORS is viewed by practitioners. The measure appears to be seen as less likely to capture data relevant to the clients' presenting issues, be less acceptable to clients, and leave clients less clear over the meaning of individual measure items. This may be due to its construction as an ultra brief measure which focuses on broad domain areas rather than problem or condition-‐specific questions or statements. In other words it may be the lack of specificity of questions that some practitioners perceive as problematic. While the text based responses practitioners give for the ORS in Appendix 2 are mixed, there is at least some support for this idea.


Length of use of measures (AP3. How long have you been using measures in your practice?) From the responses received to the question “AP3. How long have you been using measures in your practice?” it is apparent that this subset of survey respondents are highly experienced in using outcome measures in their practise. A total 99 people responded to this question, with 64% (n = 63) indicating that they have been using measures for a period of five years or more (see Figure 12). Only five percent (n = 5) of respondents can be considered to be new to measurement, having being using measures for a period of less than a year.

Figure 12. Practitioners length of use of measures

A comparison of EAP and internal service practitioners shows that the former are somewhat more seasoned in their use of measures See Table 5). More than three-‐quarters (76.5%) have been using measures for more than five years, compared with just under half (48.5%) of internal practitioners. While sizeable, the difference just fails to reach the level of statistical significance at the 95% level of confidence.

Service type Total EAP clinical Internal

clinical

Length of use Less than a year % 2.0% 6.1% 3.6%

n 1 2 3

1 -‐ 2 years % 9.8% 21.2% 14.3%

n 5 7 12

3 -‐ 4 years % 11.8% 24.2% 16.7%

n 6 8 14

More than 5

years

% 76.5% 48.5% 65.5%

n 39 16 55

Total % 100.0% 100.0% 100.0%

n 51 33 84

Table 5. Length of use of measures between EAP and internal service practitioners


Feedback received by practitioners from their measurement data (AP4. What feedback, if any, do you receive from the measure data you submit?) Respondents were asked what feedback they receive from the measurement data they submit. 93 respondents answered this question and it is clear from the responses of many that they receive little or no feedback. Nearly one third (32%) say they receive no feedback, and a further 12% receive feedback only if they ask for it (see Figure 13).

Figure 13. The type of feedback that practitioners receive on their measure data

It may be seen as encouraging that a proportion of respondents, albeit small at 16%, say they receive regular summary feedback. It is impossible without further enquiry, however, to know what this feedback consists of and whether it is in a form that they can meaningfully use. Only one in five respondents receive feedback as part of a discussion with their managers or supervisors (14%), or in a way that is part of their formal appraisal or professional development (6%). In order to test for differences between the feedback received by EAP and internal practitioners, data for three feedback conditions were merged, namely a) Feedback as part of appraisal or professional development; b) Regular feedback and discussion with supervisor or manager and c) Regular summary feedback. These were then contrasted with data for practitioners who selected the option ‘I receive no feedback on the data I submit’.

Feedback condition Total

Feedback I receive no

feedback

Service

type

EAP

clinical

% 29.6% 82.1% 56.4%

n 8 23 31

Internal

clinical

% 70.4% 17.9% 43.6

n 19 5 24

Total % 100.0% 100.0% 100.0%

n 27 28 55 Table 6. Contrasting the feedback received by EAP and internal service practitioners. Internal practitioners receive a higher level of feedback from the data they submit. The difference is statistically significant ( χ2 =15.412, p=0.000)


Table 6 shows that practitioners within internal service roles receive a higher level of feedback than those in EAP roles. Across the two practitioner groups, more than four-‐fifths (82.1%) who indicated they received no feedback were in an EAP practitioner role. Of those that receive substantive feedback, 70.4% are from internally provided services. The result is statistically significant, meaning that if the current sample were representative of the wider population from which it is drawn, the result could be generalised to that population. Additional qualitative feedback Respondents were free to further comment on the feedback, or lack of it, on the measure data that they submit, and many chose to do so. Their comments, broken down across EAP and internally based practitioners, are shown at Appendix 3. As was the case for the practitioners’ feedback on the measures they use, their comments range across a spectrum from highly positive to negative. A small selection appear below:

‘Only the feedback I can evaluate from comparison of measures -‐ usually at the beginning and end of contract. This is a source of frustration as I would like feedback in how my practice compares to other practitioners. ‘ ‘Can look at my own data whenever I want to. Can look at colleagues data. Invited to compare own against service data. I do not rely on the data alone to understand the effectiveness or efficiency of the service. I use an evaluation questionnaire also to give a more balanced and meaningful perspective’ ‘From EAPs, as a pract, receive no feedback whatsoever. For Organisation for which I am clinical advisor, I monitor all the client feedback, forward to supervisors [having addressed and difficulties-‐ which is very rare!] and supervisors discuss with counsellors, usually Quarterly.’

The impact of using measures on practise (AP6. What effect does using outcome measures have on your practice?) Practitioners have varying attitudes to measurement which may be reflected in the impact that using measures has on their practise. From a range of options, respondents were asked to describe what effect using measures of outcome has on their own practise. A total of 393 responses were received from 98 practitioners to this multiple choice question. The responses, shown below, yield some of the most unexpected findings within the data and suggest a group of practitioners far more favourably disposed to measurement than might have been generally expected (see Figure 14). Respondents indicated a wide range of benefits to their use of measures, including the identification of clients at risk (76% of respondents), determining the client’s level of distress or need at assessment (71%), and validating the client’s sense of progress (69%). Just one-‐fifth (20%) said that using measures has little or no impact on how they practice. This suggests a more neutral position than the much smaller proportion of practitioners (6%) who feel that measures represent an unhelpful intrusion into the therapeutic frame. A range of other variables were explored to determine whether they might impact on the effects of using outcome measures in practice, including service type (EAP v internal), gender, and the level of feedback received by practitioners. It was thought, for example, that the level and quality of feedback


that practitioners receive might influence their perceptions of how using measures affects their practice.

Figure14. The impact of using measures on practice. People may select more than one checkbox, so percentages may add up to more than 100%.

Service type and effect: EAP and internally based practitioners were broadly very similar in terms of the effects of using outcome measures on their practise. The proportions for each group were close to the overall means for each effect, save for one intriguing difference in their responses to the statement ‘The client’s initial score can validate their decision to seek help’ (see Figure 15). 51% of EAP practitioners (n = 26) highlighted this as an effect, compared to only 19% of internally based practitioners (n = 6). The reasons for this difference are not clear but the difference is statistically significant.

Figure 15. Effect of using measures on practise for EAP and internal practitioners. The difference between the groups on the item ‘The client’s initial score can validate their decision to seek help’ is statistically significant ( χ2 = 8.622 p = 0.003)

Gender and effects: No substantive differences were found between female and male practitioners for any of the effects of using outcome measures on practice. Feedback and effects: Contrary to expectations, for this sample of practitioners, feedback showed little impact on the effects of using outcome measures on practice. There were no significant differences in


effects between those practitioners receiving substantive feedback and those receiving no feedback, save for slight differences in the responses to the statement ‘Measures are a tool in helping me to assess my performance and identify potential areas for development’. Here the 70% of practitioners (n = 23) receiving substantive feedback agreed with the statement, compared with 53% of practitioners (n = 16) who receive none. This difference is not sufficient to reach the level of statistical significance. It may be that there is a disconnect between the feedback that practitioners receive, or the nature of that feedback, and its application to routine practice. For example, if feedback occurs in clinical supervision, it might be expected that its focus would tend towards measure data in the context of individual clients. This could arguably lead practitioners to pay a greater degree of attention to their use of measures in their practice. Feedback as part of discussions with managers, for example as part of a formal appraisal, may be more likely to focus on aggregated performance data, requiring a shift in focus by the practitioner to connect this level of data to their actual practise. Further research could usefully explore in more detail the use of measure data in case supervision, and its impact on clinical practice and outcomes.

What obstacles or challenges do practitioners face in using measures? (AP7. Which of the following present the main obstacles or challenges to using measures in your practice?) What obstacles or challenges do practitioners face in their use of measurement in their practise, be they practical or ideological? From a specified list respondents were able select more than one option, and in total this question yielded 212 responses from 109 practitioners. The most common obstacle or challenge, cited by nearly half (49%) of respondents, was time to administer measures (see Figure16). Without knowing more about the actual measurement burden upon practitioners, it could be that some see the administration of any measure as time poorly spent. But given that only 6% of respondents feel that measures intrude unhelpfully into their practise, this is likely to be only a small part of the answer. Given the clear evidence from research about the beneficial impact of using sessional measures upon outcomes, it may be helpful to explore this area further to ascertain what variables may be involved in the perception that time to administer measures is an obstacle. These may include the sector base of practitioners, their theoretical orientation, their level of training in using measures, and the number of measures they are actually required to use routinely with their clients.

Figure 16. The obstacles or challenges to using measures highlighted by practitioners. People may select more than one checkbox, so percentages may add up to more than 100%.


From the responses it is clear that a perceived lack of expertise exists in knowing how to use measures optimally with clients (17%) , and how to understand or analyse the resulting data (21%). Lack of knowledge about the cost of measures, including fee free measure options, and about suitable systems for data collection and analysis are also common and appear to act, at least to some degree, as barriers or obstacles. Despite the high proportion of respondents who selected the Other option, only 15 elaborated further and some third of those indicated that there were no significant obstacles to using measures. Of those that did identify obstacles three that are relatively typical are shown below:

‘Few people want to undertake a close of contract measure’

‘Cannot find an outcome measure that is not based on a medical model’

‘Our current use of one measure, at beginning and end of counselling, takes very little time, with the data being recorded and analysed by admin staff -‐ but anything more elaborate would I think become intrusive and counter-‐productive time wise’

When analysed by the type of service from which they responded, some sizeable differences in the barriers highlighted by practitioners become apparent (see Table 7)

Obstacle or challenge EAP practitioners n = 47

Internal Practitioners n = 41

n (1) % (2) n (1) % (2)

Time to administer 17 36% 20 49%

Cost of a system to collect or analyse the data

10 21% 21 51%

Lack of expertise in understanding/ analysing data

7 15% 14 34%

Cost of the actual measures 4 9% 11 27%

My attitudinal or ideological standpoint 5 11% 11 27%

Lack of expertise in using measures 3 6% 13 32%

Concern about how the data might be used

2 4% 7 17%

Total 57 99 Table 7. Main obstacles or challenges cited by EAP and internal service practitioners to using measures in their practice. (1) The number of practitioners that highlighted each obstacle from this practitioner group. (2) The percentage of practitioners that highlighted each obstacle as a proportion of each group.

Table 7 shows the responses of a total of 88 practitioners, 47 from EAPs and 41 from internally provided services (n = 41). Internal service practitioners highlighted many more obstacles or challenges than their EAP counterparts, with an average of 2.4 obstacles per respondent compared with an average of 1.2 for EAP practitioners. For every obstacle or challenge the proportions from internal services were higher, in some cases by a factor greater than four. The most commonly expressed obstacles for each group were time to administer measures, lack of expertise in understanding/analysing data and the cost of a system to collect or analyse the data. That the cost of a system to collect or analyse the data was expressed as an issue by EAP practitioners is curious, as the likelihood is that the majority would be engaged as affiliates.


For all challenges save for the cost of a system to collect or analyse the data, the degree of difference between the concerns of EAP and internally based practitioners may in part be explained by the nature of the relationship each group is likely to have with their organisation. EAP affiliates are engaged on the basis that if they wish to receive referrals, they need to comply with the procedures and protocols of the EAP – in other words they may simply just ‘get on with it’. Practitioners within internally provided services, by contrast, are more likely to be employed, and expect to have both a greater stake in the service and a greater say in its operation. This suggests that the challenges for internal services looking to implement or alter their measurement practise may be more significant than those facing EAPs. Practitioners’ attitudes to using measures (AP8. Overall, what best describes your attitude to using measures of outcome or impact in your practise?) 131 practitioners responded to the question which asked, from a given set of statements, which best described their overall attitude to using measures in their practise. A very sizeable majority (73%) of respondents are either generally, largely or highly positive about the value of measures, contrasted with those who are either generally sceptical (11%) or largely or highly negative (1%). (see Figure 17)

Figure 17. Practitioners’ attitudes to using measures in their practice The fact that the great majority hold positive views about measurement seems consistent with the earlier finding of the generally positive effects expressed by practitioners of using measures on their practice. A range of other variables that might be associated with attitudes to using measures were explored, including practitioners’ length of use of measures, the level of feedback received by them, their gender, and service type (i.e. EAP or internal service). Each is explored here briefly in turn. Length of use and attitudes: Practitioners’ attitudes was cross-‐tabulated against the length of time they had been using measures (see Table 8). 54 practitioners described themselves as Generally positive and 24 as Highly or largely positive. Of these positively disposed practitioners, 68.5% and 70.8% respectively had been using measures for more than five years. While noteworthy, however, the differences across these and the other descriptors of attitude were not sufficient to reach a level of statistical significance.


Attitude Total

Highly or

largely

negative

Generally

sceptical

No strong

views for

or against

Generally

positive

Highly or

largely

positive

Length

of use

Less than a

year

% .0% 20.0% 14.3% 3.7% .0% 5.1%

n 0 1 2 2 0 5

1 -‐ 2 years % 100.0% 20.0% 14.3% 11.1% 16.7% 14.3%

n 1 1 2 6 4 14

3 -‐ 4 years % .0% 20.0% 21.4% 16.7% 12.5% 16.3%

n 0 1 3 9 3 16

More than 5

years

% .0% 40.0% 50.0% 68.5% 70.8% 64.3%

n 0 2 7 37 17 63

Total % 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

n 1 5 14 54 24 98

Table 8. The relationship between practitioners’ length of use of measures and their attitudes toward them Feedback and attitude: An association between the feedback on their data received by practitioners and their attitudes to using measures might be expected, since it could be anticipated that receiving regular and substantive feedback might serve to foster a more positive attitude. Data was available for a total of 64 practitioners in all. As can be seen from Table 9, of those holding the highly or largely positive attitude, two-‐thirds (66.7%) indicated that they receive feedback on their measure data. Of those who expressed the attitude Generally positive, however, a higher proportion of practitioners (54.3%) were in the No feedback condition. Despite what might be anticipated, therefore, the data were mixed and there was no statistically significant difference between the feedback conditions.

Attitude Total

Highly or

largely

negative

Generally

sceptical

about their

value

No strong

views for or

against

Generally

positive

about their

value

Highly or

largely

positive

Feedback

condition

Feedback n 1 1 6 16 10 34

% 100.0% 50.0% 54.5% 45.7% 66.7% 53.1%

No

feedback

n 0 1 5 19 5 30

% .0% 50.0% 45.5% 54.3% 33.3% 46.9%

Total n 1 2 11 35 15 64

% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0%

Table 9. The relationship between feedback practitioners receive and their attitudes toward measures Gender and attitude: In terms of gender, there appear to be few differences between female and male respondents in their attitude to measures (see Table 10). While a higher proportion of males indicated they were highly or largely positive, taking the categories Highly or largely positive and Generally positive about their value together, the gender proportions were equal at 73%.


Gender Total

Female Male

Attitude Highly or largely positive n 15 10 25

% 15.0% 33.3% 19.2%

Generally positive about their value n 58 12 70

% 58.0% 40.0% 53.8%

No strong views for or against n 16 4 20

% 16.0% 13.3% 15.4%

Generally sceptical about their value n 10 4 14

% 10.0% 13.3% 10.8%

Highly or largely negative n 1 0 1

% 1.0% .0% .8%

Total n 100 30 130

% 100.0% 100.0% 100.0%

Table 10. The relationship between gender and practitioners’ attitudes toward measures Service type and attitude: Turning to the relationship between service type and attitude, data for a total of 106 practitioners were available. Some modest differences were found between the attitudes of EAP and internally based practitioners (see Table 11) . A higher proportion of EAP practitioners indicated that they were highly or largely positive than internal practitioners (29.3% and 14.6% respectively). A slightly higher proportion of internal practitioners had no strong views for or against. While notable, these differences were not sufficient to reach a level of statistical significance.

Service type Total

EAP

clinical

Internal

clinical

Attitude Highly or largely positive n 17 7 24

% 29.3% 14.6% 22.6%

Generally positive about their value n 30 25 55

% 51.7% 52.1% 51.9%

No strong views for or against n 7 9 16

% 12.1% 18.8% 15.1%

Generally sceptical about their

value

n 4 6 10

% 6.9% 12.5% 9.4%

Highly or largely negative n 0 1 1

% .0% 2.1% .9%

Total n 58 48 106

% 100.0% 100.0% 100.0%

Table 11. The relationship between gender and practitioners’ attitudes toward measures


4. Service based responses A total of 23 responses were received from respondents who identified themselves as responding from a non-‐clinical role (e.g. managerial, supervisory, occupational health, HR or other support staff). This relatively low level of service based responses clearly limits the conclusions that may be drawn from such a small sample. Notwithstanding that caveat, however, the responses still provide an intriguing snapshot into the measurement practices and attitudes of the services represented. Of the 23 responses, five were from EAPs, 10 from internal services and eight from others, as shown in Table 12 below.

Response to survey by role n %

EAP non-‐clinical 5 21.7%

Other non-‐clinical 8 34.8%

Internal non-‐clinical 10 43.5%

Total non-‐clinical 23 100.0% Table 12. Breakdown of service based responses by service type Sector base of non-‐EAP services (IS1. In what sector is your service located?) Of those respondents who completed the survey from the roles Internal non-‐clinical and Other non-‐clinical (i.e. those not from an EAP non-‐clinical role), respondents were asked to specify in what sector or sectors their service was located. A total of 23 responses were recorded for this question from 18 services, indicating that some services operate across more than one sector (see Figure 18). The highest proportion of respondents (28%) identified their sector as higher or further education. 22% of services indicated that they have contracts with organisations in more than one sector, the same proportion as those operating in the private sector.

Figure 18. Location of internally provided services by sector. People may select more than one checkbox, so percentages may add up to more than 100%.


NOTE: The following sections detail the responses received from all services, including EAP providers. Services provided (all services) (AS1. Which of the following interventions does your service provide?) Services were asked to outline the range of interventions they provide. From the 23 respondents a total of 106 interventions were recorded, an average of 4.6 interventions per service. Not surprisingly, the most common intervention provided is time limited counselling/therapy (see Figure 19), offered by 19 of the 23 responding services (83%). Of the four services that do not offer time limited counselling, one was an internal service that offers only longer term counselling, while the remaining three completed the survey from other internal roles, one of which involved career guidance. The remaining services recorded under Other were clinical and managerial supervision, and welfare support to return to work. More surprising, perhaps, given the constraints on resources in both EAP’s and internally provided services, is the proportion of services offering longer term counselling (57%).

Figure 19. The interventions provided by services. People may select more than one checkbox, so percentages may add up to more than 100%.

From the responses it is also clear that many of the responding services are providing a range of ‘non-‐therapeutic’ services, such as consultation for managers or supervisors (74%), psycho-‐educational programmes for staff (65%), coaching and mentoring (61%), and mediation (39%). This offers a good illustration of the ‘added value’ components that workplace based counselling services and EAPs can offer employers over and above their more therapeutically focused interventions. Service type and interventions provided: The profile of services offered by EAPs and internally provided services appears very similar, notwithstanding the caveat of low numbers (see Figure 20.) The five EAPs specified a total of 22 services in all between them, with a mean of 4.4 per EAP. The range indicated was between one and seven services. In contrast, the 10 internal services specified a total of 48 services, with a mean of 4.8 per service and a range of between one and eight services.


Figure 20. The service provision profiles of EAPs and internal services

Current use of measures (AS2. In addition to routine audit data, does your service currently assess the outcomes/impact that its interventions deliver e.g. in improved psychological health, reduced levels of stress, improved absence etc?) All 23 services responded to the question of whether they assess the outcomes or impact of the interventions that they deliver. Over two-‐thirds of services (65%; n = 15) confirmed that this was the case, with a further 13% (three services in all) indicating that this was an issue that was under consideration (see Figure 21).

Figure 21. The proportion of services that use measures to assess impact of their Interventions

The proportions of EAPs and internal services that currently evaluate outcomes is equal at 60% (based on five and 10 services respectively).


What factors motivate services to evaluate? (AS4. How important are the following factors in motivating your service evaluation?) Services were asked to rate the importance of a range of factors in providing motivation for their service evaluation activity. Fifteen services responded to this question and all rated each factor save for ‘Service improvement and development’ and ‘Monitoring the progress/recovery of clients in therapy’, which were rated by 14 services. The chart below (Figure 22) shows how the relative importance of each factor was rated by services.

Figure 22. The relative importance of the factors motivating service evaluation

Each factor may be seen as either broadly outward facing (e.g. Providing evidence of service impact to key stakeholders) or inward facing (Service development and improvement). Of the outward facing factors, the highest rated motivating factor was the need to provide evidence of service impact to key stakeholders, which was rated as critically important by 80% of services. Two thirds of services (67%) also rate the role of evaluation in helping to give the service a competitive edge as critically important. Interestingly only one-‐third rated the place of measures in fulfilling service funding requirements as of critical importance, which may also go some way to explaining why less than half (47%) saw the need to have a system in place before one was imposed as critically important. In other words, it may be that relatively few of this sample of services are currently required to measure outcomes as part of their funding requirements, and hence that factors such as service development and monitoring clients’ progress are greater incentives. It is encouraging that the two main inward facing factors -‐ Service development and improvement and Monitoring the progress/recovery of clients are rated as critical by the vast majority of services. It suggests an awareness of the role that measurement has to play in managing both client outcomes and overall service development. Clearly, given that these data are drawn from such a small sample, whether this would generalise to the wider body of workplace counselling and EAP services is open to conjecture.


Service type and motivating factors: Notable differences can be seen between the data for the three EAPs and six internally provided services that responded to this question. All of the five internal services that responded to the item Monitoring the progress/recovery of clients indicated that this was critically important, whereas this was highlighted as critically important by one EAP (33%) and moderately important (67%) by the remaining two (see Table 13).

Service funding

requirement

Service improvment/dev’ment

Giving service competitive

edge

Monitoring progress/recov.

of clients

Having a system in place before it

is imposed

Providing evidence of

service impact

EAP (n = 3)

Not important 33% 0% 0% 0% 33% 0% Moderately important

67% 33% 33% 67% 67% 0%

Critically important

0% 67% 67% 33% 0% 100%

Internal (n = 6) Not important 0% 0% 0% 0% 20% 17% Moderately important

50% 17% 33% 0% 0% 0%

Critically important

50% 83% 67% 100% 80% 83%

Table 13. The relative importance of motivation factors across EAPs and internally based services Service funding requirements appear to play a more important part for the internal services in this sample, with 50% (n = 3) of internal services indicating this was critically important and the remainder moderately important. This was in contrast to the EAPs, none of which indicated funding requirements as being critically important. The important of this issue is probably also reflected in the internal services’ responses to the issue of having a system in place before it is imposed. This is highlighted as critically important by 80% of internal services, but by none of the EAPs. Areas of outcome or impact measured by services (AS5. What outcomes do you currently evaluate?) Services were asked to specify what outcomes (from a given list) they currently evaluate. A total of 75 responses were recorded by 15 services. The proportion of services that selected each outcome are shown in Figure 23. Perhaps not surprisingly, the two most commonly evaluated outcomes are improved psychological health/wellbeing, cited by 93% of services, and reductions in levels of stress (80%). Interestingly, these two outcomes are the only two from the list of outcome areas that can be seen as not exclusively work focused. Of those outcome areas that have a specific work focus, reduced absence/return to work is evaluated by some 60% of services, improved work performance by 53%, and enhanced work commitment by 47%. Whether these and other work focused outcomes such as workplace wellbeing and job satisfaction are areas where their host or customer organisations expect counselling and other interventions to make an impact is not known. Of the 20% of services that indicated that they evaluate outcomes under Other, few were specific save for two that cited health issues, career management skills, and finding work/education/training as outcome areas that are evaluated.


Figure 23. The outcomes that services indicate are currently being evaluated

It is beyond the scope of this study to determine precisely what is being measured, for what purpose, and exactly how, in some areas of service outcome evaluation. Nonetheless the very wide range of outcome areas for which services try to capture data is likely to say something, at least, of the expectations placed upon them by their funders and commissioners. Service type and outcomes: The profile of outcomes measured by EAPs and internally provided services appears very similar, although with the caveat of low numbers (see Figure 24.) The three EAPs that responded to this question specified a total of 14 outcome areas between them, with a mean of 4.7 per EAP. The range indicated was between one and seven. By contrast, the six internal services specified a total of 28 outcome areas, with a similar mean of 4.7 per service and a range of between one and nine.

Figure 24. The outcome evaluation profiles of EAPs and internal services


Measures used to assess impact or outcome of interventions (AS6. Please tell us what measure or measures you use to assess this impact.) Respondents were asked to say what measures are used to capture data for the outcomes highlighted in the section above. Two of the 15 services that indicated they used measures did not complete this question, hence the resulting data is generated from the responses of 13 services, which returned a total of 32 separate measures – an average of 2.5 measures per service. Figure 25 below shows the proportions of services using each measure. No returns were recorded for four of the measure options presented, indicating that none were used by responding services. These were the CORE University Spoke categories, the WASAS (Work and Social Adjustment Scale), INFORM and the AUCC client problem categorisation. The position of most commonly used measure is equally split between the standardised CORE measures (the CORE-‐OM or CORE 10), and services’ own bespoke measures, each of which was recorded by 54% (n = 7) of services. While this data confirms CORE as the most commonly used standardised outcome measure among responding services, the split between standardised and bespoke measures raises interesting questions about the relative merits of each. Standardisation offers the possibility of valid and reliable measures with a degree of comparability across services, but little option to capture the finer detail of organisational and service context and priorities. Bespoke measures are capable of capturing very fine levels of service and contextual detail, but by their very nature little or no possibility of external comparison or benchmarking.

Figure 25. The measures used by services to assess the impact of their interventions. People may select more than one checkbox, so percentages may add up to more than 100%.

Only one EAP responded to this question, so it was not possible to explore any differences between EAPs and internal services in terms of patterns of measure utilisation. Looking at the data across all services, however, is it evident that some services carry a potentially high burden of measurement (one service cited eight measures in all, six of which were standard measures, although it is not clear whether these are used routinely with all clients).


Services’ views of key measures (MS1, 2 & 3. How do you rate the usefulness of this measure, according to the following criteria) At question AS6. services were asked to say what measures of outcome they use to determine the impact or outcome of their interventions. Building of the responses they gave, they were then asked to give their views on the usefulness of these, according to the same range of seven factors (a – g below) used by practitioners to rate their measures.

a) The measure captures data relevant to the clients' presenting issues b) It is acceptable to clients c) Clients are clear about the meaning of the questions or statement d) It is appropriate to use with clients whose first language is not English e) The measure is acceptable to practitioners f) It is easy to score and determine relative severity g) It is easy to determine the pre and post-‐intervention outcome

Services were given the option of providing feedback on up to three measures of their choice. Option 1 (question MS1) yielded feedback on a total of 22 measures, and questions MS2 and MS3, six and two respectively. The only measures that were selected more than once were CORE (n = 5) and service bespoke measures (n = 6). The latter were excluded from further analysis given that these are not standard measures and hence comparison would not be like for like. Figure 26 below shows the distribution of the five services’ ratings for CORE across the seven factors.

Figure 26. Service ratings of the CORE measures across seven factors of utility


The feedback from services provides a generally positive picture of the CORE measures. 80% of responses (four of the five services) either agree or strongly agree with the statements ‘It is acceptable to clients’, ‘Clients are clear about the meaning of the questions or statements’, ‘It is easy to score and determine relative severity’ and ‘It is easy to determine the pre and post intervention outcome’. It is interesting to note that three of the total of four ‘Disagree’ responses, as well as four of seven ‘Neither disagree nor agree’ statements came from the same service, and it might be imagined that this service could be either antipathetic towards the CORE measures specifically, or towards measurement in general. Further analysis, however, reveals that the service in question is an EAP, uses CORE only with corporate customers that require its use, is ‘Generally positive about their value’ with regard to using measures, and is using spreadsheet analysis of paper based data to process CORE measure data. This suggests a potentially complex interplay of factors at work, and it may be the case that services’ views of measures become conflated with a range of additional factors such as motivation to measure and the ease with which measure data can be captured, analysed and reported upon.

Systems used by services to collect, store and analyse measurement data (SY1. How do you collect, store and analyse this data?) Services were asked about what systems they use to collect, store and analyse the data generated from their use of outcome measures. A total of 18 systems were recorded by the 15 services responding.

Figure 27. The systems used by services to collect, store and analyse their data. People may select more than one checkbox, so percentages may add up to more than 100%.

The most common systems were in-‐house solutions, including bespoke databases (53%; n = 8), spreadsheet analysis of data entered from paper forms (27%; n = 4), and simple ‘paper and pencil’ analysis of paper based data (20%; n = 3). Together these account for 15 of the 18 responses. Bespoke standardised platforms (in this case CORE PC and CORE Net), formed the remaining three responses (see Figure 27). Table 14 shows the combinations of systems used by the responding EAPs and internally provided services, from which it can be seen that in two of the three services using bespoke solutions (i.e. CORE-‐PC and CORE Net) they are used alongside in-‐house databases.


EAPs (n = 3)

1. CORE PC, In-‐house bespoke database 2. In-‐house bespoke database 3. Spreadsheet analysis of paper based data

Internal services (n = 6) 1. Pencil and paper analysis of paper based data 2. In-‐house bespoke database 3. CORE Net 4. In-‐house bespoke database 5. CORE PC, In-‐house bespoke database 6. In-‐house bespoke database

Table 14. The data collection systems used by EAPs and internal services How useful do services find their systems of data collection? (SY2. (For electronic systems) How do you rate the usefulness of the system that you employ, according to the following criteria?) Services were asked to rate the usefulness of the systems they employ, according to the following set of criteria:

• Data is easy to enter onto the system • Clients can complete the measures onscreen without completing paper forms • The system provides valuable individual summary data for clients • It provides feedback on clients’ progress during therapy • The system provides valuable summary activity reports

This question sought to address the perceived utility of the various systems used by services to collect, store and analyse their data, and 12 services in all responded. While the number of respondents is low, they do offer some limited insight into how useful services appear to find the systems they employ for managing and reporting on their data (see Figure 28). The low number of responses overall and for each separate system means that no conclusions can be drawn about the relative utility of each system.


Figure 28. Service perceptions of the utility of their systems for collecting and analysing data

Length of use of measures (AS7. How long have you been using measures in your service?) Services were asked how long they had been using measures, and 15 in all responded to this question. As was the case with practitioners, their responses reveal a group of services that are relatively seasoned in their use of measures of outcome. More than two-‐thirds (67%; n = 10) have been using measures for three or more years (Figure 29). Only two of the 15 services have been using measures for less than a year.

Figure 29. Services’ length of use of measures

Of the EAPs (n = 3) and internal services (n = 6), the proportions using measures for more than three years was an equal two-‐thirds.


Frequency of data collection (AS8. Do you collect outcome data for all clients, or with periodic samples?) From the responses received from the 14 services that replied to this question it is clear that the great majority (79%) collect outcome data routinely for all of their clients (see Figure 30). Of these, three services were careful to point out that this was only with the consent of clients.

Figure 30. Services frequency of data collection with clients

One service (presumably one that has multiple contracts with different purchasers), indicated that their use of measures is dependent on whether this is a requirement of the customer. Service type and frequency of data collection: Based on data from the three EAPs and five internally provided services, a difference in the respective patterns of data collection can be noticed. Whereas all five of the internal services collect data routinely for all clients, this was the case for only one of the three EAPs. The remaining two EAPs indicated that they collect data on a periodic basis with samples of clients. What model of data collection do services employ? (AS9. At what point of their contact are clients asked to complete measures?) Services were asked to outline what model of measurement they employ with regard to the frequency at which clients complete measures. The options included the following choices:

• Pre and Post: measures used at the start and end of the intervention • Pre and Post + : at the start and end, and at points during the intervention • Sessional use of measures • Other

14 services in all responded to this question. More than half (57%; n = 8) indicated that they use a Pre and Post model – asking clients to complete a measure at the beginning and end of their therapy journey (see Figure 31). Just over a third (36%; n = 5) use what is described as a Pre and Post + model, similar to Pre and Post but with an additional measure or measures at specific points during the intervention. One service indicated that it employs post therapy measures only – in this case a bespoke measure specific to the service. No services in this sample employ sessional measures.


Figure 31. The data points at which services capture measures for clients

Given the acknowledged problems with the traditional pre and post model of therapy measurement (i.e. that by the end of therapy is too late to discover that the client shows no demonstrable improvement, or worse that the client has dropped out so no second measure can be given), further research could usefully explore what factors inform the models of data collection that services employ. Service type and model of data collection: Two EAPs and five internally provided services responded to this question. One EAP indicated that it employs a Pre and Post model and the other Pre and Post +. Of the internal services, four of the five use the Pre and Post + model, with the remaining service using Pre and Post measures.

What other areas of service quality do services measure? (AS10. In addition to measuring outcomes, which of the following areas of service quality do you routinely monitor?) The measurement of a demonstrable outcome from a therapeutic intervention generally presupposes that a client either completes therapy, or that they are measured at least twice during its course. Given that high proportions of clients often do not complete therapy, it is important that services and practitioners employ additional means of determining both clients’ needs and their overall experience of the service. A range of quality indicators may be employed for this purpose, such as measuring waiting times, the proportion of clients that have unplanned endings, and their overall satisfaction with the service they receive. Services were therefore asked about which of a range of additional common indicators of service quality they currently use to assess clients’ levels of need, the efficiency with which services are delivered, and clients’ experiences of the services provided. The results are shown in Figure 32. A total of 77 responses were recorded by the 15 services that responded to this question.


Figure 32. The additional areas of service quality monitored by services. People may select more than one checkbox, so percentages may add up to more than 100%.

All services indicated that they monitor levels of client satisfaction with the service they receive. Laudable though this is, client satisfaction is notoriously problematic, primarily because of the usually low levels of completion of satisfaction questionnaires, and potential bias caused by non-‐completion by clients that are dissatisfied. Nearly three-‐quarters (73%) monitor clients’ length of wait to an assessment or first contact session. While on the face of it this may seem high, it also suggests that more than one in four services may not be systematically measuring how long clients are waiting for a service. Similarly, only just over half (53%) appear to monitor client drop out or unplanned endings, suggesting that many services may not have an accurate reflection of the proportions of their clients that actually complete their therapy journey. It is also of concern that less than half of services (47%) appear to profile clients’ levels of risk routinely. It may be that they, and their practitioners, are relying solely or mainly on clinical judgement in assessing the degree of risk that clients represent. There is sufficient evidence, however, that client and practitioner risk ratings often differ to suggest that this is an area that needs to be more systematically addressed by services. Overall, the data seem to suggest a patchy and rather idiosyncratic approach to monitoring quality within service provision, at least among those represented in the survey, and the potential value of a quality framework which might be developed to guide and support them. Service type and service quality monitoring: Three EAPs and six internally based services responded to this question. EAPs recorded a total of 16 service quality indicators monitored, and internal services 36 in total, yielding a mean of 5.3 and 6.0 respectively. The range of service quality indicators monitored for EAPs was between four and eight, and for internal services between four and nine. The proportions of indicators recorded are broadly similar across each service type, as shown in Figure 33. The key differentiators in this small sample are the greater degree of attention paid by EAPs to service efficiency, and the greater focus on variations in practitioner performance and relative effectiveness of therapy models or interventions among internal services.


Figure 33. The areas of service quality monitored by EAPs and internal services respectively

To what purpose is the data generated primarily put? (AS11. In terms of how the data you generate are mainly used, which of the following statements apply?) What are the purposes to which the data from measuring outcomes are put, and the balance between satisfying the demands of external stakeholders and internal purposes including service and practitioner development? Fifteen services gave a total of 50 responses to this question, indicating that measurement serves a variety of different purposes. The vast majority (87%) use their measurement data to serve the needs of managers or external stakeholders in some way, and this was the most common use of data by some considerable distance (see Figure 34).

Figure 34. The purposes to which service data is put. People may select more than one checkbox, so percentages may add up to more than 100%.


Two-‐thirds of service (67%) indicate they are using data to compare current and past performance, and a little more than half (53%) are setting objectives on the basis of the data they generate. One-‐third are using data in some way to benchmark their own performance against external benchmarks or other services to gain a sense of their relative performance. Nearly two-‐thirds of services (60%) say that their practitioners have access to their individual data to support their development, and one-‐third state that data are used as part of practitioner performance assessment or appraisal. This seems to differ considerably from the experience of practitioners which was outlined earlier that highlighted that only one in five respondents receive feedback as part of discussion with their managers or supervisors, or in a way that is part of their formal appraisal or professional development. Clearly the service sample is small and may not therefore be representative of the wider picture in workplace counselling and EAP provision. Nonetheless, the data overall suggests a relatively high level of data use among responding services directed towards service and practitioner development. Service type and data use: Three EAPs and six internally based services responded to this question. Table 15 shows the purposes for which data is used by service type. All three EAPs, not surprisingly, indicated that data is used for managers/external stakeholders. No other purpose was recorded more than once by any EAP. Five of the six internal services indicated that data is also used for managers/external stakeholders, as well as assessing their performance against past performance. Four of the six also indicated that they use data to set service objectives, and that individual practitioners have access to their own data to support their development. Half of the internal services said that they benchmark performance against external benchmarks or other services, and one-‐third indicated that data are used in individual practitioner performance assessment/appraisal. Purpose EAP (n = 3) Internal (n =6)

n % n %

Data are used mainly for managers/external stakeholders

3 100% 5 83%

We use our data to assess performance against past performance 1 33% 5 83%

The service sets itself objectives on the basis of the data we generate 1 33% 4 67%

We benchmark our performance against external benchmarks or other services

1 33% 3 50%

Individual practitioners have access to their own data to support their development

1 33% 4 67%

Data are used in individual practitioner performance assessment/appraisal

0 0% 2 33%

Table 15. The purposes to which service data is put by EAPs and internal services respectively One internal service’s further comment highlights the additional higher level impact of their data by saying ‘My data is also presented to various committees and meetings and eventually will reach the executive board i.e. forms part of the Health and Well -‐ being Strategy’.


What obstacles or challenges do services face in using measures? (AS13. Which of the following present the main obstacles or challenges to using measures in your service?) Respondents were asked which factors represent the most significant challenges or obstacles to using measures in their services. Thirteen services responded to this question, which used the same response options as the question posed to practitioners. Between them they generated 52 individual responses, or an average of four per service (see Figure 35). Services perceive the time to administer measures (71%), and attitudinal or ideological standpoint (43%) as the most significant obstacles. This is in marked contrast to the views expressed by practitioners, of whom just 49% saw time to administer measures as a major factor, and only 19% said that their own attitudinal or ideological standpoint was a barrier. Services also expressed a perceived lack of expertise in using measures (33%), and concerns over the cost of both the actual measures (19%), and a system to collect and analyse the data (29%).

Figure 35. Obstacles of challenges to using measures expressed by services. People may select more than one checkbox, so percentages may add up to more than 100%.

Service type and perceptions of obstacles or challenges: Five EAPs and 10 internally based services responded to this question. The five EAPs cited a total of ten obstacles, an average of two per EAP, while the 10 internal services recorded 25, an average of 2.5 per service. The responses of each service type for each obstacle (with the exception of the Other category) are recorded at Table 16. Purpose EAP (n = 5) Internal (n =10)

n % n %

An attitudinal or ideological standpoint 4 80% 3 30%

Lack of expertise in using measures 0 0% 3 30%

Cost of the actual measures 2 40% 2 20%

Time to administer 2 40% 7 70%

Lack of expertise in understanding/analysing data 0 0% 2 20%

Cost of a system to collect/analyse the data 0 0% 5 50%

Concern about how the data might be used 1 20% 2 20%

Total 9

24 Table 16. The experience of obstacles or challenges as perceived by EAPs and internal services respectively


From the services represented in this sample, it is apparent that time to administer measures and the cost of a system to collect and analyse the data are of significantly greater concern for internal services. The most significant obstacle perceived by EAPs is that of attitudinal or ideological standpoint, expressed by four of the five EAPs (80%) which responded to the question. It is interesting to contrast this with the 11% of EAP practitioners who expressed their attitudinal or ideological standpoint as being an obstacle. Services’ attitudes to using measures (AS14. Overall, what best describes your attitude to the use of measures of outcome in your service?) As was the case for practitioners, services were asked about their overall attitude to the use of measures in their service. All 23 participating services responded to this question and their responses are shown in Figure 36 below.

Figure 36. Service attitudes to using measures

The overwhelming majority of services (87%) were either generally, or highly or largely positive about the use of measures of outcome (see Figure 36). This shows a somewhat more positive view of measurement among services than that expressed by practitioners, of whom 73% were generally, or highly or largely positive. Among services, only 8% of services were either sceptical, or highly or largely negative. The proportion among practitioners was slightly higher at 12%. Service type and perceptions of obstacles or challenges: Five EAPs and 10 internally based services responded to this question, and their responses are recorded in Table 17.

Purpose EAP (n = 5) Internal (n =10)

n % n %

Highly or largely positive 2 40% 4 40%

Generally positive about their value 1 20% 5 50%

No strong views for or against 0 0% 1 10%

Generally sceptical about their value 1 20% 0 0%

Highly or largely negative 1 20% 0 0% Table 17. Service attitudes to using measures by EAPs and internal services respectively


Of those services represented in the sample, internally based services appear to be more favourably disposed to using measures of outcome. 90% of the internal services indicated that they are generally, largely or highly positive, and none are either sceptical or negative. The proportion of EAPs that are generally, largely or highly positive is 60%, and the remaining 40% were either generally sceptical about the value, or highly or largely negative.


5. Needs for further guidance and support The following sections address questions that were directed to all survey respondents. What needs do services and practitioners have in implementing and using measures? AL2. In terms of implementing and using evaluation within your own practice or service, which of the following areas of knowledge might you welcome guidance on? Given this snapshot of current activity and attitudes to measurement in the sector from the responses received, what do services appear to need in order to support them in this area? All respondents were asked to indicate what areas of knowledge they might welcome further guidance on from a standard set of options. In all, 501 separate responses were recorded from 137 respondents, and the proportions indicating each option is shown in Figure 37 below.

Figure 37. Needs for further guidance and support across all survey respondents. People may select more than one checkbox, so percentages may add up to more than 100%.

Apart from the Other option, no option was recorded by less than 34% of respondents and no more than 47%. The range of options, from selecting appropriate measures to using evaluation data for service development purposes, encompasses a spectrum of potential uses from early beginnings in measurement to more sophisticated purposes. The fact that the variation between highest and lowest is only 13% suggests that there are significant needs that reach across all areas of this broad spectrum. Service and practitioner needs for guidance: 21 services and 116 practitioners responded to this question, and analysis of their respective responses highlights some differences in the needs expressed by the two groups, some of which appear at first rather counter-‐intuitive (see Figure 38).


Higher proportions of practitioners indicated a need for guidance in using the data from evaluation to develop a service (34%), benchmarking service data against external benchmarks or other services (47%) and implementing evaluation in a service (40%) than did service respondents. While it might be expected that these would be needs more likely to be expressed by services, these results might simply be a reflection that services feel a relatively greater degree of confidence in these areas. The fact that a relatively lower proportion of services report lack of expertise in using measures and lack of expertise in understanding/analysing data to be significant obstacles to using measures, offers some degree of support for this hypothesis.

Figure 38. The respective perceptions of practitioners and services of their need for further guidance

Significantly higher proportions of services express a need for guidance on which aspects of quality, effectiveness or impact to measure (43%), on choosing appropriate measures to determine outcomes/impact (67%), and on managing evaluation data systematically (62%). The first two areas seem reflective of many services’ routine struggles to know how best to evidence their impact, and to find and utilise ‘fit for purpose’ measures that are also valid and reliable. This may in part explain why so many services use their own bespoke measures (as reported earlier the measures most commonly used by services are the standardised CORE measures, and services’ own bespoke measures, each of which was utilised by 54% of services).


6. Conclusion and recommendation The survey provides a wide ranging and in-‐depth picture of current activity and attitudes to the use of measures in the workplace counselling and EAP sector in the UK, and the needs of its practitioners and services for further guidance and support. In large part, that picture is both positive and optimistic. A high proportion of the sample of both practitioners and services are using measures of outcome or impact to evaluate the effectiveness of the interventions that they provide, particularly in the area of psychological health and wellbeing. Furthermore, the great majority have been using measures over a long period -‐ in the case of practitioners, for more than five years, and in the case of services, for more than three years. Practitioners indicate a wide range of positive benefits to their clinical practise of using of measures, have broadly favourable views of the measures they employ, and are generally well disposed toward their use. Only a small proportion say measures have little or no impact on how they practice. They face a range of barriers to using measures effectively, including time to administer them, knowing how to use measures optimally with clients, and how to understand the resulting data. Despite the fact that nearly half receive no feedback on their data from their services, or do so only if they ask for it, they are mostly getting on with the job of incorporating measures into their practise. The services surveyed provided a wide range of therapeutic interventions and also ‘added value’ services such as consultation, psycho-‐educational programmes, coaching and mentoring and mediation, with few apparent differences between EAPs and internal services in the range offered. EAPs and internally based services have somewhat different drivers for their evaluation activity, but both appear to be monitoring or measuring similar areas of outcome or impact. The internal services were more likely to collect outcome data routinely for all clients, to use additional measures at some point during therapy to monitor client progress, to give feedback to their practitioners, and to view measurement somewhat more positively than EAPs. Analysis of the data has been restricted by the sample size, particularly in the disappointing level of responses received from services, which has limited the conclusions that may be drawn from the data as well as the option to compare the relationships between different variables. The study is still a significant undertaking, nonetheless, and future research could perhaps be targeted to refine some areas of focus and explore new ones, for example:

1. The use of measure data in case supervision, and its impact on clinical practice and outcomes

2. The level of preparation that practitioners and service managers have in incorporating routine outcome measurement, and its subsequent impact on attitudes to and engagement with the measurement process

3. What factors inform the models of data collection that services employ, and are these models consistent with evidence from research about which models deliver optimal outcomes?

4. Do factors such as attitudes to and engagement with measures change over time, and if so, what are the factors that most affect change?

5. How are practitioners and services using measures and measure data to reflect on and improve practice or services?

A wide range of needs for further guidance were expressed by respondents, with no option recorded by less than 34% of respondents and no more than 47%. This range of options, from selecting appropriate measures to using evaluation data for service development purposes, encompasses the spectrum of potential uses from early beginnings in measurement to more sophisticated purposes. This and other


data also strongly suggest the potential value of a quality framework which might be developed to guide and support them. It is hoped that the survey and its findings provide BACP and BACP Workplace with a clear picture of current practice, attitudes and needs in the workplace counselling and EAP sector, and provide data to support their strategic development of work and further research in this arena. Recommendation The findings of the survey suggest that there are wide variations in measurement practice among both services and practitioners. Within services, while areas such as client satisfaction are routinely measured, monitoring of other key areas indicative of service quality, such as unplanned ending rates and risk, appear not to be routine. Given this, and the wide range of needs expressed by respondents for further guidance, it is recommended that BACP considers the development of a further resource or resources to guide practitioners and services in developing their measurement practice and expertise. Ideally such a resource would include the following areas:

a. Why to evaluate – the key arguments supporting the need for routine evaluation of service provision

b. What to evaluate – what key areas of service delivery quality are important to measure, not focusing solely on outcomes, and also taking into account interventions other than counselling (e.g. stress management interventions)

c. How to evaluate – the measures and the mechanics of implementing and using an evaluation system to reflect upon and develop service provision

d. Case studies from services to bring the guidance to life


Appendix 1. Practitioners views of key measures (graphs and data tables) Practitioners views of key measures (refers to text contained in section 3, page 19) MP1 & 2. How do you rate the usefulness of this measure, according to the following criteria:

a) The measure captures data relevant to the clients' presenting issues b) It is acceptable to clients c) Clients are clear about the meaning of the questions or statement d) It is appropriate to use with clients whose first language is not English e) The measure is acceptable to practitioners f) It is easy to score and determine relative severity g) It is easy to determine the pre and post-‐intervention outcome

MP1 & 2. How do you rate the usefulness of this measure, according to the following criteria: a) The measure captures data relevant to the clients' presenting issues


b) It is acceptable to clients

c) Clients are clear about the meaning of the questions or statements


d) It is appropriate to use with clients whose first language is not English

e) The measure is acceptable to practitioners


f) It is easy to score and determine relative severity

g) It is easy to determine the pre and post-‐intervention outcome


Appendix 2. Qualitative feedback from practitioners on the measures they use This section contains the additional comments that were offered by practitioners on the measures that they use. Only comments that could be linked to a specific measure have been included, and all comments are in their original, unedited form. Measure Practitioner’s feedback

Becks Dep invent Becks dep inv , does not invade the psycholigical space in the way that Core does. CORE As it is based purely on the past week, a recent blip could

show false movement from start to end of counselling

Bespoke Evaluation form

Measures whether client is satisfied, whether they have returned to work etc and is confidential.

case closure Case closure allows for a summary of the client's journey/progress and outcome, that purely scoring doesn't

Client questionnaire Client questionnaires are a useful additional data source, though perhaps not enough in themselves. They enable people to make individual comment on detailed elements of their experience which would not otherwise be captured.

Closure Report It gives flexibility and for the practitioner to work with the client in a positive way. core clients value seeing the core measurements when they enter therapy and at

conclusion CORE It would be even more useful for effectiveness evaluation if the client could be

revisited with the measure 6-‐9 months after the counselling.

CORE This tool is a fairly straightforward and easy to use one. It is time consuming at the beginning and ending of working with a client and can provide information that might not come directly out of the assessments.

CORE An intrusion into clinical work. CORE Some clients need further explanation to understand the meaning and purpose of

using this measure.

CORE I have used Core in primary care and now in EAPs. I find it mostly gives a good profile of presenting symptoms. If there is a drop in score I show them the before and after.I find it helps clients to see more clearly the improvement in themselves.

CORE Excellent measure. It informs the work. CORE I have used CORE with more than 600 clients and find it very useful. It assists me

when communicating with GPs and other health/mental health professionals regarding client issues as well.

CORE AT FIRST I THOUGHT THAT IT WOULD INTERFERE WITH THE CLIENT RELATIONSHIP, BUT THEY LIKE TO SEE THEIR PROGRESS AS MUCH AS I DO. IT IS A GOOD USEFUL TOOL

CORE I use this relationally: discussing with clients the meaning for them, how they feel etc and again at the end of counselling: using it as a way of "reviewing" and ending etc

CORE Very useful for risk and the general wellbeing, problems and functioning assessment-‐ gives me an indication which area has a problem-‐ also clients like to see the before and after graphs to show they have changed. We have used it for years and it really helps to audit our service-‐ the reports are brillinat as are the workplace pre and post therapy forms as a tool for gathering information


core Core is a useful tool to be used in conjunction with a strong assessment session and to ask the client to further explain the answers to the questions. At times the clients Ihave seen find it useful to see the outcome scores to see how there is difference. I do not believe that any measure should stand alone without further understanding of what the clients understanding and experience is in relation to the measure.

Core Despite practitioner resistance initially, the clinical value is now embedded in the work and agreed by all concerned.

Core It is useful for getting the conversation going with the client. CORE 10 With clients who first language is not English, but who have a fairly good grasp of

English and with whom I can communicate easily, I find CORE no problem. However, on a couple of occasions where the client's grasp of English is not so good, it has been difficult to explain the purpose and appropriateness of the questions, although we have managed to a greater or lesser extent.

Core net the tool is unsophisticated and clumsy core om it serves to identify and confirm the focus for therapeutic intervention and

monitor progress; it also serves as a comparative tool at closure for the client's own achievement. Core net is a much more effective and user friendly tool to use this evaluative procedure than hard copy forms

CORE OM As with all measures it is a snapshot in time. Predominantly for orgs. to assess effectiveness it provides little of value to the client but can confirm what most clients are already aware of. It can be a tool to help the clinician to ascertain whether client presentation matches a more objective assessment of psychological/emotional harmony/disharmony. However we have to accept it is subjective in that client is responding to statements.

CORE OM I can use CORE therapeutically with clients and it is a useful outcome measure. In addition it provides a comprehensive assessment of risk. I do not use the CORE-‐OM online service but would like to have some workplace benchmark data which is currently not available.

CORE OM The RISK questions are useful to triangulate with counsellor's subjective impression -‐ and because risk levels are usually so low in our setting, that it stands out when they aren't..

CORE -‐OM It provides a way in to talking about difficult experiences e.g. self harm or suicidal feelings at an earler stage than might otherwise happen. Many clients find it a useful reflection of how they feel, enabling them to gain more objectivity. Unfortunately, it does not measure all dimensions of an intervention, so occasionally I might work with a client who registrs as Healthy on the scale to begin with, so there will be no discernible effect of the intervention on CORE, but it may have enabled the client to reach a decision or consider options etc.

Core outcome I think one of it's failing is that if something has occurred just prior to the last session, then this can give a false evaluation.

core outcome I find some of the questions are a bit ambiguous. I find it useful to compare the assessment stage with the last I feel it helps client's to see how much they have improved.

Core outcome measure

It's useful for promoting discussion within the session and for showing change at end of counselling programme. It is time-‐consuming to either score by hand, or enter into computer for scoring. Results cannot be immediately seen by client, unless entered online.


CORE outcome measure 34

Particularly useful when used as part of clinical work -‐ for example to reinforce progress made to clients or as a way of helping clients explore their feelings. I am required by my service to use it at first & last sessions but sometimes use it at mid-‐contract review (total 6 session contract) or where a client is unable to see their achievements in therapy.

COREnet Frequent discrepancies between CORE scores and client's presentation. Moving client from computer chair to counselling chair interrupts the flow of the session. When used with skill the CORE statements can be useful prompts for further information. Some clients and counsellors find measurement of emotion inappropriate and an anachronism

CORE-‐OM I have used CORE-‐OM with more than 500 cloients and find it reliably useful in assessing levels of distress at start of therapy. I use the measure to form the basis of some therapeutic conversations.

CORE-‐OM CORE provides a baseline assessment and risk assessment at the same time. It is possible to assess client's progress at any stage, and can provide a valuable therapeutic tool when discussed with clients in session where appropriate.

CORE-‐OM 34 I administer the measure at every meeting so that the client and I can work on their current state.

counselling evaluation

Feedback from clients experience is encouraged. We welcome all feedback positive/negative and neutral. It can help us to improve the service Its the best way to grow and improve the service by listening to clients needs. It is pointless providing a service no one can access or understand. My experience is clients like to be informed. An open, transparent, healthy attitude is welcomed.

CounsellingResource Feedback Form

This Client-‐Counsellor Feedback form consists of 14 pertinent questions spread over 3 headings.[2] 10 questions covering the working relationship between the client and myself, 2 regarding the results of my work with the client and, a further 2 questions for the client to answer based on their overall satisfaction of the service I provided to them. The invitation being made to the client to mark their closest answer to each question by putting them within 5 categories on offer. These being :-‐ Strongly Agree (2), Somewhat Agree (1), No Strong Feeling (0), Somewhat Disagree (-‐1) and Strongly Disagree (-‐2). This necessitated me to give a value or score to each of the 5 categories from which I could then make a simple spreadsheet in which all the scores/raw data could be added up to form totals. In this way not only would each individual client's scoring and total score be captured, but collectively and as the returns grew in number, an overall score would be obtained from the answers or scores given for each of the 14 questions. The values or scores given being (2),(1),(0),(-‐1),(-‐2).

DASS21 Gives a clear numerical score clients and practitioners are interested in before and after case is completed. Is helpful for clients to take to GP's if, for example, depression score is severe. Gives useful data to agencies seeking to provide evidence interventions are effective and is not as intrusive as CORE.


EAP Local Government

Some evaluation of the providers efficiency and the relationship with the counsellor. Several questions around outcomes including has counselling enabled the client to remain in the workplace / return to the workplace from sick leave, etc. All questions have a sacore of 0 -‐ 10 with a couple of lines for comments on each question.

EAP own evaluation form

The feedback measures more the efficiency of the service and the relationship with the counsellor with just one indicator as to whether or not counselling had been useful -‐ all rated 0 to 10. Then a comments box a very simple form, one sheet written on two side.

EAP's own reporting Even though I only know work for two EAP's these questions are hard to answer because they are so different.

Force QA form It is purely a feedback form on the usefulness of the TRiM assessment process. GAD 7 Brief enough to be used in time limited work -‐ brief enough to be used in

telephone counselling -‐ clients often expand on questions

GAD7 Again useful for clients to get a sense of where they are. A useful tool for both practitioner and client

GAD-‐7 It is widely used by medical practitioners with whom I work, so they can relate to it, which is extremely useful for reporting outcomes in a simple and quick way.

HADS Using this sytem was not a choice. Medical personnel wished it. HADS Clients find this method quite easy to understand and use. IEQ After frequent use experience allows the practitioner to evaluate levels of PTR

and whether the client may or my not develop PTSD

IES good for measuring symptoms, useful for client and practitioners IES It is useful to indicate whether to refer on due to the client being in danger or

developing or suspected of having developed PTSD.

Impact of Events Generally it is helpful. Some clients obviously lie and some misunderstanding the meaning of some questions -‐ never-‐the-‐less, it is generally helpful

Inform Completed by Counsellor-‐client doesn't see OCR I have amended my form when I offer it to clients. I put in the numbers 1 -‐ 10 and

ask clients to ensure they write numbers on the scale otherwise it is open to my interpretation if there were only to make a mark on the scale. Clients appear happy to do this.

ORS It is irritating for Clients to have to complete the paperwork at the start of the session before we can usefully engage, as this is a requirement to complete the measure at the start. The counsellor measure can vary enormously with regard to how a person views life generally. Some enthusiastic and very high scores and others not clear why or how they are measuring or what might have been done more to reach a 10 score. eg measuring a 9 and asking what could have been better the reply is that it was perfect and couldn't have been improved, so what does the 9 really mean or is it that some people "cannot" give a perfect score?

ORS It is quick to use and provides a good measure of the client's progress. ORS Clients can be confused as their is no 'scale' listed. ORS Too simplistic.

Intrusive to have to complete & record every session ORS I am not so keen on this measure, tho must use it for one of the EAPs I work for.

The reasons are: -‐ its cumbersome and very wasteful of paper. -‐ i am uncomfortable with the related SRS (session rating scale) that accompanies it: this is difficult for clients and in one case recently, I think it prevented the client


from returning for her 2nd session of counselling.

Own bespoke iT IS DIFFICULT TO MEASURE OUTCOMES BASED ON ORIGINAL QUESTIONS AS OTHER ISSUES MAY ARISE DURING THE COUNSELLING OR INDIVIDUALS EXPERIENCE OF 'MEASURED' ISSUES MAY DIFFER FROM MEASURES SCOPE

Own bespoke The data collected isn't currently used for affecting the service or developing the department.

PHQ 9 Brief enough to be used in time limited work -‐ brief enough to be used in telephone counselling -‐ clients often expand on questions

PHQ9 It is widely used by medical practitioners with whom I work, so they can relate to it, which is extremely useful for reporting outcomes in a simple and quick way.

PHQ9 Useful in engaging client to assess their own place and progress PHQ9 The questionnaire's focus on the symptoms makes it very acceptable to the vast

majority of clients. I also find it particularly helpful with clients who are struggling to communicate the pressures behind the symptoms. Both PHQ9 and GAD7 are used at the same time.

phq-‐9 The PHQ-‐9 appears more fitting for CBT practitioners, although some counselling clients like to 'see' a change to some of their symptoms.

presenting analysis and outcomes

useful for reflection and research on efficy

BACP House

15 St John’s Business Park

Lutterworth LE17 4HB

t: 01455 883300

f: 01455 550243

e: [email protected]

w: www.bacp.co.uk

Company limited by guarantee 2175320

Registered in England & Wales.

Registered Charity 298361

BACP Workplace is a Division of BACP

Contact detailsBACP WorkplaceBACP House15 St John’s Business ParkLutterworth LE17 4HBe: [email protected]: www.bacpworkplace.org.uk

Rick Hughes – BACP Lead Advisor: Workplacee: [email protected]

Barry McInnest: 01273 681509m: 07914 843855e: [email protected]: www.barrymcinnes.co.uk

Documents

Measuring effectiveness and impact in workplace counselling and EAP settings … · 2014-05-26 · Measuring effectiveness and impact in workplace counselling and EAP settings: Current