EVALUATION PLAN

Development Intervention The Edward Jenner Programme

Sponsor/Lead Chris Lake, Head of Professional Development

Context The new Edward Jenner programme launched in 2015.The programme has been re-designed [from the original 2013 version], in partnership with health and care staff and is for everyone working in a health and care context. The programme is highly practical and patient-focused; it’s a great way to understand the purpose, challenges and culture of the NHS.The design is flexible and enlightening, giving get a fresh perspective on the impact individual staff have on the patient experience – either directly or indirectly. It’s also a valuable refresher at any level. Between June 2015 and June 2016 XXXX participants have started the programme, assigned into 12 cohorts to date. 1001 stories have been uploaded to NHSx.

Intervention The intervention has been a desk-top exercise in the form of a quantitative and thematic analysis of 100 stories [10%] and an analysis of date taken from the Virtual Campus

Theory of change The Theory of Change applied to this programme is as follows:

Participants are motivated to take responsibility for their learning and development and have sought the most appropriate intervention for their needs

Participants are able to engage with the on-line learning and complete the requirements, applying the earning in practice

Participants will have a better understanding of the fundamentals of leadership

Participants will have a better understanding of themselves and be able to use this to improve patient care and staff engagement.

Expected measurable outcomes

As this was a desk top exercise, the Measurable were limited to what could be extracted from the

Evidence See reportReporting and Dissemination

To be uploaded to the Academy and LDP library

EVIDENCE GATHERED

Development Intervention Evaluation of the Edward Jenner programme [2015 version]

Sponsor/Lead Chris Lake Head of professional Development

Before During After

Individual Information taken from individual stories of motivations for undertaking the programme

Information taken from data analytics of participants on the programme

Information taken from individual stories of learning and change achieved as a result of undertaking the programme

Organisational Nil Nil NilProvider Nil Nil Nil

EVALUATION REPORT

Development Intervention Evaluation of the Edward Jenner programme (2015 version)

Sponsor/Lead NHS Leadership Academy Professional development team

Date 1.7.2016 Author Clare Price-Dowd

1. Executive Summary

The Edward Jenner programme is a foundation level programme in for anyone aspiring to a leadership role for the first time, or recently in a new leadership position. Launched in 2015 it has so far attracted 12,916 participants of whom 1067 have submitted a final story as of 1st July 2016.Analysis of a 10% random sample shows that that within the sample, clinicians and those working in acute trusts are most frequently accessing the programme. Everyone was able to apply the learning into practice and 89% had seen results, with a further 11% having implemented learning but it was too early to report results. No one reported not being able to apply some aspect of the learning to make change – either to themselves or their service.

The most commonly cited development seen in in the increased ability to engage others and in personal influence.

Participants would recommend making more time to undertake the learning; involving their team and being open minded to what the learning tells you.

Participants showed a strong commitment to sustaining the changes made. A number reported this as already in place.

This was a random sample that generated a 50/50 split of male to female respondents and a 70/30 split of clinical to non-clinical staff, which is not representative of the NHS as a whole. The extent to which the results can be said to be generalizable to the whole population that have submitted a story is questionable.

2. Introduction

The Edward Jenner programme is a MOOC [Mass Open Online Course] available to access free of charge for anyone working in health and social care in any part of the world. It is also open to patients.

Re-launched in 2015 the current programme replaces the original Edward Jenner programme [2013], which was originally aimed at nurses but was access by many professional groups. The original programme lacked appeal to a wider audience so was redeveloped to meet the needs of anyone new to leadership regardless of role, organisation type and background.

The programme consists of two units Launch and Foundations and is designed to be completed over 6 weeks, with application of learning in the workplace which forms the basis of a series of on-line posts and a final, optional, submission of 1000 words to gain the NHS Leadership Academy Award in Leadership Foundations.

EVALUATION REPORT

This evaluation looked at information supplied by participants as past of their stories. This related to their role, organisation, motivation for applying and learning gained.

The results are based on a random sample. The extent to which this can be seen at totally representative is not known.

3. Method

One year after the launch of the programme [5th June 2016], 1001 stories from participants who have competed the Jenner programme have been uploaded onto NHSx. These are all in the public domain. A 10% sample was selected using an on-line sampling grid system. This generated 100 random participant numbers, which became the 100 stories analysed. The benefit of a randomised sample is that it is impartial and the sample came from across the whole year.The downside is that it is not possible to positively look for specific things such as the experience of BME participants, or to get a balance of clinical and non-clinical participants.These 100 stories were downloaded and coded in batches of 10. Coding applied was limited by the information that was asked of participants in their story.

At the end of the Jenner programme, participants are supplied with a story template to complete in no more than 1000 words. In reality the word length varied a lot. The longest storied considered was 1500 words and the shortest 450. The latter was still a pass and highly reflective, with over 15 additional posts on their learning uploaded to the discussion forum so it was not surprising that this last piece was not that long. All stories are considered equally and no penalty was made for length deviations.

Participants were asked to reflect on the following questions:1. Key learning point(s) Define why this particular learning was so significant for you.

2. Background and setting Explain a bit about your role and how you see your leadership developing as a result of this programme.

3. Your actions

4. Effects of the change. What outcomes have you noticed as a result of the learning you have implemented?

5. Lessons learned, Would you have done anything differently? What advice would you give to a colleague starting the programme?

6. Next steps. What steps will you take to embed your learning into your daily practice beyond this programme?

By chance there were no fails amongst the 100 in the sample. To date 5 stories have been deemed not to meet the criteria and 6 people have uploaded a blank document. In all cases, the participants has been contacted and been asked to re-look at their submission

EVALUATION REPORT

and resubmit it which they have.

Codes were created for the following [see Appendix one for code list]. Gender [taken from name and cross checked with application form] Role Organisation type Motivation for undertaking the programme Key learning - what has participants found most useful Ability to apply learning in practice Lessons/What they wished they had done differently. Plans for sustainability

These areas translated well from the areas of the story template. Where information was not clear or not stated, this was gained from the application from.It would be possible, with more resource to analyse the participants application data further for more granularity but this is not an intention within current resources

4. Findings and Discussions

The results are limited to that which could be extracted form the story and the basic information available to view on the participants page on NHSx.In some ways this report and the results contained in it are a false positive. Uploading a story at the end of the programme is not compulsory and to date, 12,916 participants have started the Jenner programme and 1067 have submitted a story as of 15th June 2016. The completion rate is 8.2; which although it appears low is well above the national average for a MOOC. Those that have completed are the ones motivated to do so or are those who wish to have a certificate of attainment for their personal CPD. The uptake rate represents just less than 1% of the NHS staff [Source NHS Confederation May 2016 – NHS staff 1,318,000 including staff providing services into the NHS.]

1 Gender of participantsDespite using a random sampling technique, the gender split was exactly 50/50 male to female. This is not representative of the make up of the NHS where overall 77% of theNHS workforce is female.

EVALUATION REPORT

male female Trans0

10

20

30

40

50

60

2 Organisation typeParticipants in the sample were overwhelmingly from acute trusts in that they made up 82% of the sample. The rest comprised community/mental health trusts, ambulance trusts, CCGs and non-NHS provider organisations. Of the participants from private providers n=6 listed these comprised nurses, participants who described themselves as ‘managers in supply organisations’ – or similar - and a high-street optician. No one in the sample was from primary care.

acute

MH/Community Tr

ust

Ambulance

Trust

CCG

non NHS

primary

care

0102030405060708090

3 RoleParticipants in the sample were 70% from clinical backgrounds and 30% non clinical [the 6 students were all clinical] which is above the NHS % of 51.5% clinical staff [source: NHS Confederation May 2016]

There was an assumption that the Jenner programme, being a foundation level programme, would attract new leaders. However, it has become very apparent that senior leaders, who have never had any formal development, are also access in the programme. Drs universally stated their career position for example ‘I am a junior Dr, I am a

EVALUATION REPORT

anaesthetics registrar or I am a consultant gastroenterologist’. None of the Drs just stated ‘I am a Dr’ This was not true of the other professional groups where seniority was largely omitted ‘I am a nurse working in critical care’ or I am a MSK physiotherapist’

Doctors make up the largest proportion on the participants in the sample. When all grades of Dr are combined as in the second table below, they represent 33%, which is 3 times the percentage of Drs working in the NHS as a whole. Drs currently represent 11% of the workforce, whilst nurses are represented at a rate lower than the national average of 24% across the NHS.

Role with Drs separated out by seniority

Dr. Junior

Dr Mid Care

er

Dr. Consulta

ntnurse

amb/para

pharm AHPHCS

admin

non-Clin m

anag

er

studen

t

non NHS man

ager

dentist

optician IT

02468

1012141618

Role with all Drs combined

Dr [all]

nurse

amb/para

pharm AHPHCS

admin

non-Clin m

anag

er

studen

t

non NHS man

ager

dentist

optician IT

0

5

10

15

20

25

30

35

40

4 Motivation for applying.Participants were not explicitly asked for their motivation for applying but the section on background and setting universally contained this information. All stories contained at least 1 reason for undertaking the programme; this evaluation has looked at up to two reasons. The most common reason, stated both first and second was lack of development in

EVALUATION REPORT

leadership, followed by career development and being new into role with leadership responsibility. Typically participants wrote

‘I have just been promoted to a new role where I lead a team and have never had any development to do this’

‘I have never received any formal leadership development and am looking to apply for a promotion’

Some of the more interesting reasons stated were observing poor practice;

‘I don’t want to be like people I see’

Whilst others were looking for ways to work that causes less stress. Some admitted that their own style was the source of conflict, whilst others wanted to use the learning for challenging an operational issue. These ranged from things like service re-configuration, to increasing productivity.

lack o

f deve

lopment

feeliin

g junior

work iss

ues -opera

tional

work iss

ues -pers

onal sty

le

caree

r devt

new pro

motion

suggeste

d by man

ager

want to

lead ch

ange

part of a

nother pro

g

didn’t wan

t to be li

ke own m

anage

r

personal l

earning/i

nterest

revali

dation

newly

qualified

stress

01020304050607080

5 Learning gained

EVALUATION REPORT

The following was gained from looking at the first section on the template

[Key learning point(s) Define why this particular learning was so significant for you]

Participants tended to state their answer in one of two ways, either they talked about the development they had seen in themselves, as shown in the first of two tables below, which related to personal impact, ability to influence, ability to engage others, greater team working and increased confidence or they talked about the individual pieces of learning they found most beneficial.

For the latter the most beneficial was the content around patient centered care, team working, personal style, Hero vs. Host and then some of the models and frameworks presented.

It is interesting that very few participants actually mentioned wanting to develop leadership ‘skills’ when in effect what they described were largely skills. The most popular reason for applying was lack of development so how participants view what they have gained is individually described.

6 Personal impact gained from learning

confindence

influence

engag

emen

t

delega

tion

teamwork

communication

listen

ing

change

d attitiudes

trust

relati

onshipsski

lls

feedback LA

AL

complexity

decisio

n mak

ing0

5

10

15

20

25

30

35

7 Most useful piece of content [where stated]

EVALUATION REPORT

Berwick

Host vs H

eroFra

ncis

Patien

t centre

d care

HLM

Johari W

indow

Personal l

eaders

hip Style

Engag

emen

t/stak

eholders

teams

Adaptive

leaders

hip

Emotional I

ntellige

nce0

5

10

15

20

25

30

8 Application of the learning in practice

89% stated that they had been able to apply the learning in practice and had seen results that were documents. The remaining 11 % said they had implemented things that were still in progress and the results were not yet seen. No one said that they had been unable to apply the learning. This indicates that the variety and presentation of the material is meeting the needs of learners. Some of the application of learning was about personal style which correlates with the findings on increased influence and engagement, whilst others were directly about having used the skill to change a piece of practice such as bringing in new appointment systems or changing staff rostering. The two are inextricably linked but the participants chose to state their application as relative either to a direct change in service of their won ability to make change happen though their leadership capability.

yes no in progres0

102030405060708090

100

9 Recommendations to others and things that could have been done differently.

EVALUATION REPORT

Most participants gave at least one thing they wished that had done differently, most gave two or more. This review has looked at two reasons from each assignment. The most popular thing people wish they had done/would recommend was around time management, either wishing they had put more time aside or recommending others to make regular scheduled time for learning. Although the learning is designed in small chunks, participants often stated that the content was very absorbing and they wanted to spend longer on the materials than they had time to do.

Although this is a MOOC, involving others in the learning was also key. The application in practice was linked to involving others, or at least letting teams know what they were doing. Many stated that they wished they had either communicated what their plans were or had directly involved their teams more.

Interestingly, not being put off, not being to proud to admit gaps in knowledge, being open minded and not being precious figured in may stories. The learning seem to expose a number of participant to their own lack of knowledge and their recommendation was about not being to proud to admit what you don’t know and ask for help if you need it. Trying things out, experimenting, not being put off all figured too.

make re

g time

take n

otes/download

involve

others

communicate

more

don’t be t

oo proud to

admit g

aps

don’t put it

off

practice

the skill

s

start s

mall

be open

minded

experi

ment

ask fo

r help

if yo

u need

make it

fun

keep on le

arning

don’t be p

ut off

stop bein

g prec

ious0

5

10

15

20

25

30

10 Plans to sustain the learningThis was a very individual response so it was not feasible to code and present the

EVALUATION REPORT

information graphically.

There were a whole range of plans for sustainability, which ranged from the learning already being embedded and sustained to continually reviewing how the team were performing together so that old practices did not creep back in. Many participants talked about working with an aspect of self awareness and the sustainability was about themselves,

I will continue to communicate more’

‘I will continue to ask the team for their opinions instead of just getting on with what I want to implement’

Many talked about how they had never reflected before and the sustainability for them was that they would continue to reflect so that their practice remained high quality. This as linked to a need to promote continuous improvement and recognition that change was possible for even those staff that viewed themselves as junior.

A number talked about overcoming fear due to the programme, the ability to challenge was not ‘career limiting’ and many good things had come from speaking up about wanting to change things. For some participants this had led directly to recognition form senior manages, increased respect from teams and increased responsibility.

Confidence had led participants to implement change even where there were vocal opponents and the results had been the catalyst for convincing senior colleagues of the benefits of change. This was seen especially when the change was related to long established ways of doing things like rotas, administration tasks and appointments.

10% of participants on the sample stated that the learning had really stimulated them to do a further programme.

11 Word count of passed assignments

EVALUATION REPORT

This was included purely to see the range of word submitted and whether length of the assignment was an issue. None of the assignments in the sample had failed at the first submission. The word count shows a normal distribution curve. In the assignments which were under 600 words n=8 the number of additional posts were also looked at. In all cases the participant has engage with the on-line discussions had has a range of additional comments on their timeline.

>400

401-500

501-600

601-700

701-800

801-900

901-1000

1001-1100

1101-1200

1201-1300

1301-1400

1401-1500<1

5000

5

10

15

20

25

30

35

40

45

Series1

5. Recommendations for the future and lessons learned

Due to the nature of the programme and the way that the assignment is constructed, it would be hard to undertake more in-depth analysis without taking significant time or using purposive sampling to look at the experiences of particular groups.

Given that the random sample seemed to throw up a skew in terms of gender and clinical/non clinical roles, it is recommended that this study is repeated inn one year.

Appendix 1

EVALUATION REPORT

Codes: Changes seen in self

Code No

Change seen

0 Not stated 1 Confidence increased2 Increased ability to influence3 Better able to engage others4 Better able to delegate5 Increased team-working6 Better communication7 Better listener8 Change in personal attitude/bias9 Increased trust10 Better working relationships11 Increased skill12 Better able to give to receive

feedback13 Leadership at all levels, not afraid

to lead in a junior position14 Better able to deal with complexity15 Enhanced decision making

Code No Able to implement

1 Yes2 No3 In-progress

Code No

Content

0 Not stated1 Berwick2 Host vs. Hero3 Francis4 Patient centred care5 HLM6 Johari Window7 Personal leadership Style

EVALUATION REPORT

8 Engagement and stakeholders9 Teams10 Adaptive leadership11 Emotional Intelligence

Code No

Word count

1 >4002 401-5003 501-6004 601-7005 701-8006 801-9007 901-10008 1001-11009 1101-120010 1201-130011 1301-140012 <1500

Code No

Gender

1 Male2 Female3 Trans

Code No Organisation type1 Acute/FT2 MH/ Community Trust3 Ambulance trust4 CCG5 Non NHS provider6 Primary CareCode Role1 Dr. Junior2 Dr. Mid level3 Dr. Consultant4 Nurse5 Ambulance/paramedic6 Pharmacist7 AHP8 HCS9 Administration10 Non clinical manager11 Student12 Non NHS manager

EVALUATION REPORT

13 Dentist14 Optician15 IT professional

Code Motivation for learning1 Lack of previous leadership development2 Feeling too junior to be a leader3 Encountering problems at work [operational]4 Encountering problems at work [personal leadership]5 Career development6 Newly promoted7 Suggested by manager8 Want to lead change9 Part of another programme10 Didn’t want to be like their manager11 Personal interest/learning12 Revalidation13 New qualified14 Stress and frustration