SO WE DID IT! DID IT WORK? HOW DO WE KNOW?A PRESENTATION ABOUT METHODS OF EVALUATION USED FOR IPL AT UEA

BY DR SUSANNE LINDQVIST 6TH JUNE 2016

“Brilliant! Really enjoyed it and learned lots – thank you!”

“The topic was not relevant. One member of our group did not do, or say , anything.

It was a complete waste of my time and I learned nothing.”

“I didn’t realise the value at the time, but now as I have qualified I appreciate

how valuable it was”

“My group did really well, but many of the others didn’t”

“Interprofessional education occurs when two or more professions

learn with, from and about each other in order to improve

collaboration and the quality of care”

CAIPE, 2002

INTERPROFESSIONAL EDUCATION

CYCLE OF IPL DEVELOPMENT

▪ IDEA

▪ INVOLVING STAKEHOLDERS

▪ DEVELOPMENT

▪ PILOT

▪ EVALUATION OF PILOT

▪ FURTHER DEVELOPMENT OF PILOT

▪ FULL SCALE INTERVENTION

▪ EVALUATION OF FULL SCALE INTERVENTION

▪ ONGOING DEVELOPMENT

▪ NEW IDEA!

INTERVENTION

PILOT

IDEA

TODAY’S PRESENTATION WILL FOCUS ON EVALUATION

▪ IDEA

▪ INVOLVING STAKEHOLDERS

▪ DEVELOPMENT

▪ PILOT

▪ EVALUATION

▪ DEVELOPMENT

▪ FULL SCALE INTERVENTION

▪ EVALUATION

▪ ONGOING DEVELOPMENT

▪ NEW IDEA!

BRIEF OVERVIEW OF PROFESSIONS INVOLVED IN PRE-REGISTRATION IPL AT UEAIPL at UEA involves ~2500 students from 12 different courses:

Operating Department Practice

Paramedic Science

Physiotherapy

Pharmacy

Medicine

Nursing

Occupational Therapy

Speech and Language Therapy

Midwifery

Social Work

Education

Clinical Psychology

BRIEF OVERVIEW OF IPL AT UEA

IPL1: Teamworking - how individual behaviours and communication impact on team effectiveness, performance and care delivery

IPL3: Learning and working with the service user - working together with service users to explore best collaborative practice

IPL2: Interprofessional Teamworking - what different professionals contribute to the care journey and how they work together to provide high quality care, that is person-centred, compassionate and safe

IPL4: Dealing with complex and sensitive issues - listening to people with lived experiences and to those who care for them

IPL5: Interprofessional Clinial Skills – practice working side-by-side to clarify roles and responsibilities

IPL1 and IPL2 are compulsory; IPL3 and IPL4 are voluntary, IPL5 is compulsory for some, but not others…

• Completes a ½ day shadowing

• Completes a ½ day shadowing

VISION

OVERALL AIM

LEARNING OBJECTIVESLEARNING OUTCOMES

IPL1 IPL2 IPL3 IPL4 IPL5

EVALUATION OF LEARNING OUTCOMES

Kirkpatrick levels of learning (learner’s reaction, modification in attitudes, knowledge and skills, changes in behaviour, changes in practice, benefit to patients/service users)

Kirkpatrick, 1996

Bloom’s taxonomy of learning (knowledge, comprehension, evaluation, analysis, synthesis, application)

Blooms Taxonomy: Bloom, 1956; Anderson et al., 2001

Millers pyramid (knows, knows how, shows how, does)

Millers Pyramid: Miller, 1990

Mapping LOs to existing learning theories

EVALUATION CONSIDERING THE 3P MODEL TO MAXIMISE OUTPUT

Presage(Input)

Process(IPL intervention)

Product(Output)

Biggs, 2002; Frye & Hemmer, 2012

NEXT ARE SOME EXAMPLES OF EVALUATIONS FROM SOME OF THE IPL LEVELS

FEEDBACK FORMS

▪ How well the intervention as a whole helped them achieve the learning outcomes

▪ What they specifically learned

▪ How useful different elements of the intervention were

▪ How supportive their facilitator was

▪ Ideas of how to further develop the intervention

We ask students to feed forward to us about:

IPL1 – HOW HELPFUL WAS IPL1 IN HELPING YOU ACHIEVE THE FOLLOWING LEARNING OUTCOMES?

Analysis of IPL1 Student Evaluation 2014/5 and 2015/6

2014/15 (respondents: 750) 2015/16 (respondents: 864)

Learning Outcomes Helpful/Very Helpful Not very or

at all helpful

Helpful/Very Helpful Not very or

at all helpful

Constructively discuss and

identify how different

behaviours and personality

styles can impact on a team’s

ability to function effectively

63% 16% 74% 10%

Recognise the effect of

Interprofessional teamwork

on integrated and high quality

care that is person-centred,

compassionate and safe

66% 15% 83% 4%

Recognise the effect of your

own professional role and

responsibilities and of

respecting those of other

professions when delivering

integrated care as part of an

Interprofessional team

51% 9% 78% 8%

1 %

3 %

13 %

43 %

40 %

Recognise the effect of interprofessional teamwork on the delivery of integrated and high quality care

Not at all helpful

Not very helpful

Neither helpful norunhelpful

Helpful

Very helpful

2 %

6 %

14 %

49 %

29 %

Recognise the importance of your own professional role and responsibilities and of respecting those of other professions when delivering integrated care

Not at all helpful

Not very helpful

Neither helpful norunhelpful

Helpful

Very helpful

2 %

8 %

16 %

53 %

21 %

Identify behaviours and personality styles that influence a team's ability to function effectively

Not at all helpful

Not very helpful

Neither helpful norunhelpfulHelpful

Very helpful

IPL1 – HOW HELPFUL WAS IPL1 IN HELPING YOU ACHIEVE THE FOLLOWING LEARNING OUTCOMES?

Analysis of IPL2 Student Evaluation 2014/5 and 2015/6

2014/15 (respondents: 419) 2015/16 (respondents: 539)

Learning Outcomes Helpful/Very Helpful Not very or at all helpful Helpful/Very Helpful Not very or at all helpful

Describe the role and responsibilities of your own and other professions in providing integrated and high quality care that is person-centred, compassionate and safe

85% 4% 77% 9%

Demonstrate awareness of common

challenges associated with

interprofessional teamworking and

approaches to overcoming these in

order to provide integrated care

85% 4% 82% 6%

Demonstrate some key teamwork skills

and behaviours that enhance care

81% 6% 76% 9%

Recognise the impact of

interprofessional teamworking on care

from the service user perspective

88% 5% 79% 8%

Appraise and constructively assess the

performance of your peers

72% 7% 70% 9%

IPL2 – HOW HELPFUL WAS IPL2 IN HELPING YOU ACHIEVE THE FOLLOWING LEARNING OUTCOMES?

Not at all helpful2 %

Not very helpful7 %

Neither helpful nor unhelpful

14 %

Helpful55 %

Very helpful22 %

Describe the role and responsibilities of your own and other professions in providing integrated and

high quality care that is person-centred, compassionate and safe

Not at all helpful2 %

Not very helpful4 %

Neither helpful nor unhelpful

12 %

Helpful57 %

Very helpful25 %

Demonstrate awareness of common challenges associated with interprofessional teamworking and approaches to overcoming these in order to provide

integrated care

IPL2 – HOW HELPFUL WAS IPL2 IN HELPING YOU ACHIEVE THE FOLLOWING LEARNING OUTCOMES?

IPL2Please tell what specifically you have learnt from IPL2 that was helpful to your development as a

future collaborative practitioner?

“… I thought I may face prejudice from some of the nurses and paramedics, like with my group last year.

However the group of individuals I was with this year were great, easy to communicate with and got the work

done. There was mutual understanding and respect amongst us all and no judgement. It was good to see that

the minority of hostile individuals I met last year aren't reflective of the majority of lovely future health care

professionals I've seen.”

“I found the workshops particularly helpful, especially when exploring other healthcare roles and engaging with

other students from different professions. On reflection I feel that the sessions made me more aware of the

importance of collaborative working. The sessions have opened my eyes to how things can go badly when

professionals do not communicate effectively and how this can impact on patient experiences and outcomes.”

“The idea of interprofessional working for the benefit of the service user is hindered by the attitudes of

individual professionals. IPL2 helped me develop ways in which to constructively deal with these individuals.”

“… As a student, it was disappointing to hear so much negativity directed towards my field. Hopefully the process will have taught other students that not all doctors are egotistical. It made me think about how I can try to overcome prejudices about doctors when working in a team.”

IPL2Please comment on the input from the Service User/s in your session

Hugh - Session A

“I really enjoyed having the service user, it gave a real life perspective on issues that arise with patient care and

made you think about how to avoid it. I thought having first hand accounts was really useful and hit home.”

“Found the service user to be very informative and encouraging.”

Hugh - Session C

“I found our service user more of a hindrance, I personally found him very judgmental and the feedback he gave

was not very reassuring or relevant to our courses if anything he made people feel embarrassed about their

presentation, their skills as a professional and/or appearance.”

“I appreciated hearing his perspective in the first session, however I think he misunderstood his role. In the

second session he gave feedback that was not really relevant and somewhat missed the point of the IPL session.

I think sometimes even the facilitator picked up that he was not very helpful and sometimes a bit

condescending.”

FEEDBACK IMPROVES FROM IPL1 TO IPL5…

▪ IPL3 and IPL4 is voluntary and involves more service user interaction ▪ IPL5 is focussing around clinical skills

So, do we need IPL1 and IPL2?...

ATTITUDES TO OWN AND OTHER HEALTHCARE PROFESSIONS

2002-2012

▪ Students arrive with views of their own professions and views of others

▪ Doctors are seen as the least “caring” and nurses the most “caring” profession

▪ Students who completed IPL1 in the past - when it was 7 weeks long - developed positive attitudes

▪ Students who did not complete IPL1 (acting control) showed no change or developed more negative views by simply spending 7 weeks in their course and no IPL

▪ Students’ views reverted back to baseline by the time they completed their course

2013 onwards…

▪ Today, students do not develop more positive attitudes after IPL1

▪ They do however appear to develop more positive attitudes after IPL2 now than before, and since we introduced IPL5 the positive shift in attitudes after IPL2 seems to be sustained in their last year.

So, do we need IPL1?...

STUDENT FACILITATORS

ACTING VERSUS REACTING…

INTERVENTION

PILOT

IDEA

▪ Always strive to develop from Good to Great.▪ Look ahead, but check behind.▪ Agree way forward.▪ Try make a difference, if not to all, to most.▪ Involve stakeholders – in further development and appraisal of progress.▪ The ownership is on everyone – the learner too.

SO DID IT WORK?

References alluded to in this presentation

Miller, G.E. The assessment of clinical skills/competence/performance. Acaddemic Medicine, 1990; 65: 563–67.

Anderson, L.W., Krathwohl, D.R., Airasian, P.W., Cruikshank, K.A., Mayer, R.E., Pintrich, P.R., Raths, J. & Wittrock, M.C. A Taxonomy for Learning, Teaching, and Assessing: A revision of Bloom's Taxonomy of Educational Objectives. 2001, New York: Pearson, Allyn & Bacon

Bloom, B.S. (Ed.). Engelhart, M.D., Furst, E.J., Hill, W.H., Krathwohl, D.R. Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain. 1956, New York: David McKay Co Inc

Biggs, J.B. Aligning the curriculum to promote good learning LTSN Generic Centre. 2002

Anderson, E., Smith, R., & Hammick, M. Evaluating an interprofessional education curriculum : A theory-informed approach. Medical Teacher, 2015; Early Online 1-10.

Frye, A. & Hemmer, P. Program evaluation models and related theories: AMEE Guide No. 67. Medical Teacher, 2012; 34:e288–e2299.

Kirkpatrick, D. Revisiting Kirkpatrick’s four level model. Training and Development, 1996; 1: 54–59.

Freeman S, Wright A & Lindqvist S. ‘Facilitator Training for educators involved in interprofessional learning’. Journal of

Interprofessional Care, 2010; 24(4):375-385.

CAIPE, Centre for the Advancement of Interprofessional Education – a definition. 2002 + Introducing InterprofessionalEducation, 2012 http://caipe.org.uk/

Hawkes G, Nunney I & Lindqvist S. Caring for attitudes as a means of caring for patients – improving medical, pharmacyand nursing students’ attitudes to each other's professions by engaging them in interprofessional learning. MedicalTeacher, 2013; Web paper e1-e.