Do the new formative work based placed assessments improve

educational impact?Dr Hermione Race, Dr David James, Dr Caroline Fertleman

London School of Paediatrics Trainees Committee

Outline

• Implementation of two new work-based assessments nationally

• Why• How• Results of a survey assessing impact of new

WBA’s• Further developments

Introduction

• The usefulness of an assessment is a product of its reliability, validity, cost-effectiveness, acceptability and educational impact

• Traditional approaches to maximise the reliability or reproducibility of assessments can have a negative educational impact on the learner by reducing the opportunity for meaningful developmental feedback.

CbD

SAIL

Methods

New Safeguarding CbD

New DOC

Survey Design• Assessment subgroup of the trainees committee

designed survey on experience of new assessments• Had the new assessments improved educational

experience for both trainees and tutors?

• Sent to London School of Paediatric trainees and to college tutors

• June after ARCP• Responses from 161 trainees, evenly distributed

throughout training years• 35 tutor responses

Survey Results - CbD

• Majority of trainees completed one Safeguarding CbDs

• 15- 20 minutes to complete• 69% were completed with the trainer present

Survey Results - CbD

Compared with your previous experience of the old CbD, do you feel that the safeguarding CbD better allows the opportunity to:

– Assess your safeguarding knowledge, application and clinical skills?

– stimulate your learning more?– create clear and achievable learning objectives?

Survey Results – CbD

p = <0.05 p=<0.05

Trainees

Survey results - CbDTutors

p = <0.05 p = <0.05 p = <0.05

Survey Results - CbD

What are the barriers to completing a Safeguarding CbD?

Survey Results - CbD

Describe what you like and dislike about the safeguarding CbD.

Like Dislike

Like specific safeguarding CbD Assessments are still a tick box exercise

Better structure Only as good as the assessor

Opportunity to learn and get feedback

The need for reflection

Encourages trainers to make comment

Finding time with a consultant

Only suitable for certain types of placement

Survey Results - DOC

• Majority had performed 5 DOCs, taking 5-10 minutes each

• Only 49% performed in person with the trainer

Survey Results - DOC

Compared with your previous experience of SAIL, did you feel that performing the DOC better allowed you the opportunity to assess:

– Written correspondence?– Stimulate learning?– Create clear and achievable learning objectives?

Survey Results - DOCTrainees

Survey Results – DOCTutors

p = <0.05 p = <0.05 p = <0.05

Survey results - DOC

What are the barriers to completing a DOC?

Survey Results - DOC

Describe what you like and dislike about the DOC?

Like Dislike

Better Structure Difficulty in finding a consultant

Better opportunity to learn and get feedback

Less relevant for more senior trainees

Online better than paper for eportfolio purposes

Still not broad enough

Straightforward

Conclusions

• New Safeguarding CbD format favoured by trainers and trainees

• New DOC favoured by trainers but not trainees

• Trainees felt their learning was not stimulated more by the new assessments

• It is a step in the right direction

The future

• Need to change the culture of work based placed assessments in paediatrics– Possibly having more assessments

• An app to allow them to truly be supervised learning events done at the time of assessment

• Tailoring of assessments to GRID trainees and more senior trainees

Any questions?