revalidation.heiw.wales · Web viewThis Framework is for use by the Designated Bodies (DBs), and by the Wales Deanery as an appraisal provider, to review their own organisational

QMF v13: Agreed by WRDB for dissemination

All Wales Appraisal Quality Management Framework

PART 1: Background

1.1 Introduction

Effective Quality Management is crucial to the ongoing success of medical appraisal in Wales, and its contribution to other quality improvement and governance processes including revalidation. The recent publication Effective governance to support medical revalidation1states that:

‘…effective local governance and quality assurance of the systems supporting medical revalidation make an essential contribution to the improvement of safety and quality for patients’.

The Effective governance document has been incorporated in its entirety into the annual Designated Body Revalidation Progress Reports. This document focuses specifically on the medical appraisal systems which are in place to support medical revalidation, and describes a Quality Management Framework (‘the Framework’) for medical appraisal systems which applies across Wales. Agreeing a single framework is a significant step forward in ensuring appraisal is being delivered to a consistently high standard across Wales, and achieving its intended benefits for doctors, patients and organisations.

The document has been developed by a sub group of the Revalidation and Appraisal Implementation Group (RAIG). It was piloted in ABMU and in General Practice during October 2013 – January 2014. Version 10 of this paper was approved by the sub group and by RAIG in January 2014. Version 11 was circulated to all Designated Bodies for further consultation in February 2014. Version 12 incorporates all feedback and was approved by the WRDB in May 2014.

1.2 Aims ofthe Framework

This Framework is for use by the Designated Bodies (DBs), and by the Wales Deanery as an appraisal provider, to review their own organisational level arrangements for and governance of the appraisal process for all doctors. This review should enable DBs to evaluate the robustness of existing arrangements and assess the extent to which these are fit for purpose. It is envisaged that such review could dovetail with the work of Internal Audit teams and may provide evidence relevant to aspects of the independent review carried out by Healthcare Inspectorate Wales.

The Framework draws on and complements the agreed All Wales Appraisal Policy (‘the Policy’), and associated Operating Standards (‘the Standards’). Carrying out a review using the Framework enables DBs to assess compliance with the Policy and the Standards.

The Framework also informs and is informed by the Revalidation Progress Report which in turn incorporates the requirements of the Effective Governance document. Cross references are

1Effective governance to support medical revalidation: a handbook for boards and governing bodies published 2013: http://www.gmc-uk.org/GMC_revalidation_governance_handbook_51305205.pdf

Page 1 of 33

HIW

http://www.gmc-uk.org/GMC_revalidation_governance_handbook_51305205.pdf


provided to minimise duplication of effort. Together these form a suite of documents which help to establish, evaluate and improve the quality of medical appraisal in Wales and provide assurances to key individuals and groups as appropriate, including local Boards, the WRDB and the GMC. A template report is included at Appendix 1 to facilitate this and it is highly recommended that local Boards are provided with regular updates on Quality Management.

Although appraisal is a significant part of revalidation this Framework does not consider the quality management of the revalidation process as a whole, as this is being considered separately by the WRDB. Figure 1 illustrates how the Framework relates to these other processes.

1.3 Using the framework

Version 13 is being distributed to all Designated Bodies so that they can begin to work toward the standards it describes and integrate these into the ongoing delivery, management and evaluation of appraisal systems

All DBs will be required by the WRDB to use the Framework to carry out an annual review of the appraisal process by completing the table at Part 2. Timescales will be agreed at RAIG meetings but the first formal review is likely to be required in April 2015

o As there is some overlap with the Revalidation Progress Report, DBs should consider how data collected for one can be used to inform the other with minimum duplication of effort

o Following the review in April 2015 the Framework will be evaluated to ensure it remains fit for purpose

Key outcomes of the DB review against the Framework will be:

Page 2 of 33

Regulators eg HIW, Internal Audit, GMC

Revalidation Progress Report (RPR):

- Annual- DB level

- Focus on governance of systems which support

revalidation

Appraisal Quality Management Framework:

- Annual- DB level

- Focus on management and delivery of medical

appraisal

Annual National Quality Assurance of appraisal summaries

UK Quality Assurance of revalidation (under discussion)

Figure 1: The Framework in context

DB level reports and Quality Improvement

Action Plan

DB committees, board etc

WRDB, RAIG

National action plan


o An action plan identifying recommended quality improvementso A report in line with the template suggested at appendix 1 which could be shared

with relevant committees, internal audit, Board, RAIG, WRDBo Key themes will be reviewed by RAIG alongside the outcomes of the Revalidation

Progress Report to facilitate sharing of good practice, and to identify common areas of required quality improvement which could be progressed by the group

There will be national quality assurance processes, as described at appendix 7, which will be co-ordinating by RAIG and will supplement DB reviews, including:

o Annual national review of anonymised appraisal summaries and PDPs from all appraisal systems against set quality criteria, which were piloted at the 2014 event

o Periodic External Quality Assurance review (every 1-3 years)

1.4 Overview of the framework

The framework describes an ongoing process of quality management with quality standards and corresponding quality measures at its heart. Quality controls set out the elements of the system which should be in place to ensure those standards are met. Regular quality assurance reviews should be undertaken throughout the year, culminating in the annual return, to evaluate the extent to which quality controls are working and quality measures are being met. These regular reviews should inform action planning to ensure continuous quality improvement.

Suggested quality controls and quality assurance processes are provided at Appendix 2. These should assist DBs in introducing appropriate quality systems and in collating the information required to complete the self-assessment. The table at Part 2 includes a column for identified quality improvements which should form the basis for the DB’s Appraisal Quality Improvement Action Plan.

This cycle is illustrated at figure 2.

1.5 Sources of quality standards and measures

Page 3 of 33

Quality Control

Quality Assurance

Quality Improvement

Quality Standards and Measures

Figure 2: Quality Management cycle


The quality standards have been derived from a range of existing resources including the All Wales Appraisal Policy and Operating Standards, the recent publication Effective Governance to support medical revalidation, and documents published by the Revalidation Support Team (now Department of Health) in England. These are illustrated at figure 3 and key elements are cross referenced at Part 2 and / or included in appendices.

Quality standards are included to give guidance regarding what is expected in terms of performance. These have been developed by the RAIG working group and tested during the pilot. Monitoring how far each organisation is able to achieve the standards will enable benchmarking and year on year review of trends.

It is recognised that Quality Management of GP Appraisal is well established and that many of these standards are already in place, therefore high levels of compliance are expected. However the introduction of a Quality Management Framework is likely to be new for other medical appraisal systems and compliance with the standards may be variable until they are well established through systems, training and operation. While the aim of the QMF as a whole is to support the drive toward consistency and ongoing quality improvement it is recognised that this may be an incremental process and not all DBs will achieve the same levels of compliance in the early years. Development of a quality improvement action plan in all cases will provide assurances that quality is being considered and that improvements should be seen over time.

Page 4 of 33

RAIG

Appraisal PolicyAppraisal standards

Revalidation Progress Report

RSU

GP Quality Management

Framework and criteria

GMC

Good Medical PracticeFramework for

appraisal Suporting information

UKEffective Governance

for Medical Revalidation

Revalidation Support Team England (RST) standards (now DH)

Figure 3: Sources of quality standards


PART 2: Quality Standards: for completion by DBs

Criteria Quality standards Minimum Standard

Actual performance Improvement Plan

GP Other

INPUTS1. Appraiser capacity

There are sufficient trained, active appraisers to provide an annual appraisal to all relevant doctors within the DB

(Ref: Progress Report 2.3.4)

NB: MARS provides reports on 1.1 and 1.2 and maintains records of doctors whose 3 choices of appraiser have all been rejected

1.1 Actual number of appraisers

1.2 Appraiser – doctor ratio

1.3 Number of doctors who have reported difficulty accessing an appraiser in the last year

-

-

0%

2. Appraiser selection

Appraisers are selected for the role based on a process agreed by the DB which is in line with appendix 3

NB2.3: The RSU will incorporate this into a questionnaire for appraisers

2.1 The DB process is consistent with appendix 3 (please state exceptions to this and reasons for them)

2.2 There is evidence that appraisers have been selected in line with the process

2.3 Appraisers who respond to questioning confirm they are aware of the expectations of this role

85%

100%

100%

Page 5 of 33




GP Other

3. Initial Appraiser training

Appraisers have attended initial training which is consistent with that described in the Standards (reproduced at Appendix 4). This includes both training on MARS and appraisal skills.

(Ref: Operating Standards 4.4; Progress Report 2.3.2 and 5)

NB 3.2: RSU can provide details of those who have completed training at the Unit

3.1 All Active appraisers have completed initial appraisal skills training

3.2 All active appraisers have completed initial MARS training

100%

100%

4. Ongoing Appraiser Training

Appraisers have access to ongoing / update training at least annually after completion of initial training (plans for this are currently being considered by RAIG)

(Ref: Operating Standards 4.4; Progress Report 2.3.2 and 5)

4.1 Active appraisers have completed ongoing / update training at least annually (this includes local / regional and national training and online / distance learning)

75%

5. Support for appraisers

Arrangements are in place to enable

5.1There is evidence that arrangements are in place to provide ongoing support for

Yes / no

Page 6 of 33




GP Otherappraisers to access relevant support including:

Access an identified professional lead

Share / learn from (anonymised) complex appraisal issues and good practice

Access ongoing development of their appraisal skills

Access personal support when needed

See appendix 7 for suggested resource allocations to support appraisal

(Ref: Operating Standards 6.5; Progress Report 5)

NB 5.2: To be included in RSU questionnaire for appraisers

appraisers

5.2 Active appraisers who respond to questioning report satisfaction with the support they receive

75%

6. Communication and escalation channels

There are clear processes in place to enable appraisers to refer / escalate issues arising during an appraisal process which need to be considered outside

6.1 There is a process in place to enable appraisers to refer / escalate issues arising during an appraisal process which need to be considered outside appraisal

Yes / no

Page 7 of 33




GP Otherappraisal (including issues relevant to PDP, relating to job planning, clinical governance etc)

NB: 6.1 to be included in RSU questionnaire for appraisersExceptions management protocol being developed by RAIG describes escalation routes

6.2 Appraisers report that they are familiar with the processes for referring / escalating issues which arise during the appraisal process but need to be considered / managed outside appraisal

100%

7. Feedback for appraisers

Feedback is sought from doctors after their appraisal, collated independently of the appraiser and shared with them to inform their own development. (This facility will be available for all appraisers through MARS from 1/4/14 and feedback will be collated, but will need to be delivered by the DB)

(Ref: Operating Standards 6.5; Progress Report 5)

NB 7.1 MARS provides feedback reports on appraisers to a nominated appraisal lead

7.1 Appraisers receive feedback on their performance at least annually based on feedback from doctors

7.2 Overall doctor feedback received regarding the appraiser is positive (rated as good or very good)

100%

90%

8. Management of appraiser 8.1 Active appraisers complete 80%

Page 8 of 33




GP Otherperformance in this role

Appraiser performance in relation to the expectations of this role (as defined in appendix 3) is reviewed at least annually, which might be as part of one appraisal covering whole practice, and development needs are managed appropriately. (Appraiser performance reports relating to key areas eg timeliness of responses to requests, number of appraisals completed, timeliness of writing up appraisal summary will be available through MARS but will need to be managed by the DB)

NB 8.1 MARS provides reports on timescales within which summaries are completed

the appraisal summary within the specified timeframe (14 days recommended)

8.2 Appraisers work within an appraisal management structure (see appendix 7 for recommendations)

8.3 Appraisers have had an annual performance review relevant to this role which meets the needs of Whole Practice Appraisal

100%

100%

9. Presentation of supporting Information for revalidation

The appraisal system assists doctors in presenting the supporting information required for revalidation in line with the

9.1 All doctors use MARS to complete their annual appraisal

100%

Page 9 of 33




GP OtherGMC’s advice and Good Medical Practice ie CPD, Quality Improvement, Significant Events, Patient and Colleague Feedback, Complaints (MARS provides advice on domains and revalidation templates)

NB 9.1 MARS provides live access reports on engagement 10. Doctor access to information

relating to their own clinical practice

There are processes in place to enable doctors to readily access accurate relevant information relating to their own clinical practice including in particular information on complaints / significant events / untoward incidents and relevant performance data including for example information on prescribing, record keeping, discharge information sharing, mortality, engagement with clinical governance systems including CGPSAT

NB 10.1 to be included in doctor

10.1 Doctors report satisfaction with the availability of relevant information relating to their own individual clinical practice

85%

Page 10 of 33




GP Otherfeedback questionnaire

11. Whole practice appraisal

The agreed whole practice appraisal policy is publicised to doctors and appraisers. This document is utilised by doctors in preparing their evidence.

11.1 When undertaken, QA reviews demonstrate that in all cases where doctors do not represent their whole practice this is noted in the summary or PDP by the Appraiser

100%

12. Validating supporting information

There are clear processes to ensure that the appraiser validates supporting information for revalidation appropriately, and document this in a way which enables the RO to make a meaningful revalidation recommendation

NB 12.1 the agreed guidance on the quality of appraisal supporting information is available on the revalidation wales website

12.1 All appraisers have access to advice and guidance on how to validate supporting information for revalidation and how to document this

12.2 When undertaken, QA reviews demonstrate that validation decisions taken by appraisers are documented in a way which enables the RO to make a meaningful revalidation recommendation

Yes / no

90%

PROCESS

Page 11 of 33




GP Other13. Resources to support appraisal

Appropriate resources are available within the DB to manage and support the appraisal process, including support for MARS, time for doctors and appraisers to undertake appraisal and support for doctors who fail to complete the annual appraisal

See appendix 7 for suggested resource allocations

(Ref: Operating Standards 4.2, Progress Report 5)

NB 13.1: MARS provides live access reports on registrations

13.3 to be included in RSU appraiser questionnaire

13.4 to be included in doctor feedback questionnaire

13.1 Doctors with a prescribed connection requiring an appraisal within the DB are registered with MARS

13.2 Active appraisers are registered with MARS

13.3 Appraisers who respond to questioning report that they have adequate time in which to undertake all aspects of the appraiser role

13.4 Doctors report that they have adequate time in which to participate fully in the appraisal process

13.5 There is a described management / support structure within the DB which enables all requirements of this framework to be met

90%

100%

90%

90%

Yes / No

14. Managing engagement with 14.1 Doctors with whom there 95%

Page 12 of 33




GP Otherappraisal

There are systems in place to manage and monitor engagement with appraisal. These systems include:

a process for enabling ROs to agree and document when doctors have exceptional circumstances which mean they will not undertake their annual appraisal

a policy which describes the actions to be taken to manage cases of non-engagement with the appraisal process, including escalating non-engagement cases at all stages within the revalidation cycle

(Standards 2.1; revalidation progress report 2.1)

NB 14.1: MARS provides live reports on appraisal completion

14.2, 14.3, 14.4, 14.5: The RSU captures

is a prescribed connection, who have not documented extenuating circumstances, have completed an appraisal within the last year

14.2 There is regular analysis of appraisal engagement figures (available in MARS reports) reconciled against complete lists of all doctors due to have an appraisal in that quarter

14.3 All exceptions, and reasons for these, are documented

14.4 There is an annual analysis of numbers of and reasons for exceptions including documenting whether or not these have been approved

14.5 Doctors identified as not

Yes / No

100%

100%

100%

Page 13 of 33




GP Otherand reviews these for GPs in line with the exceptions management protocol

engaging with the appraisal process have been managed in line with the agreed policy

15. Managing the Appraiser – Appraisee match

There is a system for managing the Appraiser – Appraisee match, which makes provision for:

every doctor can select or be matched with one or more appropriate appraisers (NB MARS manages this process)

where possible, a minimum of two and preferably 3 appraisers are used within each 5 year revalidation cycle (NB MARS manages this)

ensuring that reciprocal appraisals do not take place

managing conflicts of interest, disputes over the processes or disputes / appeals relating to the outcomes of appraisal (eg these are appropriately covered under

15.1 There is an appropriate system in place for managing the Appraiser – Appraisee match

15.2 Proportion of doctors expressing satisfaction with selection of appraiser

15.3 There is an appropriate complaints / grievance procedure in place

15.4 When conflicts of interest, disputes or appeals arise these are resolved in accordance with the agreed process

Yes/No

95%

Yes / No

100%

Page 14 of 33




GP Othera complaints / grievance procedure which includes independent scrutiny / lay involvement as appropriate)

(Ref: Operating Standards 4.2, 7.2)

NB 15.1: documented in the all Wales appraisal policy and managed via MARS

15.3 to be considered by RAIG

16. Appraisal data management

The DB has processes for appraisal data access and management which are consistent with the MARS terms and conditions of use and the Data Protection Act

16.1 All access to appraisal data is consistent with the MARS terms and conditions of use and the Data Protection Act

100%

OUTPUTS17. Internal quality assurance of

appraisal outputs

There is regular review of the quality of

17.1 There has been internal review of at least one appraisal summary and PDP within the year for each active appraiser

Yes / No

Page 15 of 33




GP OtherPDPs and appraisal summaries both within the DB and as part of annual quality assurance review. Quality assurance reviews are used to inform appraiser training and improvements to the appraisal process (see appendix 7)

NB 17.2: MARS monitor timescales and issues automatic reminders

17.2 Appraisal summaries and PDPs are agreed within 28 days of the appraisal meeting

17.3 At least 1% of appraisal summaries and PDPs have been made available for the national quality assurance event, where they are reviewed in an anonymised format

85%

Yes/No

18. External quality assurance of appraisal outputs

The DB has participated in the programme of external quality assurance of appraisal outputs agreed by RAIG, and / or in another equivalent form of external quality assurance of appraisal outputs

18.1 Participation in agreed external quality assurance as recommended by or equivalent to that recommended by RAIG

Yes / No

Page 16 of 33


Appendix 1: Template report for Board, committees, HIW etc

It is recommended that this report is used as appropriate by Designated Bodies to keep relevant committees and boards up to date in terms of appraisal activity. The template may be adapted to suit organisational purposes. On an annual basis, following the annual QMF / RPR rounds, WRDB will expect to be copied into a report from each DB.

Medical Appraisal Report

Report to:Written by:Reporting period:Purpose of the report: To provide assurances that arrangements for and governance of

appraisal systems for doctors are: Robust Fit for purpose Operating effectively Compliant with the All Wales Appraisal Policy and

Operating Standards Supporting the revalidation process appropriately

Link to strategic objectives / current plan:Recommendations:

1. Executive Summary

2. Purpose

3. Background

4. Management and governance arrangements

5. Activity levels during the reporting perioda. Appraisal and revalidation activity

(Including number of appraisals completed, number of appraisers trained, numbers of revalidation recommendations – this is included in the annual Revalidation Progress Report)

b. Quality improvement activity(including activity relating to previous action plans, national developments and matters arising from RAIG / WRDB)

Page 17 of 33


6. Highlights of quality assurance reviews:a. Revalidation Progress Report

b. Quality Management Framework

c. Other (internal audit, HIW etc)

7. Action plan arising from quality assurance reviewsa. Action plan for the organisation

b. Action plan being progressed at an all Wales level by the Revalidation and Appraisal Implementation Group (RAIG)

8. Recommendations

Page 18 of 33


Appendix 2: Suggested quality controls and quality assurance reviews

Criteria Quality controls Quality assurance reviewsAre appropriate quality controls in place?

Are controls built into the system to ensure standards are achieved?

Do appropriate quality assurance reviews take place on an ongoing basis?

Is evidence available from appropriate quality assurance reviews to support the self assessment?

1. Appraiser selection

Policy stating minimum and maximum number of appraisals for appraisers

Description of selection process Job description and person specification

Selection data (including number of doctors considered for appraiser roles, number selected, background of appraisers)

Whether the number of appraisals each appraiser is expected to do complies with the defined range

How many appraisers have job descriptions

2. Appraiser training

Details of how initial training provided (appraiser skills and MARS) meets the standards Example programme / resources for refresher training

NB RAIG is currently developing proposals for national refresher training for secondary care appraisers

Review of the number of active appraisers who have completed initial skills training which meets the standards

Review of the number of active appraisers who have completed initial MARS training which meets the standards

Review of the number of active appraisers who have completed refresher training a year after

initial training Review of the evaluations of training

3. Support for appraisers

Description of support arrangements including written protocols on escalation

Audit of escalation of issues to determine whether these were managed in line with the agreed protocol

Records of meetings, online forums etc Records of ongoing development of appraiser skills Appraiser views on the support they receive

Page 19 of 33


Criteria Quality controls Quality assurance reviews4. Performance

management of appraisers

There is a standard appraisee feedback form (see appendix 5 for the current MARS feedback form) and a description of how this feedback is used, including when an early review of performance would be triggered based on negative feedback

Statement of specified objectives / key indicators for appraisers

Description of process for monitoring and managing appraiser performance including processes for undertaking regular performance review

Review of doctor feedback Regular review of appraiser achievement of

objectives / key indicators including egquotas, time for completing appraisal summary

Examples of appraiser performance reviews

5. Information collected for appraisal

Evidence of how the appraisal system assists doctors eg templates, guidance etc

NB The MARS templates are described in the appraisee user guide

Description of processes to enable doctors to access relevant information

Description of processes by which appraisers validate information

Outline of training provided to appraisers regarding validating information

Publication and advertisement of whole practice appraisal policy

Regular review of the whole practice appraisal policy and how it is publicised eg intranet, doctors directed to Revalidation Wales Website

Evidence from QA reviews that the whole practice appraisal policy is adhered to

Evidence from QA reviews that appraisal information is validated and documented appropriately

6. Resources to support appraisal

Support and guidance materials made available to doctors regarding the use of MARS

Description of time made available for appraisal and how this is documented (eg contract, job plan etc)

Description of structures / resources available to manage and support the appraisal process

Regular review of MARS usage figures Review of time taken to undertake appraisals

Page 20 of 33


Criteria Quality controls Quality assurance reviews

7. Managing engagement with appraisal

Description of communication and support arrangements for doctors, including for those who fail to complete the annual appraisal

Documented processes for managing exceptions Documented processes for managing non

engagement and for escalating to the GMC, including definitions of non engagement with appraisal / revalidation

Regular(recommended quarterly) review of engagement and actions taken as an outcome

Regular (at least annually) review of exceptions

8. Managing the appraisal process

Description of process for matching doctors and appraisers, and of controls relating to minimum number of appraisers within the cycle and reciprocal appraisals

Dispute / appeal process / policy with appropriate representation including independent / lay involvement as appropriate

Guidance on indemnity

Review of how many appraiser – appraisee matches meet the criteria

Doctor satisfaction with appraiser match Audit of disputes / appeals

9. Appraisal governance

A policy / protocol describing the links and information flows between appraisal and other governance processes

Description of how the policy is publicised

Review / audit of the application of the policy

10. Internal quality assurance

Description of internal review process including how the outcomes inform training and process improvements and how key themes are reported to the board

NB A suggested review form is included at appendix 6 Description of national quality assurance exercise

included at Appendix 7

Review of outcomes of internal quality assurance activities

Organisational action plan based on the outcomes of the national quality assurance event with reference to how outcomes compare with the all Wales data and reported to the board

Page 21 of 33


Criteria Quality controls Quality assurance reviews11. External quality

assuranceDescription of proposals for external quality assurance included at Appendix 8

Organisational action plan based on the outcomes of the external quality assurance review and reported to the Board

Page 22 of 33


Appendix 3: Suggested expectations of appraisers

Selection

The DB has a fair process for selecting appraisers which includes consideration of:

Who is eligible for selection as an appraiser (it is recommended that doctors from a wide range of backgrounds are eligible including eg locums, salaried GPs, staff and associate speciality doctors)

How potential appraisers are identified (eg advertising / application, self or other nomination, review of existing appraisers)

How suitable appraiser will be identified (eg how their suitability for the role will be considered)

Whether confirmation of appointment is dependent on completion of initial MARS and skills training

Job description / outline

There is a job description for the appraiser role which includes:

Key accountabilities in this role ie who they report to, who provides them with support, who they escalate issues to

Purpose of role Key responsibilities (eg to respond to doctor requests and agree meeting dates; to review

appraisal materials and prepare for the appraisal meeting; to facilitate the appraisal meeting; to validate and feedback on evidence for revalidation; to write up the appraisal summary and PDP)

Guidance on indemnity arrangements for appraisers

Person specification

There is a description of the core competencies expected of appraisers, which may include:

Medical qualification, GMC license to practise, inclusion on MPL if appropriate Has participated in a minimum of two annual appraisals Commitment to the principles of appraisal Understand the principles of revalidation Interpersonal skills, ability to facilitate the appraisal discussion Ability to manage time and work in order to achieve the performance expectations of the

role Good written communication Ability to evaluate revalidation evidence against the GMC’s requirements Commitment to personal and professional development

Performance expectations

Performance expectations might include:

How many appraisals the appraiser is expected to undertake within the year

Page 23 of 33


How quickly the appraiser is expected to respond to requests (recommend 3-5 days) Acceptable reasons for refusing requests How quickly the appraiser is expected to write up the appraisal summary (recommend 14

days) The extent to which the appraiser is expected to meet the quality criteria specified for

appraisal summaries Any meetings the appraiser is expected to attend

Best practice

Best practice has been described by the Revalidation Support Team (England) in their document ‘Quality Assurance of Medical Appraisers’ (March 2012) which can be accessed at:

http://www.revalidationsupport.nhs.uk/CubeCore/.uploads/documents/pdf/rst_quality_assurance_medical_appraisers_2013.pdf

Page 24 of 33




Appendix 4: Criteria for appraiser training

Appraisal Policy Operating Standards section 4.4

Designated bodies to ensure training programmes provided for new appraisers and refresher training for all appraisers on at least an annual basis. Designated body’s database of appraisers should be able to demonstrate training status

ORSA specifies that: The initial training for medical appraisers should cover the competencies and skills required for the organisation’s appraisal process but to inform revalidation should include:

Understanding of the purpose of appraisal and revalidation and the links between these processes and other systems for improving the quality of medical practice in the organisation and the wider healthcare system

Competency in assessing supporting information that informs the appraisal and revalidation process, speciality aspects of appraisal

Skills to conduct an effective appraisal discussion, including all the elements needed for revalidation

Ability to produce consistently high quality appraisal documentation, sufficient to inform the revalidation recommendation as well as inform personal development

Additionally, training will meet the criteria specified in AQMAR and any subsequent iteration of that document

In Wales, revalidation-ready training should cover: Wales Medical Appraisal Policy in particular ethos of appraisal in Wales, integration

with other quality improvement and patient safety processes, principles of delivery Wales Whole Practice Appraisal policy Wales Quality Indicators of Supporting Information policy Concept of agreement at appraisal – and processes for resolving disputes Quality criteria for appraisal summary and PDP For those using MARS, managing the above through MARS

Page 25 of 33


Appendix 5: MARS appraisee feedback form – Launched 1 April 2014

Doctor feedback questionnaire from QMF v10 updated at AC management Feb 2014

NB this is a word representation of a survey which is included in the online appraisal system

Survey Scoring Key:

Please score the following questions based on 1 being negative/lowest score, and 5 being positive/highest score. If any questions are non-applicable, please leave them blank.

1. My Appraiser’s Skills (rated 1 low – 5 high)

(a) My Appraiser’s preparation for the appraisal was(b) My Appraiser’s ability to listen to me was(c) My Appraiser was supportive(d) My Appraiser challenged me about how my learning changed what I do(e) My appraiser challenged me to tell them how my learning benefitted my patients (f) My Appraiser’s feedback was constructive and helpful(g) My Appraiser helped me think about new areas for development(h) My Appraiser reviewed progress against last year’s PDP(i) My documented PDP reflects my main priorities for development (j) My summary is a true and accurate reflection of the appraisal meeting(k) My summary is of a professional standard(l) Overall rating of my Appraiser and their skill in conducting my appraisal

2. The following questions refer to the supporting information (CPD, complaints, colleague and patient feedback, quality improvement ie audit / case review, Significant Event) required for revalidation:

(a) I was able to collect the supporting information required for revalidation this year (Yes / no)

If you answered NO, please outline the reasons below, referencing the type of supporting information and why you were not able to collect it:

(b) The supporting information required for revalidation was discussed and recorded appropriately (Yes / no)

If you answered NO, please outline the reasons below, referencing the type of supporting information and why you do not feel this was discussed / recorded appropriately:

(c) My appraisal summary provides me with an overview of my progress toward revalidation (Yes / no)

If you answered NO, please outline the reasons below:

(d) My PDP describes what else I need to do for revalidation (Yes / No / N/A)

Page 26 of 33


(e) I had opportunity to discuss all of my professional roles (whole practice appraisal)

3. The Administration of Appraisal

(a) The Appraisal Website, (the Medical Appraisal Revalidation System – MARS), was easy to use (1-5)

(b) I was able to select an appropriate appraiser (Yes / no)

If you answered NO, please outline the reasons below, referencing the difficulties you had with selecting an appropriate appraiser:

(c) If you have accessed guidance on any aspect of the appraisal process (eg appraisal, revalidation, MARS), please indicate below how helpful this was (1 – low 5 – high):

Guidance from the Revalidation Support Unit Guidance from the Health Board/Designated Body Guidance from the Appraiser Other (please specify)

Free Text Questions

1. In what ways has the appraisal process benefitted you?2. Please include any suggestions for improvements which could be made to the process. For

example, to the online Appraisal System (MARS), support and guidance available, appraiser skills, the appraisal process etc

3. Do you have any general comments?

Page 27 of 33


Appendix 6: example form for ongoing internal review of appraisal summaries and PDPs

Appraisal lead to Appraiser Feedback

Appraiser Date

Appraisal Lead (AL)

Doctor

AL recommendations from last feedback

(date)

Context entry2

Completed Sufficient

Summarisation in column one

sub-headings grouping use an example avoid exaggeration maximum of 4 entries

per domain?

Documentation in column two

Lists(Unless already in C1)

Dates Or use ‘none seen’

Probe, Challenge and ‘value added’ in

column three

New information Background

2 Currently only available in GP

Page 28 of 33


Goal identified Why done Learning points Effect on patient care Measurements APs identified

Insights &Reflection

Explored GP ideas & plans

Transfer of info to next appraiser

Revalidation

Revalidation date Dr progress against 6

strands WPA status Action needed

Constraints

Appropriate challenge

Avoid collusion

Last year’s PDP

completed or reasons why not

New PDP

‘audit trail’ End point described SMART

Professional Style

(including spelling, verbosity, judgements )

Page 29 of 33


Appraisals completed

(this contract year)

Write-up within 14 days

Number booked and Dates fixed

(for future appraisals)

Outstanding summaries

Requests rejected Year 2012 2013

Reject

Attended meetings3

(over the last year)

All Wales Training:

Spring meeting:

Autumn meeting:

Winter meeting:

Feedback from doctors

AL Comments and recommendations for development

Appraiser Comments

3 References are to GP training events

Page 30 of 33


Appendix 7: Suggested resource allocations

Appraisers:

A medical / professional role Role outline and performance expectations are described at appendix 3 Recognised time in which to:

o Prepare for each appraisalo Carry out the appraisal discussiono Write the appraisal summaryo Attend any required training and other relevant meetings / events as appropriate eg

performance reviews, QA For GP appraisers this equates to approximately 9 hours per appraisal which includes up to

one hour each way travelling time

Appraisal co-ordinator role:

A medical / professional role Act as line manager to appraisers, which may include:

o Provide ongoing support and developmento Respond to queries and issues which have been escalated by appraiserso Receive and feedback reports on performance and doctor feedbacko Undertake periodic reviews of appraisal summaries and PDPs

A regular sessional commitment is recommended. For GP Appraisal Co-ordinators this is 3 sessions a week for which they:

o Manage a team of 10-15 appraisers delivering 250 – 450 appraisals per yearo Facilitate quarterly meetings of their team at which they provide support and

development activitieso Regularly review appraisal summaries and PDPs and provide feedback to their

appraiserso Undertake probationary reviews for new appraisers and annual appraisals for all

established appraiserso Deal with all escalated issues raised by appraisers or DBso Contribute to appraiser recruitment, training and QAo Meet quarterly with the DBs to monitor GP compliance with appraisal and

engagement with AQs

Professional lead role:

A medical / professional role which is ideally outside the clinical governance structure Act as overall reference point for Appraisal Co-ordinators, dealing with escalated issues as

required Contribute to RAIG meetings, policy development etc

Management lead role:

A management role with a specific responsibility for appraisal / revalidation

Page 31 of 33


Appendix 8: Description of national quality assurance exercise

1. The aims of the national quality assurance exercise are to:

Review a sample of anonymised appraisal summaries and PDPs against set quality criteria To benchmark the quality of appraisal summaries in GP and other medical appraisal systems To identify themes and trends, in particular areas where further training and / or guidance

may be required To identify improvements which could be made to MARS and / or the appraisal process to

enhance the quality of appraisal summaries and PDPs To facilitate sharing of good practice and networking between appraisers To provide assurances to delegates and to others regarding the quality of appraisal

summaries and PDPs and the governance of the appraisal process

2. The process for the national quality assurance exercise is:

Designated Bodies will be responsible for identifying appropriate delegates and funding any travel and subsistence claims for their delegates

The Deanery will:o Make all administrative arrangements for the event o Identify delegates relevant to GP appraisal and fund their travel and subsistence

claimso Invite external delegates as appropriateo Access a random sample of approximately 5% of appraisal summaries and PDPs for

all participating organisationso Anonymised the summaries and PDPs and provide copies of these along with the

agreed quality criteria at the evento Lead a calibration exercise using one example summary to ensure all delegates have

a good understanding of the quality criteria and how to apply theseo Undertake analysis of the outcomes of the event and provide reports demonstrating

compliance with the criteria at DB and All Wales levelo Produce a national Quality Assurance report detailing the outcomes and collating

recommendations arising and how these will be progressed Delegates will be expected to participate in the calibration exercise and then to work in pairs

to review a small number of appraisal summaries and PDPs against the specified criteria

Page 32 of 33


Appendix 8: External quality assurance proposals

Developing an annual programme

It is proposed that a programme of external quality assurance review is developed on an annual basis by RAIG and agreed by WRDB.

The frequency of reviews will be recommended on the basis of the outcomes of internal quality management activity.

The programme will be developed based on identified need and will include:

Review of anonymised appraisal summaries and PDPs Audit of internal quality management activities eg completed Revalidation Progress Report,

internal review against Quality Management Framework Independent review of management and governance arrangements for the appraisal

process Focused review of appraiser training, support and performance management arrangements Other reviews identified as appropriate

External quality assurance review will take a number of forms, including:

Representatives outside the Deanery / DBs invited to participate in the national quality assurance review outlined at appendix 6

Appropriate individuals / organisations from outside the Deanery / DBs invited to audit relevant quality management activity

Independent review of specific aspects of the appraisal process by an agreed individual / organisation egAudit and Assurance Wales / Wales Audit Office, CHC, RST, NES etc

Logisitics

The programme will be developed at RAIG and agreed by WRDB.

The Deanery will co-ordinate External Quality Assurance activity in partnership with the DBs.

External Quality Assurance for 2014

In 2014 External Quality Assurance has been achieved through inviting a range of external representatives to attend the inaugural integrated National Quality Assurance Event. Feedback from these external representatives has been sought to inform the external review recommendations.

Page 33 of 33

Documents

revalidation.heiw.wales · Web viewThis Framework is for use by the Designated Bodies (DBs), and by the Wales Deanery as an appraisal provider, to review their own organisational