Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Competence By Design (CBD)
Assessment and Evaluation
Raja Rajamohan
Assessment
• Focuses on learning, teaching and outcomes
• Interactive process - informs faculty how well their students are learning what they are teaching
• Used to make changes in the learning environment and improve students learning and study habits
• This information is learner-centered, course based, and not graded
Evaluation
• Evaluation is a systematic and objective review of an ongoing or completed project or programme
• The aim is to determine the relevance, effectiveness and fulfilment of objectives
Angelo, T and Cross, K.P. 1993. Classroom assessment techniques a handbook for college teachers. Jossey-Bass A Wiley Imprint, San Francisco, CA. Pp 427.
Assessment Evaluation
Formative: ongoing, to improve learning
Summative: final, to gauge quality
Process-oriented: how learning is going
Product-oriented: what’s been learned
Diagnostic: identify areas for improvement
Judgmental: arrive at an overall grade/score
How are we doing this now?
• Assessment- Daily - staff
• Evaluations- Monthly/ End of rotation - site coordinator- Bi-annual – Evaluation committee/Program Director- Exams / OSCE / Simulation
RCPSC – End of 5th yearABA – PGY 2 – 5 every yearAKT - 3rd, 4th & beginning of 5th
Basic sciences exam - 2nd yrOral exams - 2x / year starting in 2nd year
Daily assessment forms
• Knowledge• History taking and physical examination skills• Technical skills• Communication skills • Clinical reasoning skills• Health promotion• Teaching• Collaboration • Comments section – Strengths / Areas for
improvement
A very good day
Problems with existing system
• Lacks validity - Assess a restricted range of competencies
• Have limited inter-rater and intra-rater reliability - Do not discriminate between residents or performances (i.e., they lack reproducibility)
• Rater errors - distribution such as the leniency/severity error (doves/hawks) - central tendency error (failure to use the entire rating scale)- correlation (halo effect) - influenced by prior impression
• Even when practice behaviors are observed on a daily basis, they frequently are not documented or not done in a timely fashion
Improving In-Training Evaluation Programs.J Gen Intern Med. 1998 May; 13(5): 317–323.Jeffrey Turnbull, Jean Gray, JohnMacFadyen
The Need for CBD
• Evidence that our existing work-place based assessment tools, such as ITERS, are inefficient and ineffective
• The existing tools focus on documenting whether or not a resident was present, rather than assessing skill level or guiding future learning
The Need for CBD
• Assessment tools promote inefficient, checklist-style feedback
• There is concern about the quality and quantity of direct observation/assessment of a resident’s abilities
• Supervisors are often unsure of how/what to assess and as a result provide ambiguous feedback such as “very good” or “needs more work”
• Lack of clear learning paths allows some supervisors to teach based on their own interests – rather than focus on the curriculum and the needs of the trainee
The Need for CBD
• There is increased culture of “failure to fail”
• The onus rests on the teacher/assessor to demonstrate that a trainee is struggling or falling behind
• It can be difficult to gather this evidence and residents (who are not ready) can be passed from one rotation to the next
The Need for CBD
• Residents are often unclear of the learning objectives of their program
• They’re not sure what they need to learn, when they need to learn or how they will be assessed and what they will be assessed on
• Residents spend much of their final year preparing for exams, instead of refining valuable clinical skills and gaining the confidence to practice independently
Competence By Design (CBD)
• Outcomes-based approach to the design, implementation, assessment and evaluation using an organizing framework of competencies
• Curriculum is organized around the outcomes expected of a resident and advancement is dependent on having achieved those expected outcomes
Competence By Design (CBD)
CBD organizes residency training into four developmental stages and clearly lays out markers for teaching and learning at each stage
Each learning experience, focuses on the identified outcomes for that stage
EPA
An Entrustable Professional Activity is a key task of a discipline(i.e. specialty or subspecialty) that an individual can be trustedto perform in a given health care context, once sufficient competence has been demonstrated.
EPA
• EPAs integrate multiple milestones from various CanMEDS Roles
• Some people think of an EPA as a basket or a bundle that holds numerous milestones
• RCEPAs allow for authentic, work-based assessment that is targeted at the daily tasks of a physician
Milestone
A milestone is an observable marker of an individual’s ability
EPA and Milestones
• EPA - basket or a funnel that aggregates or integrates a bunch of milestones.
Vision of Assessment:EPAs and Milestones
22
Entrustable Professional Activities (EPAs): for assessment• EPAs are the key tasks of
a discipline that a practitioner needs to be able to perform.
Milestones: for teaching• Milestones are the
abilitiesexpected of a health professional at a stage of development, based on the CanMEDS Roles.
CBD : Vision of assessment
EPAs and milestones provide staff and residents clear expectations of the skills and abilities they need at each stage of training
Help plan teaching and learning opportunities
Based on observations of resident’s ability to perform the EPA staff will assess if they can be entrusted to perform this EPA in the future
As a teacher, how will CBD affect me?
Each rotation will include a series of competency milestones that must be taught and learned
Residents will start to see us as someone who helps them develop their skills
Based on direct and indirect observation, short focused feedback will be given to learners and brief records of the observed performance will be completed
As a teacher, how will CBD affect me?
As a supervisor, you will observe residents as they perform an EPA and you will coach the residents to improve their performance
Refer to the milestones associated with an EPA to give feedback on specific elements of the task
As you become familiar you may ‘unpack’ an EPA (i.e. look at all of the milestones) to plan teaching or help a resident who is struggling to progress
Milestones and EPAs within Each Stage of Training
27
• Provide clear learning and teaching paths
• Remove need to teach or evaluate everything at once – focused on discrete clinical tasks
• Allow for regular, meaningful feedback, tracking and coaching
• Generate practical and demonstrable evaluation data
Assessment for learning
EPA : Providing neuraxial anesthesia for ASA 1 and 2 patients undergoing scheduled or emergency non-obstetrical surgery
Key Features
This EPA includes the technical skill of administering a spinal or epidural anesthetic, as well as the decision-making regarding proper patient selection, ensuring procedure appropriateness, identifying contraindications, full discussion with patient of risks/benefits/complications, and postoperative disposition including addressing postoperative VTE prophylaxis and resumption of other anticoagulant/antiplatelet medications.
EPA : Providing neuraxial anesthesia for ASA 1 and 2 patients undergoing scheduled or emergency non-obstetrical surgery
Assessment plan:
Part A: Patient management
Supervisor does assessment based on direct observation and/or case review and debrief
Use Form 1. Form collects information on:Type of surgical procedure: general surgery; gynecology; urology; orthopedic surgery; plastic surgery
Collect 5 observations of achievement At least three assessors
Part B: Procedure
Supervisor does assessment based on direct observation
Use Form 2. Form collects information on:Procedure: spinal/epidural
Collect 10 observations of achievement No more than 2 observations by the same assessor
EPA : Providing neuraxial anesthesia for ASA 1 and 2 patients undergoing scheduled or emergency non-obstetrical surgery
Relevant milestones (Part A)
1 F ME 1.4.1 Perform focused clinical assessments with recommendations that are well-documented (key points ; history, physical examination of spine, review investigations including coagulation profile; awareness of latest guidelines from professional bodies like ASRA)
2 TP ME 2.4.3 Establish and perform patient-centred perioperative anesthetic management plan for neuraxial anesthesia
3 F ME 2.4.4 Ensure that the patient and family are informed about the risks and benefits of each treatment option in the context of best evidence and guidelines
4 F ME 3.1.10 Determine the most appropriate anesthetic management in relation to patient’s condition and surgical procedure
5 F ME 3.3.12 Describe the indications, absolute and relative contraindications, risks and alternatives for neuraxial blocks
6 F COM 1.6.3 Assess a patient’s capacity to understand and appreciate the issues and risks, participate in decision-making and their capacity to give informed consent
7 F COM 5.1.2 Document clinical encounters to adequately convey clinical reasoning and the rationale for decisions to offer neuraxial blocks (over GA)
8 F COL 2.2.1 Identify communication barriers between anesthesiologists and surgeons
9 F COL 2.2.2 Communicate clearly and directly to promote understanding, manage differences and resolve conflicts
10 F S 2.3.1 Identify unsafe clinical situations involving learners and manage them appropriately
11 F P 2.2.2 Monitor institutional and clinical environments and respond to issues that can harm patients or the delivery of health care
EPA : Providing neuraxial anesthesia for ASA 1 and 2 patients undergoing scheduled or emergency non-obstetrical surgery
Relevant milestones (Part B)
1 F ME 1.3.7 Apply knowledge of the anatomy of the spine and spinal cord, including surface anatomy and deeper structures
2 TTD ME 3.1.10 Apply appropriate monitors correctly for the planned procedure
3 TTD ME 3.4.6 Demonstrate effective procedure preparation, including gathering required equipment and optimal positioning
4 F ME 3.4.2.1 Seek assistance as needed when unanticipated findings, difficulties, or changing clinical circumstances are encountered 5 F ME 3.4.7 Perform neuraxial anesthesia in a skillful, fluid, and safe manner
6 F COM 1.2.1 Optimize the physical environment for patient comfort, privacy, engagement, and safety
7 F COM 5.2.7 Appropriately document anesthetic care and technique in an accurate, complete, timely, and accessible manner
8 F S 1.1.2 Use technology to develop, record, monitor, revise, and report on learning
9 F S 1.1.3 Demonstrate a structured approach to monitoring progress of learning in the clinical setting
10 S F 2.3.1 Identify unsafe clinical situations involving learners and manage them appropriately
11 F P 2.2.6 Demonstrate a commitment to patient safety and quality improvement through adherence to institutional policies
12 F P 2.2.5 Monitor institutional and clinical environments and respond to issues that can harm patients or the delivery of health care
How will it work ?
• Royal College’s MAINPORT ePortfolio (an enhanced version of MAINPORT developed for CBD) to track and record learning and assessment via milestones and Entrustable Professional Activities (EPAs)
• However, many university partners already have systems in place that pose challenges to how data is shared, accessed and integrated
• The Royal College is in the process of finalizing a clickable prototype of MAINPORT ePortfolio, which will clarify the data entry processes and assessment workflows
How Will it Work? Learner
33
Learner
Supported by technology
Plan learning• Views Learning Plan EPAs &
milestones• Selects training activities • Requests observation
from Observer
Participates in learning• Works with Observer &
peers• Reflects on learning• Contributes evidence
of learning
Reflects on observations & learnings
Participates in observation encounter
How Will it Work? Observer
34
Observer
Supported by technology
Supports learning• Works with learners• Contributes narrative• Reviews evidence
Prepares for observation• Reviews and accepts
requests for observation• Reviews EPAs and
milestones
Participates in observation encounter
Evaluates EPA/milestones & records data via
technology
What Happens to the Observation Data?
35
Program Director
Supported by technology
Guides learning• Meets with learners regularly• Assigns EPAs to Observer• Reviews learner progress using technologyReviews program status• Reviews program data and investigates issues • Prepares for Competence Committee meetings
Competence Committee
Assess overall achievement of EPAs & milestones• Review learner EPAs/milestones• Approve changes to learner status
• Promote learner to next stage of training• Request Royal College certification• Modify Learner program plan• Monitory overall progress of learner
• Provides group decision process rather than single summative assessment
Supported by technology
Competence Committee
• Critical component of CBD
• Review and make decisions related to the progress of residents and ensure they achieve the national standards established by each Royal College discipline
Competence Committee -Composition
• Reports to the Residency Training Committee
• Chaired by a faculty member who is not the program director
• Minimum 3 members
• Composed of training committee members and clinical supervisors
Competence Committee -Responsibility
• Monitor the progress of each resident in demonstrating achievement of the EPAs or independent milestones within each stage
• Synthesizing the results from multiple assessments and observations to make decisions related to:
I. promotion of residents to the next stage of trainingII. review and approve individual learning plans to address areas
for improvementIII. determine readiness to challenge the Royal College
examinationsIV. determine readiness to enter independent practice V. determining that a trainee is failing to progress within the
program
Competence Committee -Responsibility
• Monitoring the outcome of learning or improvement plan established for a resident
• Maintaining confidentiality and promoting trust
Challenges
• Determining the EPAs and Milestones for each stage
• Number of observations for each EPA• Daily assessment ?• Electives / rotations in other programs• Data - Mainport or local• Training and support for faculty/training
committee
• Assessment is a key component of CBD, with a focus on methodic work-based assessment
• Program Directors, Observers and Learners share responsibility for learning in residency
• Opportunity to improve the way we teach and assess trainees