Observable Practice Activities (OPAs) Process-Based (POPA) Mid/End of Rotation Feedback (formative) Entrustment (1-5) Content-Based (COPA) Curricular Milestones

Observable Practice Activities (OPAs)

Process-Based (POPA)

Mid/End of Rotation Feedback (formative)

Entrustment (1-5)

Content-Based (COPA)

Curricular Milestones

Reporting Milestones

End-of-trainingEPAs

External ReportingNAS report

Longitudinal Assessment(summative)

Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment . J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518

Mapping Mapping

Mapping

• We mapped with the following in mind:– when assessing a trainee on a given OPA, would the

evaluator feel as if they were also assessing the mapped milestones?

– We removed all soft calls


OPA: Minimize unfamiliar terms during patient encounters

Milestones

ICS-A2 - Effectively use verbal and nonverbal skills to create rapport with patients/families

ICS-D1 - Deliver appropriate, succinct, hypothesis-driven oral presentations

SBP-B2 - Work effectively as a member within the inter-professional team to ensure safe patient care

GIMPGY-1 Wards

COPA: Identify causes of delirium

Reporting MilestonesPC-1 - Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s).

MK-1 Clinical knowledge

COPA: Manage exacerbations of obstructive lung disease

Reporting MilestonesPC-2 Develops and achieves comprehensive management plan for each patient. PC-3 Manages patients with progressive responsibility and independence. MK-1 Clinical knowledge MK-2 Knowledge of diagnostic testing and procedures

COPA: Initiate cost-effective workup for anemia

Reporting MilestonesPC-1 Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s)MK-1 Clinical knowledge MK-2 Knowledge of diagnostic testing and proceduresSBP-3 Identifies forces that impact the cost of health care, and advocates for, and practices cost-effective care

COPA: Assess and treat pain

Reporting MilestonesPC-2 Develops and achieves comprehensive management plan for each patient PC-3 Manages patients with progressive responsibility and independence MK-1 Clinical knowledge PROF-3 Responds to each patient’s unique characteristics and needs

POPA: Demonstrate shared decision-making with the patient

Reporting MilestonesPC-2 Develops and achieves comprehensive management plan for each patient PROF-1 - Has professional and respectful interactions with patients, caregivers and members of the interprofessional team (e.g. peers, consultants, nursing, ancillary professionals and support personnel)PROF-3 Responds to each patient’s unique characteristics and needs ICS-1 - Communicates effectively with patients and caregivers.

POPA: Use feedback to improve performance

Reporting MilestonesPBLI-3 - Learns and improves via feedbackPROF-4 - Exhibits integrity and ethical behavior in professional conduct.

Mapping OPAs to MilestonesRotation 1 Score Patient Care

Medical Knowledge

Practice Based Learning and Improvement

Interpersonal and Communication

Skills Professionalism

Systems Based

PracticeOPA 1 3 A8 OPA 2 3 A4, A5, A6, A8 OPA 3 2 A4, A5, D1 OPA 4 3 A3 OPA 5 2 F2 F4 OPA 6 2 C3 OPA 7 3 A2 E2, E3 OPA 8 2 C4, E2, F6, F7, A6,

Rotation 2 Patient CareMedical

Knowledge




Systems Based

PracticeOPA 9 3 F5, F6, A4, A6, A8, OPA 10 3 F5, F6, A4, A6, A8, OPA 11 3 C2, C3, C4 B1, B2 B1 E2, E3OPA 12 4 C4 A4, A6, A8, B2 OPA 13 2 C3, F5, F6 A4, A6, A8, B2 OPA 14 2 F5, F6, A4, A6, A8, B2 OPA 15 3 F5, F6, A4, A6, A8, OPA 16 2 F5, F6, A4, A6, A8,


Rotation 1 Score Patient CareMedical

Knowledge




Systems Based



Knowledge




Systems Based



Mapping OPAs to Milestones


Knowledge




Systems Based



Knowledge




Systems Based





Knowledge




Systems Based

PracticeOPA 1 3 3 OPA 2 3 3 OPA 3 2 2 OPA 4 3 3 OPA 5 2 2 2 OPA 6 2 2 OPA 7 3 3 3 OPA 8 2 2 2


Knowledge




Systems Based

PracticeOPA 9 3 3 3 OPA 10 3 3 3 OPA 11 3 3 3 3 3OPA 12 4 4 4 OPA 13 2 2 2 OPA 14 2 2 2 OPA 15 3 3 3 OPA 16 2 2 2




Knowledge




Systems Based

PracticeOPA 1 3 3 OPA 2 3 3 OPA 3 2 2 OPA 4 3 3 OPA 5 2 2 2 OPA 6 2 2 OPA 7 3 3 3 OPA 8 2 2 2


Knowledge




Systems Based

PracticeOPA 9 3 3 3 OPA 10 3 3 3 OPA 11 3 3 3 3 3OPA 12 4 4 4 OPA 13 2 2 2 OPA 14 2 2 2 OPA 15 3 3 3 OPA 16 2 2 2

AVG 2.67 2.67 3.00 2.67 2.50 3.00



Traditional Competency-Based Evaluations

OPA Based Evaluations

Patient Care Patient Care

Titrate Insulin Manage Vent Read EKG Diagnose Hepatitis

The Milestones are the apples to be compared

Titrate Insulin Manage Vent Read EKG Diagnose Hepatitis

MK A1

1. Understand the relevant pathophysiology and basic science for common medical conditions

PC E2

2. Make appropriate clinical decision based upon the results of diagnostic tests

PBLI B1

1. Identify learning needs (clinical questions) as they emerge in patient care activities

Approximately 3000

Assessments


Resident A Resident B

Resident C

Resident A Resident B

Resident C

Resident D Resident E


Resident F Resident F


6 nurses

Reporting Milestones (ACGME)

Example: PGY-3 30 Month NAS Report

1 1.5 2 2.5 3 3.5 4 4.5 5

Accepts responsibility and follows through on tasks.Recognizes system error and advocates for system improvement.

Transitions patients effectively within and across health delivery systems.

Exhibits integrity and ethical behavior in professional conduct.

Has professional and respectful interactions with patients, caregivers and members of the interprofessional team (e.g. peers, consultants, nursing, ancillary professionals and support personnel).

Works effectively within an interprofessional team (e.g. peers, consultants, nursing, ancillary professionals and other support personnel).

Communicates effectively in interprofessional teams (e.g. peers, consultants, nursing, ancillary professionals and other support personnel).

Communicates effectively with patients and caregivers.

Responds to each patientâ€™s unique characteristics and needs.

Learns and improves via feedback.

Appropriate utilization and completion of health records.Learns and improves via performance audit.Develops and achieves comprehensive management plan for each patient.

Requests and provides consultative care.

Monitors practice with a goal for improvement.

Learns and improves at the point of care.

Clinical knowledge

Manages patients with progressive responsibility and independence.

Gathers and synthesizes essential and accurate information to define each patientâ€™s clinical problem(s).

Identifies forces that impact the cost of health care, and advocates for, and practices cost-effective care.

Knowledge of diagnostic testing and procedures.

Skill in performing procedures.

1.8

3.8

Entire Residency

NAS report Jul-Dec 2013

Accepts responsibility and follows through on tasks.Exhibits integrity and ethical behavior in professional conduct.








Appropriate utilization and completion of health records.Learns and improves via performance audit.Monitors practice with a goal for improvement.


Develops and achieves comprehensive management plan for each patient.



Clinical knowledge





1.8

3.8

PGY-1s

Accepts responsibility and follows through on tasks.Transitions patients effectively within and across health delivery systems.








Appropriate utilization and completion of health records.

Recognizes system error and advocates for system improvement.Requests and provides consultative care.Learns and improves via performance audit.



Clinical knowledge







1.8

3.8

PGY-2

Accepts responsibility and follows through on tasks.Works effectively within an interprofessional team (e.g. peers, consultants, nursing, ancillary professionals and other support personnel).






Recognizes system error and advocates for system improvement.



Develops and achieves comprehensive management plan for each patient.Requests and provides consultative care.Learns and improves via performance audit.





Clinical knowledge





1.8

3.8

PGY-3

Accepts responsibility and follows through on tasks.Learns and improves at the point of care.


Learns and improves via performance audit.






Develops and achieves comprehensive management plan for each patient.Manages patients with progressive responsibility and independence.Exhibits integrity and ethical behavior in professional conduct.





Clinical knowledge





1.8

3.8

Single PGY-1 – Reviewed in Depth by Competency Committee 982 Assessments

Has professional and respectful interactions with patients, caregivers and members of the interprofessional team (e.g. peers, consultants, nursing, ancillary professionals and support personnel).Exhibits integrity and ethical behavior in professional conduct.


Accepts responsibility and follows through on tasks.






Requests and provides consultative care.Manages patients with progressive responsibility and independence.

Learns and improves via performance audit.

Clinical knowledge








1.8

3.8

Single PGY-1 NOT Reviewed in Depth by Competency Committee717 Assessments

Takes leadership role of teaching healthcare teamIdentify common reasons for hospital re-admission from hospice careDemonstrates respectful behavior to all members of the health care team.Communicate with family members whose loved ones have cognitive impairment or altered mental statusWas this resident receptive to the unique information/learning/skills development that nurses have to offer?Demonstrate cultural sensitivityCreate and deliver on-site projects as assignedConstruct a treatment plan for patient with heart failureConduct a comprehensive psychiatric interview of a medically ill patientIdentify jugular venous distension on physical exam.Identify the cause of heart failure exacerbationsManage a patient with dysfunctional uterine bleedingManage COPDUtilize scarce resources in a creative and practical mannerUndertake the psychotherapeutic treatment of at least one patient with a psycho-physiological illnessIdentify an S3 on physical examConstruct a comprehensive psychiatric treatment plan for a medically ill patientDiscuss diet restrictions with a patientInterpret wet prepManage hypertensionManage diabetesManage hyperlipidemiaManage depressionAssist colleagues in the provision of duties.Demonstrate respectful and professional manner during sensitive physical exam maneuversDemonstrate safe, accurate, and complete hand-offs.Role model effective communication skills in challenging situationsProvides regular feedback to other members of the team.Respond to pages in timely and courteous mannerProvide care for patients in non-traditional ways between office visits (telephone, email, group visits)Safely â€œsign outâ€ patients among members of health care team�Track, coordinate, and ensure follow up on messages, tests, consults, and care at other facilitiesDemonstrate proper ethics in medical researchAssist patient and families in setting and clarifying goals at the end of lifeProvide age-appropriate cancer screening (breast, cervical)Demonstrate communication regarding patient care to other healthcare team members in a timely mannerManage patients care in a safe manner.Manage complicated diabetes patientsRecognize when to refer to specialistStandard of Work AssessmentInitiate antibiotic therapy for neutropenic fever.Workup thyroid diseaseManage anticoagulation in a patient with suspected or known venous thromboembolismUtilize narcotic table to titrate pain medicationsManage chronic sinusitisRecognize acute stroke and activate stroke teamManage and escalate care in a patient with sepsisEscalate care in a critical or unstable patientManage benzodiazepines in a patient with alcohol withdrawalEvaluate patients with undifferentiated urgent medical needCreate a collaborative learning/teaching environment that incorporates other staff members including nursing?Adjust type, dose, and duration of therapy for pneumonia based on clinical courseRecognize the need for urgent/emergent medical consultationManage diabetic ketoacidosisPrepare patients and families for inpatient versus outpatient hospice transitionsWork -up spinal cord compressionExplore and respond to patient/family experiences when transitioning from curative to comfort careDemonstrate appropriate use of blood productsCoping with Complexity AssessmentManage acute pain syndromes from orthopedic sourcesFormulate antibiotic discharge planning for neutropenic fever.Initiate workup for a new breast lumpSuggest adequate peri-operative DVT prophylaxisPerform a routine pelvic and breast examManage peri-operative hypertensionAssess postoperative feverManage hyperglycemia in the peri-operative diabetic patientOverall Knowledge AssessmentRecognize and manage impending respiratory failureDiffferentiate deep-seated soft tissue infections versus superficial infectionsUse an opioid conversion table to titrate pain managementUtilize pharmacologic and non-pharmacologic methods to manage deliriumDescribe community resources available to elderly patientsObtain thorough dermatologic historyDemonstrate rehabilitation exercises for chronic low back painDemonstrate a cost effective workup of loss of consciousnessDemonstrate empathy, compassion, and a commitment to relieve pain and sufferingManage postoperative delirium.Minimize peri-operative pulmonary riskManage the interdisciplinary teamPerform pre-operative cardiac risk stratificationInitiate a cost effective work-up for iron deficiency anemiaChoose the appropriate consultative services for a given clinical conditionMinimize peri-operative cardiac riskEvaluate memory loss/dementia and suggest initial management planManage anticoagulation for venous thromboembolism in the cancer patient.Manage derangements of sodiumGuide and support bedside presentations that engage the patient and focus the discussion around the patientâ€™s central concernsManage hyperuricemia during tumor lysis syndrome.Communicate and monitor for side effects of pain medicationsAssess pain using an established pain scaleManage end of life care family meeting.Recognize and manage conflicts of interest (such as caring for family members and professional associates as patients)Manage peri-operative anticoagulation in the patient taking chronic anticoagulants including anti-platelet agentsRefer patients with acute and chronic renal failure for renal replacement therapyDemonstrate shared decision-making with the patientManage extremes of blood pressureTeach physical findings for junior members of the health care teamPerform bedside presentations that engage the patient and focus the discussion around the patientâ€™s central concernsDemonstrate appropriate use of blood products and EPO analogues in CKD and ESRD populationManage peri-operative NSTEMIManage anticoagulant therapy in cardiac patientsStabilize patients with urgent or emergent medical conditions and transfer to a higher level of care when necessaryDevelop a hypothesis driven research questionManage a patient with perimenopausal symptomsOrder advanced imaging in the work-up of a pain syndromeManage CKD patients in the peri-operative settingTitrate narcotic pain medicationInitiate non-pharmacological treatments for chronic pain.Manage non-narcotic adjuvant pain medicationUses shared decision making strategies for complicated or ambiguous pain management scenariosInitiate a pain regimen for a chronic pain syndromeManage pancreatitisDiagnose a chronic pain syndromeDescribe rash using dermatologic termsManage complications of immunosuppressive therapyTitrate cardiac medicationsCommunicate effectively with the consulting teamModify the differential diagnosis and care plan based on clinical course and data as appropriateOrder appropriate GI proceduresDemonstrate proficiency in the placement of a central venous catheterManage a patient at-risk for osteoporosisDemonstrate proficiency in the placement of an arterial lineTreat pain encountered with terminal illnessEvaluate functional abilities of geriatric patientManage gastrointestinal bleedingManage acute coronary syndromeSupervise PGY-1 residents in the care of patients overnightWrite initial admission orders on geriatric patient in the post hospital settingBegin initial work-up for gastrointestinal bleedingEvaluate obstructive jaundiceInterpret advanced EKGs and use as tool to make management decisionsInterpret basic and advanced EKGsGather subtle, sensitive, and complicated information that may not be volunteered by the patientDiscuss a 2nd line or 3rd line smoking cessation plan for a patient who failed 1st line therapy.Manage acute hepatitisEvaluate cause of falls and suggest initial management planTake a rheumatologic historyPerform a focused physical exam on post hospital discharge patientManage post-procedure complicationsManage arrythmiasManage cardiomyopathyInterpret pulmonary artery catheter dataManage common sequelae of cirrhosisIdentify patients who may benefit from liver transplantationRefer patients for pulmonary rehabilitationManage diarrhea in a hospitalized patientManage inotropes and vasopressor agents in the management of shock and heart failure.Manage acute chest syndrome in sickle cell patients.Autonomy AssessmentPerform arterial linesIntegrate clinical evidence into decision makingProvide anticipatory guidance during routine visits in a primary care settingDifferentiate the causes and findings of cardiogenic from distributive and septic forms of shockDeliver appropriate goal-directed therapy for severe sepsisManage a patient on chronic home oxygen therapy who presents with the complaint of dyspnea.Perform medication reconciliation on post hospital discharge patientAdjust long-acting and/or short-acting narcotics in cancer and sickle cell patients.Determine cause of leukocytosisRecommend GI procedures and imagingUse feedback to improve performanceInterpret cardiac biomarkersUse low tidal volume strategy where appropriate for ARDS.Manage a hospitalized immunosuppressed renal transplant patientManage inflammatory bowel diseaseIdentify and manage arrhythmiasPrescribe antiplatelet therapy for vascular diseaseIdentify appropriate cervical cancer screening.Select initial therapy for patients diagnosed with HIVManage acute and chronic viral hepatitisInitiate diagnostic work-up for suspected Heparin Induced Thrombocytopenia.Take a history specific to gastroenterologyEvaluate disorders of hemostasisInterpret thoracentesis findingsOrder imaging and tissue studies to properly stage malignanciesDevelop treatment plan for patients with shortness of breathPerform a breast exam.Obtain a history and physical exam in a timely mannerWrite a prescription for the appropriate amount of oxygen at rest, exertion, and sleepCommunicate effectively with consultantsInterpret results of a graded exercise stress test (treadmill with EKG) and calculate Duke Score.Triage patient to proper level of careDevelop initial treatment plan for patients with chest painInitiate diagnostic testing in patients with suspected venereal diseases.Determine screening for viral hepatitis.Refer patients for allergy skin testingPerform pap smear.Determine duration of treatment for osteomyelitisPerform work-up of suspected lung cancer/noduleSelect type, dose, and duration of antibiotic therapy for skin and soft-tissue infections.Diagnose and manage GI infectious diseases (including diarrheal diseases)Identify and assess clinically significant drug interactions between antiretroviral medications and concomitant drug therapy.Initiate workup for polycystic ovarian syndrome.Evaluate myeloproliferative disorders, myeloma and/or MGUSDetermine cause of anemiaManage chronic heart failureDemonstrate pre-procedure evaluation for someone pending bronchoscopyIdentify and begin initial management plan for basic arrhythmiasProvide first line therapy to a cystic fibrosis patient presenting with a new productive cough.Identify bronchoscopy indications for patientsInitiate end of life discussions with patients with incurable malignanciesUse clinical insight to develop research questions.Develop initial treatment plan for patients with suspected infectionDetermine treatment for latent TB infection.Manage individuals with rhinitisDiscuss therapeutic options for problems identified by interventional pulmonologyRecommend appropriate route for nutritional supplementationEvaluate urinary incontinence and suggest initial management planDescribe resources available to hospice patientsPerform a history and physical exam for patients with HIV.Manage multiple patients simultaneouslyImplement the appropriate mode of ventilatory assistance for acute respiratory failureInitiate evaluation for acute chest syndrome in sickle cell anemia.Recognize the scope of his/her abilities and ask for supervision and assistance appropriatelyPerform central linesInterpret chest x-rays for common lung disordersPerform a musculoskeletal examinationInitiate work-up of spinal cord compressionDetermine the causes of thrombocytopeniaAdjust short and long acting narcotics for chronic painManage anticoagulation of venous thromboembolism in the cancer patientUtilize oximetry and arterial blood gas results to determine the need for supplemental oxygen therapyIdentify the salient features of fulminant hepatic failureDiagnose inflammatory bowel diseaseDistinguish between cirrhotic and non-cirrhotic portal hypertensionPerform thoracentesisManage hypokalemia and hyperkalemiaInterpret a cosyntropin stimulation testManage acute renal failureFormulate cost-effective approach to the diagnosis of chronically elevated transaminasesAssess risk factors for acute hepatitis, including the use of herbal and over the counter agentsAcquire accurate and relevant historyProvide accurate, complete, and timely documentationInterpret serologic testing for hepatitis A, hepatitis B, and hepatitis CDose drugs appropriate to the level of renal functionManage heart failureRefer patients for hospice and palliative careMinimize unnecessary care including testsInterpret EKGs and use as tool to make management decisionsInitiate management of hypertensive emergencyDifferentiate between various types of hemolytic anemiaInitiate workup for acute renal failureManage daily ventilator changes for respiratory failureEvaluate altered mental statusImplement a diagnostic work-up for metabolic acidosis and alkalosisRecognize acute stroke and begin initial managementManage anticoagulation for patients with neurologic diseasesInitiate workup of non neutropenic fever.Manage direct thrombin inhibitors for Heparin Induced Thrombocytopenia.Formulate a cost-effective approach to the diagnosis of cholestatic liver diseaseRecite and apply the criteria for decision making on extubating patientsPrescribe emergency treatment plan for a patient at risk for anaphylaxisOrder diagnostic testing for neurologic diseasePerform an accurate physical examDiagnose neutropenic feverInitiate the basic workup for a patient with undifferentiated insterstitial lung disease.Describe the benefits and limitations associated with different modes of mechanical ventilationDemonstrate appropriate use of ultrasound during invasive procedures.Refer patients for appropriate cardiac imagingDetermine need for plasmapheresis in the setting of TTP.Initiate workup for disorders of sodiumRecognize the risk factors and clinical signs/symptoms associated with sleep apnea.Initiate workup for acid base disturbancesManage kidney disease for inpatients on other servicesInterpret pulmonary function testsDifferentiate a transudative from an exudative pleural effusion, correctly apply further testing and treatment based on this differentiationDemonstrate accurate medication reconciliationInitiate diagnostic testing for venous thromboembolism in the cancer patientInitiate antibiotics for pneumoniaPerform intubationInterpret blood glucose readingsCommunicate with primary care physicians

Develop prioritized differential diagnosesInitiate investigation of abdominal painInitiate antibiotic therapy for neutropenic feverManage hypothyroidismRecommend bronchoscopy for various clinical scenariosInitiate venous thromboembolism prophylaxisInitiate diagnostic workup for hemoptysis.Provide appropriate preventive careManage hyperkalemia in the renal patientPrescribe medication for seizure disorderEvaluate hypercalcemiaRecommend proper monitoring and management of diabetic microvascular and macrovascular complicationsMinimize unfamiliar terms during patient encountersWork-up lung nodule or massInitiate CIWA protocol in patients at risk for alcohol withdrawalManage parathyroid abnormalities in the renal patientTitrate insulin based on glucose readingsManage individuals with asthmaLocalize neurologic lesionsEvaluate individuals with urticaria/angioedemaRecommend acute dialysis for selected patientsEvaluate thyrotoxicosisUse teach-back method with patients regarding medications and planEvaluate complex medical patient in a timely mannerManage hyperkalemia during tumor lysis syndromeDevelop an evidence-based diagnostic and therapeutic planInitiate fall precaution orders in patients at risk for fallsDiagnose spinal cord compressionManage derangements of potassiumIdentify the appropriate clinical question for consultative servicesPerform a comprehensive neurological examinationRefer appropriate patients with acute and chronic renal failure for renal replacement therapyEvaluate hypogonadismIdentify the indications and contraindications for liver transplantationAdjust type, dose, and duration of therapy for pneumonia based on historical risk factors and clinical courseSelect type, dose, and duration of antibiotic therapy for complicated urinary tract infectionsInitiate antibiotic(s) for pneumoniaIdentify clinical questions as they emerge in patient care activities and access medical information resourcesManage exacerbations of obstructive lung diseaseManage endocarditisDemonstrate proficiency in endotracheal intubationMake appropriate recommendations for adjustment of ventilator settingsManage elevated blood pressureDifferentiate cardiac versus non-cardiac chest discomfortRefer for right heart catheterization when indicatedInitiate basal bolus insulin therapy and manage blood glucose over timeAssess and treat painSelect antibiotics and duration of therapy for treatment of bacteremia.Select antibiotic therapy and duration for organisms with antibiotic resistanceSelect type, dose, and duration of therapy for post-operative infectionsDocument cross-cover careDetermine diagnostic testing and treatment of infectious colitis.Initiate work-up of diabetic foot ulcerInitiate antibiotic(s) for skin and soft tissue infectionsInitiate cost-effective workup for anemiaWrite initial admission orders for pancreatitisInitiate cost effective workup of venous thromboembolismEvaluate pituitary adenomaManage suspected meningitisInitiate workup of fever of unknown originSelect type, dose, and duration of antibiotic therapy for osteomyelitisWrite initial admission orders for gastrointestinal bleedingPerform paracentesisCounsel patient on lifestyle modifications for aggressive risk factor modificationsIdentify causes of delriumInitiate evaluation for acute chest syndrome in sickle cell diseaseDistinguish between hepatitic, cholestatic, and infiltrative patterns of liver diseaseDiagnose the cause of loss of consciousness and differentiate syncope from other etiologiesDetermine the appropriate initial laboratory evaluation for a suspected immune deficiencyIdentify the cause of diarrhea (acute or chronic)Manage pain complaints overnightDemonstrate appropriate use of blood products and EPO analogues in CKD and ESRD patientsBegin initial management plan for basic arrhythmiasIdentify when it is appropriate to intubate a patient in acute respiratory failureInitiate appropriate antibiotic(s) for skin and soft tissue infectionsDeliver appropriate goal-directed therapy for severe sepsis.Manage hyperglycemiaIdentify how to change a ventilator to improve ventilation and oxygenation in acute respiratory failureManage alcohol withdrawalManage derangements of electrolytesInitiate appropriate antibiotic(s) for pneumoniaIdentify appropriate circumstances to use non-invasive positive pressure ventilation (NIPPV) such as CPAP or BiPAP for acute respiratory failureRecognize delirium and identify potential causesInitiate workup and management of feverInterpret chest x-rays for common lung disorders.Manage decompensated liver diseaseDiscuss indications for GI proceduresLead an interprofessional team meeting or â€œhuddleâ€ �Collaboratively assess and refine office systems based on workflow analysis and patient experienceManage and write ventilator ordersDiagnose the cause of loss of consciousness and differentiate syncope from other etiologies.Diagnose acute coronary syndrome (unstable Angina, NSTEMI, STEMI)Manage heart failure (acute, chronic, systolic and diastolic)Perform arterial lines.Demonstrate basic EKG reading skillsPerform central lines.Improve care via Plan-Do-Study-Act (PDSA) cycles

2

4

Entire Residency – All OPAs from Jul 1st to Feb 28th

Assist colleagues in the provision of duties.Demonstrate safe, accurate, and complete hand-offs.Respond to pages in timely and courteous mannerProvides regular feedback to other members of the team.Demonstrate shared decision-making with the patientPerform bedside presentations that engage the patient and focus the discussion around the patientâ€™s central concernsDemonstrate empathy, compassion, and a commitment to relieve pain and sufferingInterpret pulmonary function testsInitiate cost effective workup of venous thromboembolismInitiate basal bolus insulin therapy and manage blood glucose over timeDifferentiate a transudative from an exudative pleural effusion, correctly apply further testing and treatment based on this differentiationInitiate fall precaution orders in patients at risk for falls

Distinguish between cirrhotic and non-cirrhotic portal hypertensionDiagnose the cause of loss of consciousness and differentiate syncope from other etiologies

Distinguish between hepatitic, cholestatic, and infiltrative patterns of liver diseaseInitiate CIWA protocol in patients at risk for alcohol withdrawal

Formulate a cost-effective approach to the diagnosis of cholestatic liver diseaseInitiate diagnostic workup for hemoptysis.

Formulate cost-effective approach to the diagnosis of chronically elevated transaminasesInitiate venous thromboembolism prophylaxisGather subtle, sensitive, and complicated information that may not be volunteered by the patientInterpret serologic testing for hepatitis A, hepatitis B, and hepatitis CManage common sequelae of cirrhosisManage derangements of potassiumAssess risk factors for acute hepatitis, including the use of herbal and over the counter agentsRecognize the risk factors and clinical signs/symptoms associated with sleep apnea.Refer appropriate patients with acute and chronic renal failure for renal replacement therapyCommunicate effectively with the consulting teamIdentify the salient features of fulminant hepatic failure

Identify the indications and contraindications for liver transplantationWas this resident receptive to the unique information/learning/skills development that nurses have to offer?

Demonstrates respectful behavior to all members of the health care team.Communicate with primary care physicians

Minimize unfamiliar terms during patient encountersAcquire accurate and relevant history

Standard of Work AssessmentDemonstrate accurate medication reconciliation

Autonomy AssessmentPerform an accurate physical exam

Provide accurate, complete, and timely documentationIdentify appropriate circumstances to use non-invasive positive pressure ventilation (NIPPV) such as CPAP or BiPAP for acute respiratory failureIntegrate clinical evidence into decision makingInitiate antibiotics for pneumoniaIdentify when it is appropriate to intubate a patient in acute respiratory failureUse feedback to improve performanceAssess and treat painModify the differential diagnosis and care plan based on clinical course and data as appropriateIdentify causes of delriumMinimize unnecessary care including testsInitiate antibiotic(s) for pneumoniaIdentify the appropriate clinical question for consultative servicesRecognize the scope of his/her abilities and ask for supervision and assistance appropriatelyManage elevated blood pressureDevelop an evidence-based diagnostic and therapeutic planDevelop prioritized differential diagnosesCoping with Complexity AssessmentUse teach-back method with patients regarding medications and planIdentify clinical questions as they emerge in patient care activities and access medical information resourcesOverall Knowledge AssessmentManage exacerbations of obstructive lung disease

Initiate workup of non neutropenic fever.Diagnose neutropenic fever

Adjust short and long acting narcotics for chronic painInitiate diagnostic testing for venous thromboembolism in the cancer patient

Utilize oximetry and arterial blood gas results to determine the need for supplemental oxygen therapyRecommend bronchoscopy for various clinical scenarios

Initiate work-up of diabetic foot ulcerManage derangements of electrolytes

Dose drugs appropriate to the level of renal functionIdentify how to change a ventilator to improve ventilation and oxygenation in acute respiratory failure

Initiate appropriate antibiotic(s) for skin and soft tissue infectionsDiagnose the cause of loss of consciousness and differentiate syncope from other etiologies.

Document cross-cover careDeliver appropriate goal-directed therapy for severe sepsis.

Refer for right heart catheterization when indicatedManage extremes of blood pressure

Manage acute renal failureManage hyperglycemia

Work-up lung nodule or massManage hyperkalemia during tumor lysis syndrome

Initiate workup and management of feverManage hyperkalemia in the renal patient

Recognize delirium and identify potential causesManage pain complaints overnight

Initiate antibiotic(s) for skin and soft tissue infectionsManage parathyroid abnormalities in the renal patient

Interpret chest x-rays for common lung disorders.Initiate cost-effective workup for anemia

Refer patients for hospice and palliative careDiagnose spinal cord compressionInitiate appropriate antibiotic(s) for pneumoniaDemonstrate appropriate use of blood products and EPO analogues in CKD and ESRD patientsManage alcohol withdrawalInitiate evaluation for acute chest syndrome in sickle cell diseaseManage and write ventilator ordersImplement a diagnostic work-up for metabolic acidosis and alkalosisWrite a prescription for the appropriate amount of oxygen at rest, exertion, and sleepInitiate antibiotic therapy for neutropenic feverPerform central lines.

1.8

2.8

Single PGY-1 - OPAs from Jul 1st to Feb 28th

Trend line of all PGY-1 Milestone Assessments By Month n = 105721

Standard Deviation of all PGY-1 Milestones Assessments by Month

Validity?

• Validity – how well one can trust the results of a test as interpreted for a specific purpose1

– Not a property of the instrument, but of interpretation

1Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006 Feb;119(2):166.e7-16. Review. PubMed PMID: 16443422

Validity?• Content: do instrument items represent the construct?

• Response Process: are the construct and the thought processes of observers the same?

• Reliability – internal consistency, temporal stability, inter-rater reliability, generalizability

• Consequences: do scores really make a difference?

• Relations to other variables

1Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006 Feb;119(2):166.e7-16. Review. PubMed PMID: 16443422

1 4.07

2 4.05

3 3.91

4 3.89

5 3.79

6 3.77

7 3.77

8 3.71

9 3.60

10 3.57

11 3.55

12 3.49

13 3.44

14 3.40

15 3.32

16 3.32

17 3.27

18 3.26

19 3.19

20 3.07

21 3.05

22 3.05

23 3.00

24 2.91

Selected to be Chief

Resident

Had issues brought before

competency committee

Patient Care

R3 Class Milestone Average Jul 1st to Feb 1st

Communication

1 4.42

2 4.28

3 4.27

4 4.14

5 4.13

6 4.13

7 4.09

8 4.08

9 4.02

10 3.98

11 3.87

12 3.87

13 3.84

14 3.81

15 3.71

16 3.64

17 3.64

18 3.60

19 3.49

20 3.42

21 3.35

22 3.24

23 3.08

24 3.02

Professionalism

1 4.39

2 4.29

3 4.28

4 4.24

5 4.24

6 4.20

7 4.15

8 4.13

9 4.10

10 4.10

11 3.99

12 3.91

13 3.90

14 3.88

15 3.77

16 3.76

17 3.71

18 3.69

19 3.55

20 3.45

21 3.43

22 3.33

23 3.29

24 3.10

Name 2012 ITE MK Milestones

A 8 1

B 4 2C 10 3D 11 4E 21 5F 6 6G 14 7H 7 8I 18 9J 9 10K 12 11L 1 12M 22 13N 2 14O 19 15P 13 17Q 17 18R 5 19S 15 20T 16 21U 20 22

Name 2013 ITE MK MilestonesA 20 1

B 15 2

C 4 3

D 1 4

E 19 5

F 7 6

G 2 7

H 5 8

I 11 9

J 21 10

K 18 11

L 12 12

M 6 13

N 16 14

O 14 15

P 22 16

Q 23 17

R 8 18

S 9 19

T 13 20

U 17 21

V 10 22

W 3 23

2012 2013

Research Directions

• Study the progression of assessment entrustment decisions made over time for internal medicine residents by faculty physicians, peer physicians, and allied health providers

• Identify assessment biases present in these entrustment decisions, including bias associated with age, sex, resident year of training, attending experience level, or job description of assessor

• Determine validity of assessment entrustment decisions made over time for internal medicine residents in relation to other variables such as standardized test scores, and other clinical assessments

OPAs Reporting Milestones

Rides in Driveway

Rides on Sidewalk

Rides on Quiet Street

Rides to work on time

Rides in Traffic

Rides in Triathlon

Rides Bicycle Safely

Falls off Bike

Rides 1000 feet without Stopping

Rides on Quiet Road

Rides in Rush Hour Traffic

Rides Tour de France

NarrativesLevel Of Entrustment

1 2 3 4 5

Level Of Entrustment 1 2 3 4 5

OPAs

Titrate cardiac medications

Manage pancreatitis

Titrate insulin based on glucose readings

Manage parathyroid abnormalities in the renal patient

Adjust short and long acting narcotics for cancer and sickle cell patients

Deliver appropriate goal-directed therapy for severe sepsis

Develops and achieves comprehensive management plan for each patient. (PC2)

Care plans are consistently inappropriate or inaccurate

Inconsistently develops an appropriate care plan

Consistently develops appropriate care plan

Appropriately modifies care plans based on patient’s clinical course, additional data, and patient preferences

Role models and teaches complex and patient-centered care



NarrativesReporting Milestones/EPAs

Observable Practice Activities (OPAs)

Process-Based (POPA)

Mid/End of Rotation Feedback (formative)

Entrustment (1-5)

Content-Based (COPA)

Curricular Milestones


End-of-trainingEPAs

External ReportingNAS report

Longitudinal Assessment(summative)


Mapping Mapping

Residency Education Coordinators

Khan Robertson Houk Falciglia Cohen Chaudhary Lenz Schoch Kamath

Residency Program Staff

Dell Diers Warm FichtenbaumMathis Mueller O’Toole TolentinoPai

LeeHeldAshbrookLee Wood

TEAM EFFORT

Wexler

Schauer

Discussion questions from the survey

• Is it possible to have a generic map that can be used for both Medicine and Peds?– You could design an excel solution which Eric

initially did before medhub figured out how to do it for us

– Other ideas?


• Specific directions for Med-Peds reporting– Will be straightforward with the same forms– ACGME sent me a copy of the reporting form

when I called them and asked them to; it is literally just the 2 sets put together into 1 form

What about similar milestones?

• Some of this is subject to opinion, but about 8-10 are largely the same

• What do I mean?

• This is split into PC, MK and SBP as one slide and PBLI, Prof, and ICS as second slide

• ALERT: COMPLICATED VISUALS!!!

PC, MK, SBP

IM Peds

PC 1

Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s) PC 1

Gather essential and accurate information about the patient

PC 2Develops and achieves comprehensivemanagement plan for each patient PC 2

Organize and prioritize responsibilities to provide patient care that is safe, effective and efficient

PC 3Manages patients with progressive responsibility and independence PC 3

Provide transfer of care that ensures seamless transitions

PC 4 Skill in performing procedures PC 4Make informed diagnostic and therapeutic decisions that result in optimal clinical judgment

PC 5Requests and provides consultative care PC 5 Develop and carry out management plans

MK 1 Clinical knowledge MK 1

Locate, appraise, and assimilate evidence from scientific studies related to their patients’ health problems (EBM)

MK 2Knowledge of diagnostic testing and procedures

SBP 1Works effectively within an interprofessional team SBP 1

Coordinate patient care within the health care system relevant to their clinical specialty

SBP 2Recognizes system error and advocates for system improvement SBP 2

Advocate for quality patient care and optimal patient care systems

SBP3

Identifies forces that impact the cost of health care, and advocates for, and practices cost-effective care SBP3

Work in inter-professional teams to enhance patient safety and improve patient care quality

SBP 4Transitions patients effectively within and across health delivery systems

IM Peds

PC 1



PC 2Develops and achieves comprehensivemanagement plan for each patient PC 2

Organize and prioritize responsibilities to provide patient care that is safe, effective and efficient

PC 3Manages patients with progressive responsibility and independence PC 3


PC 4 Skill in performing procedures PC 4

Make informed diagnostic and therapeutic decisions that result in optimal clinical judgment

PC 5 Requests and provides consultative care PC 5 Develop and carry out management plans

MK 1 Clinical knowledge MK 1

Locate, appraise, and assimilate evidence from scientific studies related to their patients’ health problems

MK 2Knowledge of diagnostic testing and procedures

SBP 1Works effectively within an interprofessional team SBP 1

Coordinate patient care within the health care system relevant to their clinical specialty



SBP3

Identifies forces that impact the cost of health care, and advocates for, and practices cost-effective care SBP3


SBP 4Transitions patients effectively within and across health delivery systems

IM Peds

PC 1



PC 2Develops and achieves comprehensivemanagement plan for each patient PC 5 Develop and carry out management plans

MK 2Knowledge of diagnostic testing and procedures PC 4

Make informed diagnostic and therapeutic decisions that result in optimal clinical judgment

SBP 1Works effectively within an interprofessional team SBP3




SBP 4Transitions patients effectively within and across health delivery systems PC 3


IMPeds

PBLI 1Monitors practice with a goal for improvement PBLI 1

Identify strengths, deficiencies, and limits in one’s knowledge and expertise

PBLI2Learns and improves via performance audit PBLI2

Identify and perform appropriate learning activities to guide personal and professional Development

PBLI3 Learns and improves via feedback PBLI3Systematically analyze practice using quality improvement methods, and implement changes with the goal of practice improvement

PBLI 4Learns and improves at the point of care PBLI 4 Incorporate formative evaluation feedback into daily practice

Prof 1

Has professional and respectful interactions with patients, caregivers and members of the interprofessional team Prof 1

Humanism, compassion, integrity, and respect for others; based on the characteristics of an empathetic practitioner

Prof 2Accepts responsibility and follows through on tasks Prof 2

Professionalization: A sense of duty and accountability to patients, society, and the Profession

Prof 3Responds to each patient’s unique characteristics and needs Prof 3

Professional Conduct: High standards of ethical behavior which includes maintaining appropriate professional boundaries

Prof 4Exhibits integrity and ethical behavior in professional conduct Prof 4

Self-awareness of one’s own knowledge, skill, and emotional limitations that leads to appropriate help-seeking behaviors

ICS1Communicates effectively with patients and caregivers Prof 5

Trustworthiness that makes colleagues feel secure when one is responsible for the care of patients

ICS2

Communicates effectively in interprofessionalteams Prof 6

The capacity to accept that ambiguity is part of clinical medicine and to recognize the need for and to utilize appropriate resources in dealing with uncertainty

ICS3Appropriate utilization and completion of health records ICS1

Communicate effectively with patients, families, and the public, as appropriate, across a broad range of socioeconomic and culturalBackgrounds

ICS2

Demonstrate the insight and understanding into emotion and human response to emotion that allows one to appropriately develop andmanage human interactions

IMPeds

PBLI 1Monitors practice with a goal for improvement PBLI 1

Identify strengths, deficiencies, and limits in one’s knowledge and expertise

PBLI2Learns and improves via performance audit PBLI2

Identify and perform appropriate learning activities to guide personal and professional Development

PBLI3 Learns and improves via feedback PBLI3Systematically analyze practice using quality improvement methods, and implement changes with the goal of practice improvement

PBLI 4Learns and improves at the point of care PBLI 4 Incorporate formative evaluation feedback into daily practice

Prof 1

Has professional and respectful interactions with patients, caregivers and members of the interprofessional team Prof 1

Humanism, compassion, integrity, and respect for others; based on the characteristics of an empathetic practitioner

Prof 2Accepts responsibility and follows through on tasks Prof 2

A sense of duty and accountability to patients, society, and the Profession

Prof 3Responds to each patient’s unique characteristics and needs Prof 3

High standards of ethical behavior which includes maintaining appropriate professional boundaries

Prof 4Exhibits integrity and ethical behavior in professional conduct Prof 4

Self-awareness of one’s own knowledge, skill, and emotional limitations that leads to appropriate help-seeking behaviors

ICS1Communicates effectively with patients and caregivers Prof 5

Trustworthiness that makes colleagues feel secure when one is responsible for the care of patients

ICS2Communicates effectively in interprofessional teams Prof 6

The capacity to accept that ambiguity is part of clinical medicine and to recognize the need for and to utilize appropriate resources in dealing with uncertainty

ICS3Appropriate utilization and completion of health records ICS1

Communicate effectively with patients, families, and the public, as appropriate, across a broad range of socioeconomic and culturalBackgrounds

ICS2

Demonstrate the insight and understanding into emotion and human response to emotion that allows one to appropriately develop and manage human interactions

IM Peds

PBLI3Learns and improves via feedback PBLI 4

Incorporate formative evaluation feedback into daily practice

Prof 4

Exhibits integrity and ethical behavior in professional conduct Prof 3

High standards of ethical behavior which includes maintaining appropriate professional boundaries

ICS1Communicates effectively with patients and caregivers ICS1

Communicate effectively with patients, families, and the public, as appropriate, across a broad range of socioeconomic and cultural backgrounds

Bottom line

• 22 IM milestones• 21 Peds milestones• 43 total• Of these, 9 have significant overlap, so there

are actually 43 minus 9 or 34 SEPARATE milestones to report


• What about overlapping milestones?– Since the goal is to establish whether residents are

progressing and competent, is there a reason not to use the benefit of evaluations from both sides together?

Questions?

http://www.google.com/url?sa=i&rct=j&q=university%20of%20cincinnati%20college%20of%20medicine&source=images&cd=&cad=rja&docid=AAgdSP6KxOwmJM&tbnid=pm84d6N4enpGGM:&ved=0CAUQjRw&url=http://www.med.uc.edu/ortho/trauma101/Trauma101_Faculty.html&ei=OAlOUbLdOJGO8wTwhIDQDQ&psig=AFQjCNGJt6hUbqRKPBt8giFxsKF_BWblHQ&ust=1364154928254793

Documents

Observable Practice Activities (OPAs) Process-Based (POPA) Mid/End of Rotation Feedback (formative) Entrustment (1-5) Content-Based (COPA) Curricular Milestones