Marita S. Teng, MDProgram DirectorDepartment of OtolaryngologyMount Sinai School of MedicineNew York, NY

MANAGING THE ACGME SURVEY

OPDO MEETINGNOVEMBER 11, 2011

Online survey, 34 questionsAdministered annually for

residents and fellowsTiming: mid- January thru

early JuneRequires 70% completion

rateAggregate program-level

data provided to PD and DIO if 70% response rate reached

Site visitors and RRC have data access regardless of response rate

ACGME SURVEY – WHAT IS IT?

Topics covered Duty Hours Faculty Evaluation Systems Educational Content Resources Overall Experience

ACGME SURVEY – WHAT IS IT?

From the ACGME website “Data from the survey are intended as a diagnostic

tool to focus the field staff’s questions during the on-site visit… [and] to verify and clarify the information from them. Field staff will probe and clarify any grey shaded areas, and pay particular attention to the duty hour items and to items with a substantial number of non-compliant responses.”

If residents indicate problem areas corrected, site visitor will document.

If questions/phrases misunderstood by residents, site visitor will document.

DATA FROM SURVEY

“Substantial number of noncompliant responses” 20% is a common institutional

standard Internal systematic review and

action plan requiredRRC may send warning

letters to programs for poor results

Accreditation cycle may be shortened, effectively triggering an immediate site visit

WHEN TO WORRY?

Small programsOne or two disgruntled residents can significantly alter the data

Confidentiality is affected Interns

Data often reflect their experiences on services other than Otolaryngology, and are hard to pinpoint because of varying rotation schedules and anonymity of survey

PROBLEMS SPECIFIC TO US

But no “excuses” are accepted!

Survey instructions sent out only after a intro/Q&A session

Explain purpose of survey, clarify questions

Survey form available on ACGME website – go item by item

This is not the forum for complaints

Consider NOT having the PD conduct the session Associate PD? Chief resident(s)? Program coordinator?

“SAFE SURVEYING”

Remind the residents about certain items: Duty hours Confidentiality of your

evaluation system Availability/accessibility of

Goals & Objectives Resources & systems for

communication/support education committee

meetings faculty mentors chief or senior residents institutional ombudsman

Sleep deprivation education SAFER curriculum (ppt)

“SAFE SURVEYING”

We are better than most!

Focus on areas of around 5% or greater national rates of non-compliance

OTOLARYNGOLOGY PROBLEM AREAS

80-hour week (over 4 wks)1 day off in 7 (over 4 wks)No more than Q3 call (over 4 wks)10 hours off between duty periods24 + 6 continuous hours in-houseHome call does not preclude “rest and reasonable

personal time”Called into hospital from home counts towards

80hrs

DUTY HOURSSEVEN QUESTIONS

Never

Rarely

Sometimes

Very Often

Extremely Often

How often did you break the rule that there should be a 10-hour time period provided between all daily duty periods and after in-house call?

71.9%

22.6% 4.9% 0.7% 0%

COMPILED OTOLARYNGOLOGY DATA:

DUTY HOURS

Duty hour question guide (PDF) for residents on ACGME website

Extremely

Very Somewhat

Slightly

Not at all

How sufficient is the instruction you receive from faculty and staff in your program?

59.7% 34.3%

5.3% 0.4% 0.1%

Thinking about the faculty overall, how interested are they in your residency education?

64.1% 28.6%

5.8% 1.6% 0%

Thinking about the faculty overall, how effective are they in creating an environment of scholarship and inquiry?

61.4% 30.4%

6.8% 1.4% 0%

COMPILED OTOLARYNGOLOGY DATA:

FACULTY

Consider faculty workshops Mentorship Giving feedback Teaching techniques

Resources GME offi ce Dean’s offi ce Faculty Development

Programs

FACULTY ISSUES

Extremely

Very Somewhat

Slightly

Not at all

How satisfied are you that your program treats your evaluations of faculty members confidentially?

62.5% 28.0%

5.5% 1.3% 0.4%

How satisfied are you with the way your program uses the evaluations that residents provide to improve the program?

55.0% 30.4%

10.5% 2.4% 0.6%

Overall, how satisfied are you with the written or electronic feedback you receive after you complete a rotation or major assignment?

50.4% 35.8%

10.8% 2.3% 0.7%

COMPILED OTOLARYNGOLOGY DATA:

EVALUATION

Confidentiality is compromised by nature of being a small program Consider pooling attending or rotation evaluations, then

distribute at the end of each yearEvaluations need to be meaningful

Shortening the form Requiring written comments Issues identified by residents should be followed up;

improvements should be documented

EVALUATION ISSUES

Extremely

Very Sometimes/somewhat

Slightly

Not at all

How satisfied are you with the opportunities your program provides for you to participate in research or scholarly activities?

59.6% 31.0%

7.9% 1.3% 0.2%

In your opinion, how often do your rotations provide an appropriate balance between your residency education and other clinical demands?

50.7% 37.8%

10.2% 1.2% 0%

How often has your clinical education been compromised by excessive service obligations?

0.7% 1.9% 15.8% 35.3%

46.3%

COMPILED OTOLARYNGOLOGY DATA:

EDUCATIONAL CONTENT

Time for research How protected is the time?

Balance between education and service Physician extenders – “non-educational” work Continually re-evaluate quality of didactics Faculty mentorship & teaching

EDUCATIONAL CONTENT ISSUES

Extremely

Very Sometimes/somewhat

Slightly/rarely

Not at all

How often do you work in interdisciplinary teams to care for patients?

64.8% 28.9%

5.6% 0.7% 0.1%

How satisfied are you with your program’s process to deal confidentially with problems or concerns residents might have?

58.8% 30.0%

8.5% 2.2% 0.5%

To what extent does your program provide an environment where residents can raise problems or concerns without fear of intimidation or retaliation?

64.9% 25.7%

6.8% 2.1% 0.4%

COMPILED OTOLARYNGOLOGY DATA:

RESOURCES

Interdisciplinary teams Tumor Board? Case conferences? Combined Grand Rounds? QM or QI teams?

Dealing with problems confidentially Institutional resources in the GME

offi ceDealing with problems without

fear of intimidation or retaliation Resident-only meetings

RESOURCE ISSUES

Which of the following best summarizes your opinion of your residency program?

OVERALL EXPERIENCE

Great

Good

Neutral

Negative

Very negative

0 20 40 60 80

NationalOtolaryngology

Explain and prepare residents Purpose of the survey Specific questions Can re-administer survey

internally Increase communication

Regular meetings with residents Regular meetings with faculty Lots of channels

Increase overall moraleWe are in good shape!

Take as opportunity to improve program

TAKE-HOME POINTS

Mount Sinai School of Medicine GME offi ce Scott Barnett, MD – Associate Dean for GME Paul Johnson

Ben Malkin, MD Associate Program Director, ENT, MSSM

THANK YOU