Can Unannounced Standardized PatientsAssess Professionalism and CommunicationSkills in the Emergency Department?Sondra Zabar, MD, Tavinder Ark, MSc, Colleen Gillespie, PhD, Amy Hsieh, MPA, Adina Kalet, MD,Elizabeth Kachur, PhD, Jeffrey Manko, MD, and Linda Regan, MD

AbstractObjectives: The authors piloted unannounced standardized patients (USPs) in an emergency medicine(EM) residency to test feasibility, acceptability, and performance assessment of professionalism and com-munication skills.

Methods: Fifteen postgraduate year (PGY)-2 EM residents were scheduled to be visited by two USPswhile working in the emergency department (ED). Multidisciplinary support was utilized to ensure suc-cessful USP introduction. Scores (% well done) were calculated for communication and professionalismskills using a 26-item, behaviorally anchored checklist. Residents’ attitudes toward USPs and USP detec-tion were also surveyed.

Results: Of 27 USP encounters attempted, 17 (62%) were successfully completed. The detection rate was44%. Eighty-three percent of residents who encountered a USP felt that the encounter did not hinderdaily practice and did not make them uncomfortable (86%) or suspicious of patients (71%). Overall, resi-dents received a mean score of 60% for communication items rated ‘‘well done’’ (SD ± 28%,range = 23%–100%) and 53% of professionalism items ‘‘well done’’ (SD ± 20%, range = 23%-85%). Resi-dents’ communication skills were weakest for patient education and counseling (mean = 43%,SD ± 31%), compared with information gathering (68%, SD ± 36% and relationship development (62%,SD ± 32%). Scores of residents who detected USPs did not differ from those who had not.

Conclusions: Implementing USPs in the ED is feasible and acceptable to staff. The unpredictability ofthe ED, specifically resident schedules, accounted for most incomplete encounters. USPs may representa new way to assess real-time resident physician performance without the need for faculty resources orthe bias introduced by direct observation.

ACADEMIC EMERGENCY MEDICINE 2009; 16:915–918 ª 2009 by the Society for Academic EmergencyMedicine

Keywords: standardized patients, graduate medical education, assessment, OSCE, professionalism,communication, assessment

W hat options exist for assessing communica-tion and professionalism skills? As resi-dency programs seek to comply with the

Accreditation Council for Graduate Medical Education(ACGME) Outcomes Project,1 robust modalities to eval-uate clinical performance and effectiveness of educa-tion are in high demand. The ACGME’s Toolboxcontains numerous tools for assessing communicationskills,2 but many of these rely on self-assessment ortrained observers present during patient encounters.Patient complaints and postvisit surveys are useful forobtaining information, but offer limited opportunityfor physicians to translate feedback into practicechange. Unannounced standardized patients (USPs)3–5

present a method of measuring physicians’ communi-cation and professionalism skills in a real practice set-ting without the artificiality inherent in observedstructured clinical exams (OSCEs).6–8

We hypothesized that USPs can provide a real-time,accurate alternative to direct observation and OSCEs.

ª 2009 by the Society for Academic Emergency Medicine ISSN 1069-6563doi: 10.1111/j.1553-2712.2009.00510.x PII ISSN 1069-6563583 915

From the New York University School of Medicine (SZ, TA,CG, AH, AK, EK, JM, LR), New York, NY; and The JohnsHopkins University School of Medicine (LR), Baltimore, MD.Received February 27, 2009; revision received May 21, 2009;accepted May 22, 2009.Presented at The Gold Foundation Symposium, ‘‘How Are WeTeaching Humanism in Medicine and What is Working?’’September 27–29, 2007, Chicago, IL; and the 9th Annual Inter-national Meeting on Simulation in Healthcare (IMSH), January10–14, 2009, Lake Buena Vista, FL.Supported by Picker Institute Challenge Grant 2007.Address for correspondence and reprints: Sondra Zabar, MD;e-mail: sondra.zabar@nyumc.org.

The purpose of this project was to 1) describe theprocess of conducting a USP program in an emergencydepartment (ED), 2) determine if implementing USPs inthe ED is feasible, and 3) present preliminary results ofa USP performance assessment.

METHODS

Study DesignThis was a prospective, nonrandomized, cohort studyto assess professionalism and communication abilitiesof emergency medicine (EM) residents using USPs.Informed consent was obtained from all participants.Research activities in this study were approved by theNew York University School of Medicine InstitutionalReview Board through a resident registry wherein resi-dents are asked to consent to allow inclusion of theireducational and performance data in a research data-base. Data, therefore, are reported only for those resi-dents for whom such consent was obtained.

Study Setting and PopulationThe Bellevue ED is a busy Level 1 trauma center at anacademic medical center in New York City. The ED seesapproximately 100,000 visits per year.

Fifteen EM residents in their second year of post-graduate training (PGY-2) participated in the EM Pro-fessionalism and Communication Training (EMPACT)Program. At the conclusion of the EMPACT training,residents were informed that they might be visited byUSPs during their subsequent time working in theED. However, residents were blinded as to the exactdate of the visit or patient complaint.

Study ProtocolLogistics. We required involvement from most EDstaff areas including nurses, attending physicians, medi-cal records (MRs), registration, informatics, and radio-logy. To ensure fidelity for each USP visit, we created apreexisting MR with a unique number, patient nameand identifying information, prior visits, and testresults. Each resident was scheduled to receive twoUSPs in urgent care (where residents’ schedules wererelatively predictable) during the 4 to 6 weeks after theEMPACT curriculum.

USP Scenarios. We used two USP cases previouslyvalidated in OSCEs, representing common ED chal-lenges and requiring only communication-based inter-ventions. In the first case (a misread x-ray), residentsneeded to educate an angry patient recalled for a mis-read x-ray (skills: delivering bad news, dealing with achallenging patient, accountability), and in the second(a repeat visitor), care for a dissatisfied patient withchronic pain who repeatedly uses the ED (skills: han-dling emotion, patient education, accountability).

USP Training. Eight actors were recruited. On aver-age, each received seven hours of training consisting of1) discussion of character and situation, 2) calibrationof emotional tone, 3) role play for standardization,4) practice with attending and chief residents for realism,5) review of ‘‘ground rules’’ for safety and nondetec-

tion, 6) review and practice with rating checklist, and7) preparatory observational visit to the ED. Actorswere compensated at a rate of $25 ⁄ hour for both train-ing and in-ED time.

USP Encounter. Unannounced standardized patientsmet project staff while residents attended a requiredconference. USPs were introduced to the triage nurse,the MR administrator, and the attending. The USPswere triaged per standard procedure.

During the encounter, USPs complied with any (non-invasive) exam and accepted all appointments andprescriptions, which were canceled postencounter. Ifthe resident insisted on any course of action that madethe USP feel unsafe, the USP was to ask for the attending,send a short message service (SMS) text message toproject staff, or simply leave the ED. Hospital billingcanceled the visit at the end of the day. Total time inthe ED was 1.5 to 4 hours ⁄ visit. Immediately followingthe encounter, the USP debriefed and completed abehaviorally anchored checklist that assessed residentskills and the USP’s satisfaction with the visit.

Post-USP Survey. At the end of the project, all EMresidents (including those who did not see a USP;n = 30) were surveyed about their attitudes towardUSPs using a four-point scale (1 = strongly disagree,4 = strongly agree) and open-ended questions. Todetermine detection rates, residents were asked if theyhad encountered a USP and if so to identify the USP’ssex and chief complaint.

Data AnalysisUnannounced standardized patients assessed residents’professionalism and communication skills and theirsatisfaction with the patient-centeredness9,10 of thevisit using a three-point scale: ‘‘not done,’’ ‘‘partiallydone,’’ and ‘‘well done.’’ Scores were calculated as thepercentage of well-done items (Table 1). Professional-ism and communication skills were scored from 13items and patient centeredness from eight items. Over-all recommendation ratings were obtained using afour-point scale. Reliability estimates (Cronbach’salpha ‘t’) and descriptive statistics (means, standarddeviations [SDs], and ranges) are reported. Correla-tions (Pearson’s r) between scores earned in the twoseparate cases are also reported to assess stability ofperformance.

RESULTS

Seventeen of 27 visits were successfully conducted andevaluated. Resident scheduling problems explainedmost incomplete encounters. Five residents were visitedby USPs from both cases, and seven residents from onecase.

USP DetectionSeven of 12 residents who encountered a USP pro-vided information on detection; four of their nineencounters were detected (44% detection rate). Five of18 residents who did not see a USP indicated that theydid (28% false-positive rate). One of the residents who

916 Zabar et al. • PROFESSIONALISM AND COMMUNICATION

reported a false detection reported ignoring thatpatient.

USP PerformanceThe reliability of scores (Table 1) suggests adequateinternal consistency (a > 0.60). Residents performedbetter in the misread x-ray case than in the repeatvisitor case in professionalism (70% vs. 35%, t = 2.81,p = 0.048) and patient-centeredness (66% vs. 40%,t = 1.96, p = 0.05). Communication (r = 0.73, p = 0.16)and recommendation scores (r = 0.81, p = 0.09) werehighly, albeit not significantly, correlated between thetwo cases, but professionalism (r = 0.24, p = 0.70), andpatient-centeredness were not (r = 0.08, p = 0.90), sug-gesting case content matters most in these domains.

Postevaluation SurveyEighty-three percent of residents who encountered aUSP felt that it did not hinder their daily practiceand did not make them uncomfortable (86%) or sus-picious of patients (71%). A minority of those resi-dents who encountered a USP felt that the encounterimproved their practice behavior (14%), made them

think more (29%), or led them to be more self-aware(43%).

DISCUSSION

Our results show that developing and implementing aUSP program in the ED is feasible and acceptable toresidents. Considering the drawbacks of OSCE assess-ment and direct observation, combined with increasingdemands on faculty time and decreasing funding, USPsmay offer an objective, cost-effective method for evalu-ating accurate practice skills.

The biggest challenge faced while implementing theUSP program was the unpredictability of the ED. Occa-sionally, USPs were mistakenly examined by anotherresident. Both content (highly trained SP, realisticcases) and logistic factors (dedicated program coordi-nator, electronic MRs, team collaboration) are neces-sary for successful integration. Total cost, in terms ofboth time and money, is likely greater up front, withdecreased workload, time, and expense as USPs andstaff become trained. Further study of the costs isneeded.

Table 1Resident Performance with USPs

Domains of Assessment Items Mean, % ±SD, % Range, % Reliability

Communication* 60 28 23–100 0.91Information gathering Used appropriate questions

Clarified informationAllowed to talk without interrupting

68 36 0–100 0.82

Relationship development Communicated concernNonverbal enhanced communicationAcknowledged emotionsWas accepting ⁄ nonjudgmentalUsed words you understood

62 32 20–100 0.85

Education and counseling Asked questions to see what you understoodProvided clear explanationsCollaborated with you in identifying next steps

43 31 0–100 0.78

Professionalism* 53 20 23–85 0.62Accountability Disclosed error

Personally apologizedTook responsibility for situation

49 23 0–80 0.60

Manage difficult situation Avoided assigning blameMaintained professionalism

91 16 60–100 0.85

Giving bad news Prepared you to receive newsGave you opportunity to emotionally respondProvided appropriate next steps

42 34 0–83 0.63

Treatment plan and management Assessed resourcesArranged for follow-upDiscussed plan

50 39 0–100 0.66

Patient Centeredness* Fully explored my experienceExplored my expectationsCame to an agreementTook a personal interest in meEarned (regained) my trustAcknowledged impact of errorDidn’t make me feel wasting timeI was given enough information

43 29 0–75 0.91

Recommendation� 2.3 0.9 1.0–3.5 0.90

n = 12 residents, 17 visits; reliability assessed with Cronbach’s alpha. Four-point scale: 1 = not recommend; 2 = recommend withreservations; 3 = recommend; 4 = highly recommend.USPs = unannounced standardized patients.*Mean percentage of items rated as ‘‘well done’’ (not, partly, or well done).�’’Would you recommend this physician to a family member of friend?’’

ACAD EMERG MED • September 2009, Vol. 16, No. 9 • www.aemj.org 917

Even with a high detection rate, residents reportedvalue in the USP program for learning and patient care.It is possible that informing residents that USPs wouldbe visiting them in the ED improved performance.More importantly, the majority of residents did not feelthat the possibility of encountering a USP had any neg-ative impact on their daily practice, suggesting thatUSPs in the ED will not risk real patient safety. Thecase of the resident who reported ignoring a patientthought to be ‘‘unannounced’’ represents an unantici-pated and anomalous professionalism issue, we believe,not causally related to the use of USPs; it demonstrateshow USPs can provide useful information to programdirectors.

LIMITATIONS

There was a small sample size, with a relatively largeproportion of failed USP visits. However, the failurerate improved as the project progressed. Even with oursmall numbers, it appears that two cases and the itemson the behaviorally anchored checklist can discriminateresidents based on their communication skills.

CONCLUSIONS

With the ACGME placing greater importance on evalua-tion of patient outcomes, we believe that our projectrepresents a new way to assess real-time resident perfor-mance. Despite being time-consuming and subject to theunpredictability of the ED, implementing unannouncedstandardized patients in the ED is feasible and acceptableto staff. Future comparison of unannounced standard-ized patients with observed structured clinical examscores will enable educators to determine how well thesemethods assess performance in actual practice.

References

1. Accreditation Council for Graduate Medical Educa-tion (ACGME). ACGME Outcome Project. Available

at: http://www.acgme.org/outcome/comp/compCPRL.asp. Accessed Sep 20, 2008.

2. Accreditation Council for Graduate Medical Educa-tion, American Board of Medical Specialties.Outcome Project Toolbox of Assessment Methods.Available at: http://www.acgme.org/outcome/assess/toolbox.asp. Accessed Jun 20, 2009.

3. Gorter S, Scherpbier A, Brauer J, et al. Doctor-patient interaction: standardized patients’ reflec-tions from inside the rheumatological office.J Rheumatol. 2002; 29(7):1496–500.

4. Kravitz RL, Epstein RM, Feldman MD, et al. Influ-ence of patients’ requests for direct-to-customeradvertised antidepressants: a randomized controlledtrial. JAMA. 2005; 293(16):1995–2002.

5. Ozuah PO, Reznik M. Using unannounced standard-ised patients to assess residents’ professionalism.Med Educ. 2008; 42(5):532–3.

6. Fiscella K, Franks P, Srinivasan M, Kravitz RL,Epstein R. Ratings of physician communication byreal and standardized patients. Ann Fam Med. 2007;5(2):151–8.

7. Talente G, Haist SA, Wilson JF. The relationshipbetween experience with standardized patientexaminations and subsequent standardized patientexamination performance: A potential problem withstandardized patient exam validity. Eval Health Prof.2007; 30(1):64–74.

8. Srinivasan M, Franks P, Meredith LS, Fiscella K,Epstein RM, Kravitz RL. Connoisseurs of care?unannounced standardized patients’ ratings of phy-sicians. Med Care. 2006; 44(12):1092–8.

9. Marshall GN, Hays RD. The Patient SatisfactionQuestionnaire Short Form (PSQ-18). RAND Corpo-ration, Paper P-7865. Available at: http://www.rand.org/pubs/papers/P7865/. Accessed Jun 20, 2009.

10. Elwyn G, Edwards A, Wensing M, Hood K, AtwellC, Grol R. Shared decision making: developing theOPTION scale for measuring patient involvement.Qual Saf Health Care. 2003; 12:93–9.

918 Zabar et al. • PROFESSIONALISM AND COMMUNICATION