Designing and Conducting Simulation-BasedResearch

abstractAs simulation is increasingly used to study questions pertaining to pe-diatrics, it is important that investigators use rigorous methods to con-duct their research. In this article, we discuss several importantaspects of conducting simulation-based research in pediatrics. First, wedescribe, from a pediatric perspective, the 2 main types of simulation-based research: (1) studies that assess the efficacy of simulationas a training methodology and (2) studies where simulation is usedas an investigative methodology. We provide a framework to help struc-ture research questions for each type of research and describe illus-trative examples of published research in pediatrics using these 2frameworks. Second, we highlight the benefits of simulation-based re-search and how these apply to pediatrics. Third, we describesimulation-specific confounding variables that serve as threats tothe internal validity of simulation studies and offer strategies to mit-igate these confounders. Finally, we discuss the various types of out-come measures available for simulation research and offer a list ofvalidated pediatric assessment tools that can be used in futuresimulation-based studies. Pediatrics 2014;133:1091–1101

AUTHORS: Adam Cheng, MD,a Marc Auerbach, MD, MSc,b

Elizabeth A. Hunt, MD, MPH, PhD,c Todd P. Chang, MD,d

Martin Pusic, MD, PhD,e Vinay Nadkarni, MD,f and DavidKessler, MD, MScg

aUniversity of Calgary, Section of Emergency Medicine,Department of Pediatrics, Alberta Children’s Hospital;bDepartment of Pediatrics, Section of Emergency Medicine,Yale University School of Medicine, New Haven, Connecticut;cDepartments of Anesthesiology, Critical Care Medicine andPediatrics, Johns Hopkins University School of Medicine,Baltimore, Maryland; dDivision of Emergency Medicine, Children’sHospital Los Angeles, Los Angeles, California; eOffice of MedicalEducation, Division of Educational Informatics, New YorkUniversity School of Medicine, New York, New York; fDivisionof Anesthesia and Critical Care Medicine, Children’s Hospital ofPhiladelphia, University of Pennsylvania School of Medicine,Philadelphia, Pennsylvania; and gDepartment of Pediatrics,Division of Pediatric Emergency Medicine, Columbia UniversityCollege of Physicians and Surgeons, New York, New York

KEY WORDSsimulation, pediatric, research, education, study design

ABBREVIATIONSSBEI—simulation-based educational interventionsSBR—simulation-based research

Drs Cheng, Auerbach, and Kessler conceptualized the paper,contributed intellectual content, drafted the initial manuscript,and reviewed and revised the manuscript; Drs Hunt, Chang,Pusic, and Nadkarni contributed intellectual content andreviewed and revised the manuscript; and all authors approvedthe final manuscript as submitted.

www.pediatrics.org/cgi/doi/10.1542/peds.2013-3267

doi:10.1542/peds.2013-3267

Accepted for publication Jan 23, 2014

Address correspondence to Adam Cheng, MD, FRCPC, AssociateProfessor, University of Calgary, KidSim-ASPIRE Research Program,Section of Emergency Medicine, Department of Pediatrics, AlbertaChildren’s Hospital, 2888 Shaganappi Trail NW, Calgary, Alberta,Canada T3B 6A8. E-mail: adam.cheng@albertahealthservices.ca

PEDIATRICS (ISSN Numbers: Print, 0031-4005; Online, 1098-4275).

Copyright © 2014 by the American Academy of Pediatrics

FINANCIAL DISCLOSURE: The authors are members of theINSPIRE network, which receives infrastructure support fromthe Laerdal Foundation for Acute Medicine, project funding fromRbaby Foundation, and funding for biannual meetings from theSociety for Simulation in Healthcare and the InternationalPediatric Simulation Society.

FUNDING: No external funding.

POTENTIAL CONFLICT OF INTEREST: The authors have indicatedthey have no potential conflicts of interest to disclose.

PEDIATRICS Volume 133, Number 6, June 2014 1091

STATE-OF-THE-ART REVIEW ARTICLE

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

Health care simulation can be definedas a tool, device and/or environmentwith which the learner or subject in-teracts to mimic an aspect of clinicalcare.1 The technologies used to enablehealth care simulation include a widevariety of products anddevices, includingmannequins (with varying degreesof realism), computer/screen-basedsimulators, inert animal products,task trainers, and human cadavers.1

This technology, when applied for train-ing health care providers, is created oradapted to help address practical clinicalproblems.1 The field of pediatric simu-lation has grown rapidly in the pastdecade, both as an educational inter-vention2–9 and as an investigativemethodology.10–14 Recent articles havedescribed important attributes of simu-lation research,15 simulation-based edu-cational interventions (SBEI),16 and thetypes of research studies that should beconducted to advance the science ofsimulation.17 Although the quantity ofsimulation-based research (SBR) is onthe rise, the quality is highly variable.1,18 Ina recent systematic review of simulation-based educational research, 22.5% ofstudies had a randomized controlledstudy design, 15.1% were multicenterstudies, and only 5.3% reported patientand/or health care outcomes.1 Specificstrategies to improve the quality of SBRare not well described in the literature.An emphasis on study design and op-timizing research methodology is nec-essary to optimize the impact of futureSBR in pediatrics.

Thisarticleaimstodescribe theeffectiveuse of simulation for pediatric research.First we present 2 categories of SBR andprovide a framework to help structureresearch questions for each type ofresearch. Second,we provide examplesfrom the field of pediatrics whilehighlighting advantages of SBR. Nextwediscuss the key simulation-specific var-iables that must be carefully controlledwhen conducting SBR. Lastly, we discuss

various types of outcome measures forsimulation research.

CATEGORIES OF SBR

Research on the Efficacy ofSimulation as a TrainingMethodology

Researchabout simulationasa trainingmethodology examines whether the spe-cific features of simulation experiencesadd to overall educational effectiveness. Asystematic review by Issenberg high-lighted that high-fidelity medical sim-ulations (eg, simulators that changeand respond to the user) are educa-tionally effective and that simulation-based education complements medicaleducation in patient care settings.19

A recent systematic review and meta-analysis noted that compared with nointervention (eg, a control group or pre-intervention assessment), simulation-based training was effective in improvingthe knowledge, skills, and behaviors ofhealth care professionals.1 In pediatrics,simulation has been effectively used toteach neonatal20–22 and pediatric re-suscitation,3,6,7 crisis resource manage-ment,8,9,20,23–24 anesthesia,25–27 proceduralskills5,28–30 (eg, gynecology examination,airway management), and surgicalskills31–33 (eg, endoscopy and minimallyinvasive surgery). Although the scope ofsimulation-based education in pediatricsis growing, few comparative studies havehelped to clearly define the optimal in-structional design features of effectivepediatric SBEI.

Theresearchagendahasclearly shiftedfrom “if” simulation works to examin-ing “who, what, when, where, why andhow.” Cook et al characterized featuresof effective SBEI.34 However, a key ques-tion that remains largely unansweredfor simulation educators is: How do SBEIneed to be modified for different educa-tional contexts? Comparative research iswarranted to explorewhich instructionaldesign features have the optimal impactfor specific learning objectives, learner

groups, and learning environments. Ex-amples of comparative pediatric studies,using the various instructional designfeatures as a framework, are describedin Table 1.

Research Using Simulation as anInvestigative Methodology

Research using simulation as an in-vestigative methodology leverages thestandardization provided by simulationto answer diverse research questionsthat otherwise could not be answeredfeasibly, safely, ethically, or in a timelyfashion in clinical settings. The simu-lated environment is used as an exper-imental model to study factors affectinghuman and systems performance inhealth care. Mannequin-based simula-tion has been particularly useful in thiscontext. In this form, a mannequin con-nected to a computer that controls itsvital signs and physical findings pro-vides health care providers a realisticclinical experience. Theuseofmannequin-based simulation allows the researcherto have complete control over nearlyevery aspect of the clinical environment,including but not limited to the type,location, and size of equipment; the ageand clinical status of the patient; and thecomposition, number, and experience ofthe health care providers.

SBR studies in this category can begrouped based on the performance-shaping factors that can enhance ordegradeperformanceandsubsequentlyimpact patient safety and risk.35,36 Thevarious performance shaping factorsthat allow for a systematic approach toimproving safety and error reduction inclinical medicine include (1) individuals(eg, fatigue, stress, experience), (2)teams (eg, team structure, communica-tion), (3) work environment (eg, noiselevels, resource availability), (4) technol-ogy (eg, use of clinical decision supportor electronic health records), (5) systemsfactors (eg, work schedule and flow,policies, and procedures), and (6) patient

1092 CHENG et al

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

factors (eg, clinical presentation).36 Byusing simulation as an investigativemethodology, investigators can system-atically identify latent safety threats, testnew technology and protocols, and im-prove the health care environment with-out any potential for harm to real patients.Lessons learned from research per-formed in the simulated environmentcan then be applied to the real clinicalenvironment to optimize patient careprocesses and outcomes. Table 2 pro-vides examples of studies that use sim-ulation as an investigative methodology.

ADVANTAGES OF SBR

The use of SBR in pediatrics confersseveraldistinctadvantages.Unlikeclinicalresearch in which patient presentationsare variable and unpredictable, SBRallows for standardized patient pre-sentations that can be provided ondemand. It is also permits the mostimportant clinical variables, apart from

the variable of interest, to be carefullycontrolled and accounted for. Stan-dardization of the simulated environ-ment for research can potentially beachieved provided the research teamhascarefullyaccounted for themajorityof the confounding variables (clinicaldiagnosis, clinical progression, etc).The authenticity of the simulated envi-ronment is particularly important whenit is being used as a surrogate for thereal clinical environment. Researchersshould ensure that, to the best of theirability, all elements in the real clinicalenvironment that could affect participantperformance are also appropriately rep-resented during the simulations.36 Be-cause it is not always possible to controlevery factor that could affect participantperformance during a simulation (eg,institutional culture), optimizing authen-ticity in the environment can often be bestachieved by using a real clinical space(eg, in situ simulation) to conduct thesimulations. Another major advantage is

that recruitment of individuals and/orteams of pediatric health care pro-fessionals can be scheduled accordingto convenience, thus allowing for morepredictable recruitment. Additionally,there is no risk for patient harm whenusing simulation to test new technology,protocols, or clinical spaces, enablingthe researcher to allow a study subjectto make patient care errors, such thatcontributing factors can be fully ob-served and analyzed. Much like clinicalresearch, SBR also has some challenges.These challenges are listed, along witha summary of benefits, in Table 3.

KEY ELEMENTS OF SBR DESIGN

Researchassessing the effectivenessofsimulation as a training methodologyshares similar design considerationswith traditional research in medicaleducation. In a recent article, Cook andBeckman outline important issues indesigning experimental research ineducation.37 One of the key issues theyhighlighted was the importance of de-scribing both the educational inter-vention and the comparison group insufficient detail to allow replication inother contexts. Thus, it is important tofirst address potential threats to theinternal validity of traditional educa-tion research studies, such as subjectcharacteristics, selection bias, history,instrumentation, testing, location, par-ticipant attitude, and implementation.37

In addition, for research assessingsimulation as a training methodology,several distinct elements of study design(ie, simulation-specific confounding var-iables), including simulator selection,scenario design, confederates, realism,debriefing, and video capture/reviewmust be carefully controlled to mitigatethreats to the internal validity of the re-search study.

Many of the same simulation-specificconfounding variables described abovemay be important for research usingsimulationasaninvestigativemethodology.

TABLE 1 Examples of Pediatric Studies With Simulation as the Subject of Research: Examining theEffectiveness of Simulation as an Educational Intervention

Instructional Design Featureof the Simulation Intervention

Description of Intervention

Clinical variation High-fidelity simulation (eg, simulator with audio speakersenabled and physical signs visible, audible, and palpable)versus standard mannequin (eg, simulator with audiodisconnected, and physical signs not audible or palpable)for pediatric life support training (6 clinical scenarios wereintegrated into the educational intervention to ensureclinical variation)3

Cognitive interactivity Telesimulation for teaching intraosseous insertion indeveloping countries4

Curricular integration A simulation-based program (integrated into the existingcurriculum) for teaching residents the pediatric gynecologyexamination5

Distributed practice A simulated pediatric mock code program with mock codesrandomly called over a 48-mo period of time7

Feedback Scripted vs nonscripted debriefing for pediatric resuscitationeducation2

Group practice Emergency department personnel managing pediatric traumaas teams in their own environment, in situ simulation-basedteam training8

Multiple learning strategies Multidisciplinary pediatric trauma team training usinghigh-fidelity trauma simulation (curricular integration,cognitive interactivity, clinical variation, distributed practice,and feedback all part of the educational intervention)9

Repetitive practice Repetitive pediatric simulation resuscitation training (multiplescenarios) vs standard pediatric simulation resuscitationtraining (1 scenario)6

STATE-OF-THE-ART REVIEW ARTICLE

PEDIATRICS Volume 133, Number 6, June 2014 1093

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

Additionally, these confounding varia-bles are important in multicenter re-search studies in which standardizationof the study protocol is of paramountimportance (eg, high likelihood ofvariability between sites). These issuesshould also be carefully considered forSBR research in other potential studygroups(eg,adultstudies, interprofessionalstudies). Table 4 provides an overviewof standardization strategies for threatsto the internal validity of SBR studies.Wedescribe how each of these simulation-specific confounding variables affectSBR in pediatrics, in which issues re-lated to patient age, parental presence,equipment size, and disease typemay all influence study design andstandardization.

Simulator Selection

Because several options for infant andpediatric simulators exist, researchersmust consider the functionality andfeatures of the simulator when de-signing the study. The functionality ofcommercially available infant and pe-diatric simulators is highly variable,with differences in their ability to sim-ulate eye opening and closing, locationand quality of pulses, size and compli-ance of lungs and chest, and design and

anatomy of the airway. Studies usingscenarios and mannequin-based sim-ulation may require a certain level offunctionality and realism to accuratelysimulateacertainmedicalproblem.Forexample, if a study isdesigned toassessthe impactofareal-timefeedbackdeviceon the depth of chest compressions, itwouldbe important toselect a simulatorwhich, at a minimum, allows for chestcompressions to a depth greater thanthat required by resuscitation guide-lines (eg, at least 5 cm for children oradults). Similarly, a study to assess theimpact of a trauma checklist on themanagement of head injury requires asimulator that couldmimic deteriorationin level of consciousness in which theeyesareabletoopenandcloseandpupilscan react to light. Failure to consider thefunctionality of the simulator may in-fluence the relevanceandaccuracy of thestudyoutcomes. If a particular function iscrucial to the study, it should be men-tioned in the methodology prominently.The most logical strategy would be tochoose the same simulatorwith all of thedesired functionality for all researchsessions. For multicenter research, thismay have resource implications if not allsites have the desired simulator avail-able, that is, somesitesmaynotbeable to

enroll subjects if therequiredsimulatoris integral to thestudydesignandcannotbe made available to them.

Scenario Design

For either type of SBR, scenarios shouldbe developed that can be delivered ina uniform fashion from participant toparticipant, group to group, and, ifmulticenter, from institution to in-stitution. For example, a research studyto test the impact of an SBEI on man-agementofpediatricanaphylaxisrequiresthe scenario be standardized in a fashionthatwillensureeachgroupofparticipantsis exposed to a case of similar difficulty,with similar challenges in decision-making and clinical care. Allowing toomuch variation in case delivery wouldchange the intervention of interest oradd unnecessary confounders. To en-sure scenarios are delivered in a stan-dard fashion, researchers can considervariousstrategies, theselectionofwhichis dependent on the research question,goal of the study, participant charac-teristics, and outcome measures: (1)control the duration of the scenario bylimiting the overall time (ie, scenario isstopped at a certain time independentof participant actions/interventions)and/or setting transitions from one

TABLE 2 Examples of Studies With Simulation as the Environment for Research: Simulation as an Experimental Model to Study Factors Affecting Humanand Systems Performance in Health Care

Performance Shaping Factor Focus of Simulation Research Example of Study

Individuals Assessing and describing the relationship betweenindividual factors and performance

A simulation study of rested versus sleep-deprivedanesthesiology residents evaluated psychomotorperformance, mood, and sleepiness (adult study)48

Teams Assessing and describing the relationship betweenteam processes and team performance

In situ simulation was used to identify latent safetythreats (eg, team member responsibilities, providerworkload) among health care teams in a pediatricemergency department10

Environments Assessing the impact of the surrounding environmenton performance

Simulation was used to identify latent safety threats(eg, resources, equipment) in a new pediatricemergency department11

Technological factors Evaluating the effect of technology on performance Comparison of a video-laryngoscope versus traditionallaryngoscopy in pediatric intubation12

Systems factors Simulation used to model and understand system-leveloperations

Discrete event simulation (computer modeling) used toconstruct a patient flow model in a pediatricemergency department13

Patient factors Simulation used to describe individual and teamperformance for specific patient conditions

Simulated cardiopulmonary arrest in a pediatric patientwere used to identify delays and errors incardiopulmonary resuscitation by pediatric residents14

1094 CHENG et al

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

clinical state to the next at predefinedtimes, independent of subject inter-ventions (eg, normotensive to hypo-tensive at 5 minutes). Doing so allowsresearchers to see if certain tasks aredone in a predefined time frame, withthe benefit of standardizing scenarioduration. Theunfortunate consequenceof this strategy is that sometimesconceptual realism is sacrificed(eg, patient spontaneously converts fromventricular tachycardia to sinus rhythmwithout intervention). (2) Alternatively,researchers can control the responsesof the simulated patient by settingtransitions from one clinical state toanother based on subject interventionsand independent of time (eg, bloodpressure changes from normotensive tohypotensive if 20 mL/kg normal salinefluid bolus is not given in the first 5minutes). Doing so allows clinical pro-gression based on participant inter-ventions (ie, high conceptual realism),but the downside is that the duration ofthe scenario may be highly variablefrom group to group. (3) Finally,researchers can control confederatebehaviors by clearly standardizingverbal, audio or visual cues that areprovided to confederates and facili-tators (eg, capillary refill, level of con-sciousness). These cues can be tied toparticipant actions/inaction, patienttransitions in physiology, or certaintime points in the scenario. During SBR,improvisation must be minimized forconfederates and facilitators and onlyused to maintain standardization andrealism of the scenario. Careful reviewof the scenario template and training ofscenario facilitators is recommendedto establish reliability. Pilot testingscenarios before starting a researchstudy will help investigators identifyand correct potential pitfalls beforeenrollment begins. This is particularlyimportant for multicenter research, inwhich sites will be using different re-search coordinators. Pilot testing pro-vides an opportunity to train research

TABLE 3 Benefits and Challenges of Simulation Research

Benefits Example

Recruitment: individuals and teams can bescheduled for recruitment by convenience; theresearch is not dependent on a specific set ofpatients being available

Recruitment for a study assessing the impact of teamtraining on resuscitation performance can bescheduledaccording toprovideravailabilityandnotpatient availability

Efficiency: simulated patients can be created andscheduled “on demand,” thus allowing for studyof rare conditions

A study assessing the impact of a video-laryngoscopeon intubation success rates in pediatric difficultairway patients12

Standardization: the clinical context andenvironment can be replicated to ensureconsistency across providers, team, andinstitutions

A standardized “home” environment can be createdfor a study using simulation to train families howto manage their child’s seizures at home

Ease of collaboration: research can be conductedacross multiple sites provided the simulatedresearch environment is standardized (easierto standardize simulated environments thanactual clinical environments)

A study assessing the impact of a scripted debriefingfor pediatric providers was conducted acrossmultiple sites in North America2

No risk of patient harm: interventions can be testedwithout any risk of harm to real patients

A study assessing the impact of a new CPR feedbackdevice in pediatric cardiac arrest can be tested onsimulated patients before approval for use on realpatients

No concerns related to protected healthinformation

A study can be conducted on sensitive topics such aschild abuse using simulated patients and will notinvolve protected information

Multiple options for outcome measures: data canbe collected from the subjects, extracted fromthe simulator, captured by video/audio, orcollected in the real clinical environment, someof which are difficult to capture in clinicalresearch

Inanongoingstudyassessing thequality ofCPRduringpediatric cardiac arrest, quality of CPR data arecollected from the simulator, team performance israted via video review, and subjective data arecollected via semistructured interviews

Challenges Example

Authenticity: the simulated clinical situation maynot be authentic enough to immerse the subjectin the clinical context (eg, researchenvironment, patient, clinical monitoring,clinical equipment, clinical team members,family members, etc)

In a study using a case of pediatric septic shock, thesimulator is unable to produce key physicalfeatures such as delayed capillary refill, coolextremities, mottled skin, and poor color, thuspotential influencing learner behaviors

Physiology: standardized scenarios may sufferfrom lack of the expected physiologic variabilityof a real patient; this may be undesirable insome research (eg, subtle signs of neurologicimpairment)

Variability in heart rate after specific interventions(eg, fluid bolus, inotropes) in a pediatric patientwith hypovolemic shock will need to bepreprogrammed or controlled by the facilitator toensure conceptual realism

Recruitment: recruiting health care professionalsto participate as subjects in research can bedifficult without support from leadership and/or funding

Recruitment of subjects for a team training studyrequiring 5 providers per session is challengingwithout leadership supporting methods ofreleasing providers to attend sessions

Cost: SBR requires simulators, space, time, andexpertise, which can be a significant capitalinvestment

A multicenter study using a pediatric simulator andvideo capture requires all sites to have the sameequipment and appropriate expertise

Best practices: a lack of best practices forreporting SBR makes it difficult to assimilateresults from various studies

A recent systematic review of SBR describes themethodologic limitations and variability ofsimulation studies1

Outcome measures: insufficient work on thetransfer of SBR to the real clinical environment;more work needs to demonstrate impact onpatient care and outcomes

A recent systematic review of SBR highlights the smallproportion of studies with real patient outcomes1

Funding: insufficient funding opportunitiescompared with clinical research

Many SBR projects are unfunded. More robustpediatric simulation studies demonstratingimprovements in care and patient outcomes willhelp increase opportunities for funded simulationresearch

CPR, cardiopulmonary resuscitation.

STATE-OF-THE-ART REVIEW ARTICLE

PEDIATRICS Volume 133, Number 6, June 2014 1095

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

facilitators ahead of time and for the re-search team members to share theirexperiences and struggles and offersuggestionsforstreamliningtheresearchprocess.Sharingvideosofpilotruns(bothsuccesses and failures with descriptionsof lessons learned) allows sites to havea sharedmentalmodel of exactly how thescenarios should be managed.

Confederates

Confederates, or actors, can be used inSBR to increase realismandhelp create

and/ormanipulate a situation for studypurposes. In adult studies, confederatesare used in the role as members of thehealth care team or as the patient. Inpediatric research, confederates can beintegrated into the simulated environ-ment as family members or caregiversto enhance pediatric-specific aspects ofclinical care, or children (in selectedcircumstances) can be recruited asconfederates to play the role of the sickpatient. In contrast to adult studies, theuse of real children to play the role of

a sick patient may be at times imprac-tical (or impossible) because youngerchildren as less likely to adhere to thepredefined confederate role orareunableto reliably reproduce desired physicalfindings (eg, tachypnea). This limitationcreates an exaggerated reliance on sim-ulation technology in pediatrics. As such,the pros and cons of using a child asaconfederateshouldbecarefullyweighed,and the relative benefits of using a simu-lator as the patient should be consideredbefore making a final decision.

TABLE 4 Standardization Strategies to Mitigate Threats to Internal Validity of SBR Studies

Threat/Potential Confounding Variable Description Standardization Strategy

Simulator selection The simulator selected does not do what the researchrequires or different simulators are used fordifferent participants

Ensure simulator has desired functionalityUse the same simulator for all sessionsAccount for variation in simulator choice in analyses

Scenario design The research scenario is not designed and implementedin a standardized fashion, and, as a consequence,study participants receive a different simulatedexperience

Preset or limit the duration of the scenarioSet transitions from one clinical state to the next atpredefined times or based on participant interventions(or lack thereof)

Standardize verbal, audio and/or visual cuesDevelop and review scenario templatePilot run scenarios to learn potential range of subjectactions and develop preplanned responses

Train research facilitatorsConfederates Confederates, or actors, do not act or portray their role

in a standardized fashion, thus potentially influencingthe behavior and actions of participants

Careful scripting of confederate roles (minimizeimprovisation)

Confederate cue cardsTrain confederates (eg, training video, pilot run)

Realism: The degree of physical, conceptual and emotional realismis not carefully controlled for each participant/groupPhysical realism

Use the same simulator for all sessions. Standardizethe environment: same equipment type and location,identical human resources, same size of room, samenoise level. Orient participants to the simulator andsimulated environment (eg, scripted orientation)

Conceptual realism See “Scenario design”Emotional realism See “Confederates”

Debriefing The debriefing portion of the simulation is not standardizedin a fashion that accounts for the various features of thedebriefing process.

Standardize debriefing session by controlling for the5 Ws of debriefing researchWho (debriefer characteristics)What (method of debriefing/content of debriefing)When (timing of debriefing)Where (environment for debriefing)Why (debriefing theory)

Video capture and review Video is not captured and/or reviewed in a standardizedfashion

Determine ideal video angle and number of viewsrequired to capture behaviors of interest

Determine if video capture of monitor displaying vitalsigns is necessary

Ensure adequate audio capturePilot run

Study outcomes Data elements for study outcome measures are notcollected in a standardized or reliable fashion

Implement strategies to make data collection asstandardized and reliable as possible: pilot cameraangles, study sensitivity and reliability of simulatorsensors, review data shortly after capture to identifyany problems before further enrollment, train andcalibrate video reviewers or other conducting datacollection and abstraction, calculate interraterreliability

1096 CHENG et al

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

As an example of howconfederatesmaybe used in research, an SBEI may beused to teach residents how to com-municate with family members, andconfederates could be scripted to playthe role of parents who are interactingwith the participants. The use of con-federates requires careful scripting ofconfederate roles, which can be tai-lored to address the research question(eg, issues of health literacy, culturefactors in pediatrics, delivering badnews). Unfortunately, no research todate has described the ideal way totrain confederates for SBR, althoughthere are descriptions of multiple meth-ods used in a single study.38 Strategiesthat can be used to orient confederatesto their roles include the following: (1)development of a scenario script ortemplate with detailed description ofconfederate roles, (2) confederate cuecards that can be used as a quick ref-erence during the scenario, (3) confed-erate training videowith expertmodelingof desired confederate actions, and (4)confederate training session with pilotresearch sessions prior to initiation ofthe study. During pilot sessions, inves-tigators will be able to see how par-ticipant and confederate behaviorstend to deviate from expected, thusallowing time to revise the study protocoland supporting materials to be moreresilient to the variability associatedwithhuman actors and participants. Carefulconsideration of strategies to standard-ize confederate behaviors in multicenterresearch is particularly crucial; individ-uals selected to be confederates maydiffer in background, experience, andexpectations.

Realism

Several ways of categorizing simulationfidelityorrealismhavebeendescribed.39,40

Although the impact of realism on thequality of simulation-based pediatric ed-ucation is controversial,2,41 investigatorsshould be attentive to the importance ofrealism when running simulation sce-

narios for research purposes. Enhancedlevels of realism help to immerse par-ticipants in the simulated experience,whereas a lower level of realism maylead to disengaged participants. A vari-able level of realism from scenario toscenario can introduce a confoundingvariable that may potentially affect theway individuals or teams perform. Whendesigning a scenario for SBR, there are3 important components of realism toconsider.39,40 “Physical realism” refersto the physical properties of the sim-ulation mannequins and environmentused to run the scenario. Standardizingthe environment involves providing thesame equipment and human resour-ces, as well as positioning the equip-ment in the same location to which theparticipants are accustomed and inthe same fashion for all participants.While doing so may help to achievestandardization among groups and/orsites (eg, in amulticenter study), it mayalso systematically introduce a bias thatfavors participants from one institutionwhere, for example, the resuscitationcart is placed in the exact spot they areused to in the real clinical environment.Furthermore, replicating certain noisesor distractors (eg, phone call or page)typically found during real patient caremay help to promote standardization butalso inadvertently introduce a confound-ing variable (eg, one institution typicallyhas less ambient background noisecompared with another). As such,while researchers attempt to achievecomplete standardization of the physi-cal environment, they must also con-sider the introduction of confoundingvariables when doing so. One effectivestrategy is to orient all subjects to thefeatures of the simulator and the physi-cal environment and effectively removingunfamiliarity with the simulator or spaceas a potential confounder. This can beachieved by providing a scripted ori-entation to the research environment.“Conceptual realism” refers to the theory,meaning, concepts, and relationships

attached to each simulated scenario.39

Specifically, conceptual realism involvesclinical authenticity with “if–then”relationships presented during thesimulation,39 such as, “If fluid is givenfor hypovolemic shock, then the bloodpressure should increase.” A consistentdegree of conceptual realism reliesheavily on carefully designed scenariosand facilitators who are familiar withthe scenario. Finally, “emotional re-alism” relates to the feelings that areevoked in subjects as a result of par-ticipating in the simulation.39 Manag-ing the degree of emotional realism insubjects can be difficult but is especiallyimportant when individual or team per-formance is an outcome measure. Thedegree and nature of interaction be-tween subjects and confederates canoften have a strong impact on emo-tional realism (eg, a confederate play-ing the role of a parent starts cryingduring the scenario in an unscriptedmanner); this must be understood byresearch confederates, who should becarefully scripted in the manner de-scribed earlier.

Debriefing

Studies assessing the efficacy of simula-tion as a training methodology shouldcarefully consider the relative value ofdebriefing as part of the overall learningexperience.19 Conversely, many studiesusing simulation as an investigativemethodology may not involve debrief-ing at all. Although debriefing has beencharacterized as the most importantelement of simulation-based education,failure to standardize the debriefingintroduces a major threat to the validityof any SBEI. A recent review of thedebriefing literature outlined the keycharacteristics of debriefing as the 5 Wsof debriefing research: who (debriefercharacteristics), what (content andmethods of debriefing), when (timing),where (environment), andwhy (theory).42

Each of these debriefing characteristicsshould be carefully standardized and

STATE-OF-THE-ART REVIEW ARTICLE

PEDIATRICS Volume 133, Number 6, June 2014 1097

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

reported when assessing simulationas an educational intervention. For ex-ample, if using multiple debriefers ina study, each debriefer should have thesame level of expertise and should betrained to use the same method ofdebriefing. This is particularly crucialwhen 1 element of the debriefing is theintervention of interest in the study.Standardization of the other debriefingcharacteristics will allow for isolationof the specific debriefing variable (eg,location of debriefing: in resuscitationroom vs in separate debriefing room).

Video Capture and Review

Many SBR studies use video to captureindividual or team performance andthen rate the videos using assessmenttools as an outcome measure.2 Usingvideo in this manner requires the re-searcher to consider the ideal video an-gle(s) and the number of views requiredfor capturing the desired behaviors.Similarly, microphone placement andaudio interference are important, partic-ularly for studies focusing on communi-cation. Researchers should also considerwhether the vital signs monitor display isa necessary as an adjunct to the videoviews for raters.

Improperly or inadequately capturedvideo or audio can hinder the rater’sability to accurately score performance.This should be accounted for when cal-culating the sample size for studiesrequired video capture and review. Inmulticenter studies in which videocapture hardware and software variesfrom site to site, there is a greater needto standardize the methods of videocapture and account for dropout re-lated to technical issues when calcu-lating sample size. On the basis of ourcollective experience in conductingSBR with video review, we have occa-sionally lost up to 10% of video becauseof issues with poor camera angle, soundquality, or problems with technology. Assuch, we recommend including video

capture and review as part of the pilottesting process in which pilot videosare reviewed for quality (ie, video, au-dio, and camera angle). Also considerincreasing your sample size a priori toaccount for lost video; however it willbe important to assess whether thereis any systematic bias to the lost videos.

OUTCOMES FOR PEDIATRIC SBR

The selection of outcome measures forSBR primarily depends on the researchquestion. One should choose outcomemeasures that are relevant, measure-able, andhold aplausible association tothe intervention. Outcomes for bothtypes of SBR may be framed based onKirkpatrick’s hierarchy of evidence,with learner’s attendance at the base ofthe pyramid (eg, satisfaction); knowl-edge, skills, and attitudes of participantsin the middle; and behavior change andclinical outcomes in respectively higherpositions.43 Satisfaction data are easierto capture but less impactful than evi-dence of actual process of care or patientimprovements based on the intervention.In quantitative SBR, methods to measureoutcomesmost commonly fall within 1 of3 categories: (1) the simulator itself asa measurement tool, (2) observationalchecklists, and (3) clinical and/or trans-lational outcomes. We focus our discus-sion on these 3 categories as they pertainto pediatric simulation research.

Simulator as the Measurement Tool

Most pediatric simulators are able tomeasure and record specific datapoints related to the passive physio-logicstateof thesimulatoraswellas theactionsperformedon it by participants.These provide objective measurements(eg, timing of head tilt, chin lift, or pulsecheck; depth and rate of chest com-pressions) that can be exported intoa research database for analysis. Sev-eral studies have leveraged the simu-lator’s ability to precisely capture timeto study an intervention’s impact on

time to performance of a skill or pro-cedure.44–46 As technology evolves, sowill the ability to collect and storevarious types of data in usable formatsfor research.

One potential pitfall to using simulationtechnology tomeasureoutcomes is thatthe accuracy of certain measurementsis largely unknown. For example, somesimulators can provide detailed logs ofhow deeply chest compressions areperformed.However, informationaboutprecision or validity of this measure-ment isunknown. Forexample, if a studyismeasuring depth of compressions asthe main outcome measure, how doesthe researcher know if the complianceand depth of the simulator chest wallmatches that of a live infant or pediatricpatient? More research is needed tovalidate proxy measurements from sim-ulators in the clinical world. Industrypartnershipscanhelptoaddresssomeofthese limitations. In the meantime, it isimportant for the commercial simulationand research community to collectivelyexplore and document the validity andreliability of these features.

Observational Checklists

Observational checklists are often usedto assess technical skills, behavioralperformance, and/or clinical perfor-mance in SBR studies.2,3,6 Discussion onvalidation and psychometrics are outsidethe scope of this review, but researchersshould ensure that the assessment toolsused are reliable and valid for the studypopulation and specific context of in-terest. Simply using a published checklistmay not be sufficient, and pilot studies toassess the checklist can improve therigor of the study. One of the advantagesof simulation is the ability to control forother variables and measure a person’sperformance on a standard model andsetting. The choice of checklist will de-pend on the specific study objectives,along with the relative strengths andweaknesses of each checklist. Several

1098 CHENG et al

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

observational checklists for pediatriccare have been developed and vali-dated in a simulated environment. Ta-ble 5 summarizes several clinical andbehavioral assessment tools that havebeen validated for pediatric resuscitationand provides examples of pediatric pro-cedural skills checklists.

If observational checklists are used asan outcome measure, the researchercan apply the tool in real-time and/orretrospectively by video review. Real-time review allows for rapid acquisitionof data. However, reliability of data col-lected inreal-time ishighlydependentonrater familiarity with the tool and theability of the rater to accurately assessperformance in real-time while con-currently recording scores. Conversely,video recording allows reviewers topause, rewind, or repeatedly reviewperformance tomore thoroughlyextractobjectivedetails. Useof videoalsoallowsthe researcher to more easily blind therater to study purpose or group allo-cation. Our research network has lev-eragedtechnologytosharevideosonlineand therefore make available to a large

groupof raters.47 Regardless ofwhetherreal-time and/or recorded review isused in a study, the implementation of arater training process before thestudy will help to improve interraterreliability.2

Clinical/Translational Outcomes

The ultimate measure of any medicalintervention ishowitaffectspatientcareand clinical outcomes. This is particu-larly important because it is unclear thedegree to which selected human per-formance measures in a simulated en-vironment (eg, observational checklists)correlatewith truepatientand/orhealthcare outcomes. Because of the size andcostof conducting suchstudieswithrealpatient outcomes, there are far fewerexamples of SBR measuring clinicaloutcomes. In Cook’smeta-analysis of 609technology-enhanced simulation arti-cles, only 32 studies reported patient/health outcomes.1 In a recent study,7

Andreatta demonstrated improved sur-vival rates frompediatric cardiac arrestafter implementing a longitudinal sim-ulationmock code program. Studies like

this are especially challenging becausethere are typically numerous confound-ing variables that have an impact onclinical outcomes, and learner groupshave other sources of learning outside ofthe study intervention. In an attempt toaddress these challenges, several groupshave begun to form longitudinal data-bases to measure the impact of educa-tional interventions over time (eg, theAmerican Heart Association’s Get With theGuidelines—Resuscitation registry). Amulticenter pediatric network, the In-ternational Network for Simulation-basedInnovation, Research and Education(INSPIRE, http://www.inspiresim.com)hasbeen formed to help achieve the samplesize and power needed to measure moreinfrequent clinical outcomes. These ini-tiatives have the potential to facilitate theincorporation of clinical outcomes intofuture pediatric SBR studies.

Summary

Theeffectiveuseofsimulationresearch inpediatrics isdependentonunderstandingthe benefits and challenges of SBR, thesimulation-specific threats to the internal

TABLE 5 Examples of Assessment Tools for Pediatric Resuscitation and Procedural Skills

Focus of Tool Assessment Tool(No. of Items on Tool)

Study Subjects Principles Addressed in Tool Reported IRR

Resuscitation Simulation Team AssessmentTool (94 items)49

Teams of pediatric residentsat a tertiary care pediatrichospital

Clinical and behavioral performanceduring pediatric resuscitation(team performance)

ICC = 0.81

KidSIM Pediatric ResuscitationAssessment Tool (26 items)50

Pediatric residents at a tertiarycare pediatric hospital

Clinical and behavioral performanceduring pediatric resuscitation(team leader performance)

Pearson’scorrelationcoefficient =0.453–0.617(medium to highcorrelation)

Clinical Performance Tool(21 items)2,51

Pediatric resuscitation teamsfrom 14 tertiary care pediatrichospitals

Clinical performance duringpediatric resuscitation (teamperformance)

IRR = 0.63–0.81

Tool for Resuscitation AssessmentUsing Computerized Simulation(72 items)52

Pediatric residents, fellows andrecent graduates at 2 tertiarycare pediatric hospitals

Clinical performance and leadershipbehavior (team leaderperformance)

ICC = 0.80

Proceduralskills

Intraosseous Procedure Scale53 Emergency physicians Intraosseous needle insertion in6 mo old infant

ICC = 0.947

Infant Lumbar Puncture GlobalSkills Assessment54–56

Pediatric residents, emergencymedicine residents

Lumbar puncture ICC = 0.71

Infant Lumbar PunctureSubcomponent SkillsChecklist54–56

Pediatric residents, emergencymedicine residents

Lumbar puncture ICC = 0.52

ICC, intraclass correlation coefficient; IRR, interrater reliability.

STATE-OF-THE-ART REVIEW ARTICLE

PEDIATRICS Volume 133, Number 6, June 2014 1099

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

validity of simulation studies, and theimplementation of strategies in simula-tion research studies to minimize thesethreatsastheyexist inapediatriccontext.Selecting valid outcome measures thatare relevant, consistently measureable,and hold a plausible association to theintervention being studied is an impor-tant component in designing simulationresearch studies. Careful considerationof these elements, along with the es-tablishment of a common research

agenda for the pediatric simulationcommunity, will help to ensure thathigh-quality SBR that tackles themostpertinent questions in pediatrics willcontribute to improving the quality ofcareandclinical outcomes forpediatricpatients.

ACKNOWLEDGMENTSThe authors would like to acknowledgethecontributionsofmembersof the Inter-national Network for Simulation-based

Pediatric Innovation, Research and Edu-cation (INSPIRE) who have participatedin SBR studies, and whose efforts overthe years have helped to shaped the con-tentof thismanuscript. Theauthorswouldalso like to acknowledge the Laerdal Foun-dation for Acute Medicine and the RbabyFoundation for supporting INSPIRE re-search, and the Society for Simulation inHealthcare and the International PediatricSimulation Society for supporting the bi-annual INSPIRE research meetings.

REFERENCES

1. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professionseducation: a systematic review and meta-analysis. JAMA. 2011;306(9):978–988

2. Cheng A, Hunt EA, Donoghue A, et al. Ex-amining pediatric resuscitation educationusing simulation and scripting debriefing:a multicenter, randomized-controlled trial.JAMA Pediatr. 2013;167(6):528–536

3. Donoghue AJ, Durbin DR, Nadel FM, StryjewskiGR, Kost SI, Nadkarni VM. Effect of high-fidelitysimulation on Pediatric Advanced Life Sup-port training in pediatric house staff: a ran-domized trial. Pediatr Emerg Care. 2009;25(3):139–144

4. Mikrogianakis A, Kam A, Silver S, et al.Telesimulation: an innovative and effectivetool for teaching novel intraosseous in-sertion techniques in developing countries.Acad Emerg Med. 2011;18(4):420–427

5. Loveless MB, Finkenzeller D, Ibrahim S,Satin AJ. A simulation program for teach-ing obstetrics and gynecology residentsthe pediatric gynecology examination andprocedures. J Pediatr Adolesc Gynecol.2011;24(3):127–136

6. Auerbach M, Kessler D, Foltin JC. Repetitivepediatric simulation resuscitation training.Pediatr Emerg Care. 2011;27(1):29–31

7. Andreatta P, Saxton E, Thompson M, Annich G.Simulation-based mock codes significantlycorrelate with improved pediatric patientcardiopulmonary arrest survival rates.Pediatr Crit Care Med. 2011;12(1):33–38

8. Hunt EA, Heine M, Hohenhaus SM, Luo X,Frush KS. Simulated pediatric trauma teammanagement: assessment of an educa-tional intervention. Pediatr Emerg Care.2007;23(11):796–804

9. Falcone RA Jr, Daugherty M, Schweer L,Patterson M, Brown RL, Garcia VF. Multidis-

ciplinary pediatric trauma team training us-ing high-fidelity trauma simulation. J PediatrSurg. 2008;43(6):1065–1071

10. Patterson MD, Geis GL, Falcone RA, LeMasterT, Wears RL. In situ simulation: detection ofsafety threats and teamwork training ina high risk emergency department. BMJQual Saf. 2013;22(6):468–477

11. Geis GL, Pio B, Pendergrass TL, Moyer MR,Patterson MD. Simulation to assess thesafety of new healthcare teams and newfacilities. Simul Healthc. 2011;6(3):125–133

12. Fonte M, Oulego-Erroz I, Nadkarni L, Sánchez-Santos L, Iglesias-Vásquez A, Rodríguez-Núñez A. A randomized comparison of theGlideScope videolaryngoscope to the stan-dard laryngoscopy for intubation by pediat-ric residents in simulated easy and difficultinfant airway scenarios. Pediatr EmergCare. 2011;27(5):398–402

13. Hung GR, Whitehouse SR, O’Neill C, Gray AP,Kissoon N. Computer modeling of patientflow in a pediatric emergency departmentusing discrete event simulation. PediatrEmerg Care. 2007;23(1):5–10

14. Hunt EA, Vera K, Diener-West M, et al. Delaysand errors in cardiopulmonary resusci-tation and defibrillation by pediatric resi-dents during simulated cardiopulmonaryarrests. Resuscitation. 2009;80(7):819–825

15. Weinger MB. The pharmacology of simulation:a conceptual framework to inform progressin simulation research. Simul Healthc. 2010;5(1):8–15

16. McGaghie WC, Issenberg SB, Petrusa ER,Scalese RJ. A critical review of simulation-based medical education research: 2003–2009. Med Educ. 2010;44(1):50–63

17. Cook DA. One drop at a time: research toadvance the science of simulation. SimulHealthc. 2010;5(1):1–4

18. McGaghie WC, Issenberg SB, Cohen ER,Barsuk JH, Wayne DB. Translational educa-tional research: a necessity for effectivehealth-care improvement. Chest. 2012;142(5):1097–1103

19. Issenberg SB, McGaghie WC, Petrusa ER, LeeGordon D, Scalese RJ. Features and uses ofhigh-fidelity medical simulations that lead toeffective learning: a BEME systematic review.Med Teach. 2005;27(1):10–28

20. Thomas EJ, Taggart B, Crandell S, et al.Teaching teamwork during the NeonatalResuscitation Program: a randomized trial.J Perinatol. 2007;27(7):409–414

21. Duran R, Alada�g N, Vatansever U, Küçüku-�gurluo�glu Y, Süt N, Acunas B. Proficiency andknowledge gained and retained by pediatricresidents after neonatal resuscitation course.Pediatr Int. 2008;50(5):644–647

22. Campbell DM, Barozzino T, Farrugia M, SgroM. High-fidelity simulation in neonatal re-suscitation. Paediatr Child Health (Oxford).2009;14(1):19–23

23. Gilfoyle E, Gottesman R, Razack S. De-velopment of a leadership skills workshopin paediatric advanced resuscitation. MedTeach. 2007;29(9):e276–e283

24. Jankouskas T, Bush MC, Murray B, et al.Crisis resource management: evaluatingoutcomes of a multidisciplinary team.Simul Healthc. 2007;2(2):96–101

25. Schilleman K, Witlox RS, Lopriore E, MorleyCJ, Walther FJ, te Pas AB. Leak and ob-struction with mask ventilation during sim-ulated neonatal resuscitation. Arch Dis ChildFetal Neonatal Ed. 2010;95(6):F398–F402

26. Shavit I, Keidan I, Hoffmann Y et al. Enhanc-ing patient safety during pediatric sedation:the impact of simulation-based training ofnonanesthesiologists. Arch Pediatr AdolescMed. 2007;161:740–743

1100 CHENG et al

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

27. López YM, Pilar FJ, Medina JA, et al. Cour-ses on mechanical ventilation in pediatrics:first experience in Spain. Pediatr Pulmonol.2007;42(11):1072–1077

28. Ballard HO, Shook LA, Iocono J, Turner MD,Marino S, Bernard PA. Novel animal modelfor teaching chest tube placement. J KyMed Assoc. 2009;107(6):219–221

29. Gaies MG, Morris SA, Hafler JP, et al. Re-forming procedural skills training for pedi-atric residents: a randomized, interventionaltrial. Pediatrics. 2009;124(2):610–619

30. Sudikoff SN, Overly FL, Shapiro MJ. High-fidelitymedical simulation as a technique to improvepediatric residents’ emergency airway man-agement and teamwork: a pilot study. PediatrEmerg Care. 2009;25(10):651–656

31. Nakajima K, Wasa M, Takiguchi S, et al. Amodular laparoscopic training program forpediatric surgeons. JSLS. 2003;7(1):33–37

32. Thomson M, Heuschkel R, Donaldson N,Murch S, Hinds R. Acquisition of compe-tence in paediatric ileocolonoscopy withvirtual endoscopy training. J Pediatr Gas-troenterol Nutr. 2006;43(5):699–701

33. Hamilton JM, Kahol K, Vankipuram M, AshbyA, Notrica DM, Ferrara JJ. Toward effectivepediatric minimally invasive surgical sim-ulation. J Pediatr Surg. 2011;46(1):138–144

34. Cook DA, Hamstra SJ, Brydges R, et al.Comparative effectiveness of instructionaldesign features in simulation-based educa-tion: systematic review and meta-analysis.Med Teach. 2013;35(1):e867–e898

35. Vincent C, Taylor-Adams S, Stanhope N.Framework for analysing risk and safetyin clinical medicine. BMJ. 1998;316(7138):1154–1157

36. LeBlanc VR, Manser T, Weinger MB, MussonD, Kutzin J, Howard SK. The study of factorsaffecting human and systems performancein healthcare using simulation. SimulHealthc. 2011;6(suppl):S24–S29

37. Cook DA, Beckman TJ. Reflections on ex-perimental research in medical education.Adv Health Sci Educ Theory Pract. 2010;15(3):455–464

38. Kassab ES, King D, Hull LM, et al. Actortraining for surgical team simulations. MedTeach. 2010;32(3):256–258

39. Rudolph JW, Simon R, Raemer DB. Whichreality matters? Questions on the path tohigh engagement in healthcare simulation.Simul Healthc. 2007;2(3):161–163

40. Dieckmann P, Gaba D, Rall M. Deepening thetheoretical foundations of patient simula-tion as social practice. Simul Healthc. 2007;2(3):183–193

41. Donoghue AJ, Durbin DR, Nadel FM,Stryjewski GR, Kost SI, Nadkarni VM. Per-ception of realism during mock resuscita-tions by pediatric housestaff: the impact ofsimulated physical features. Simul Healthc.2010;5(1):16–20

42. Raemer D, Anderson M, Cheng A, Fanning R,Nadkarni V, Savoldelli G. Research regardingdebriefing as part of the learning process.Simul Healthc. 2011;6(suppl):S52–S57

43. Kirkpatrick DI. Evaluating Training Programs:The Four Levels. 2nd ed. San Francisco, CA:Berrett-Koehler; 1998

44. Schunk D, Ritzka M, Graf B, Trabold B. Acomparison of three supraglottic airwaydevices used by healthcare professionalsduring paediatric resuscitation simulation.Emerg Med J. 2013;30(9):754–757

45. Rahman T, Chandran S, Kluger D, et al.Tracking manikin tracheal intubation usingmotion analysis. Pediatr Emerg Care. 2011;27(8):701–705

46. Martin P, Theobald P, Kemp A, Maguire S,Maconochie I, Jones M. Real-time feedbackcan improve infant manikin cardiopulmo-nary resuscitation by up to 79%—a rando-mised controlled trial. Resuscitation. 2013;84(8):1125–1130

47. Cheng A, Nadkarni V, Hunt EA, Qayumi K;EXPRESS Investigators. A multifunctionalonline research portal for facilitation ofsimulation-based research: a report fromthe EXPRESS research collaborative. SimulHealthc. 2011;6(4):239–243

48. Howard SK, Gaba DM, Smith BE, et al.Simulation study of rested versus sleep-

deprived anesthesiologists. Anesthesiology.2003;98(6):1345–1355, discussion 5A

49. Reid J, Stone K, Brown J, et al. The SimulationTeam Assessment Tool (STAT): development,reliability and validation. Resuscitation. 2012;83(7):879–886

50. Grant EC, Grant VJ, Bhanji F, Duff JP, Cheng A,Lockyer JM. The development and assess-ment of an evaluation tool for pediatricresident competence in leading simulatedpediatric resuscitations. Resuscitation. 2012;83(7):887–893

51. Donoghue A, Ventre K, Boulet J, et al; EXPRESSPediatric Simulation Research Investigators.Design, implementation, and psychometricanalysis of a scoring instrument for simu-lated pediatric resuscitation: a report fromthe EXPRESS pediatric investigators. SimulHealthc. 2011;6(2):71–77

52. Brett-Fleegler MB, Vinci RJ, Weiner DL,Harris SK, Shih MC, Kleinman ME. Asimulator-based tool that assessespediatric resident resuscitation compe-tency. Pediatrics. 2008;121(3). Available at:www.pediatrics.org/cgi/content/full/121/3/e597

53. Oriot D, Darrieux E, Boureau-Voultoury A,Ragot S, Scépi M. Validation of a perfor-mance assessment scale for simulatedintraosseous access. Simul Healthc. 2012;7(3):171–175

54. Gerard JM, Kessler DO, Braun C, Mehta R,Scalzo AJ, Auerbach M. Validation ofglobal rating scale and checklist instru-ments for the infant lumbar punctureprocedure. Simul Healthc. 2013;8(3):148–154

55. Auerbach MA, Chang TP, Reid J, et al. Arepediatric interns prepared to perform infantlumbar punctures? A multi-institutional de-scriptive study. Pediatr Emerg Care. 2013;29(4):453–457

56. Kessler DO, Auerbach MA, Pusic M, Tunik MG,Foltin JC. A randomized trial of simulation-based deliberate practice for infant lumbarpuncture skills. Simul Healthc. 2011;6(4):197–203

STATE-OF-THE-ART REVIEW ARTICLE

PEDIATRICS Volume 133, Number 6, June 2014 1101

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

DOI: 10.1542/peds.2013-3267 originally published online May 12, 2014; 2014;133;1091Pediatrics 

Vinay Nadkarni and David KesslerAdam Cheng, Marc Auerbach, Elizabeth A. Hunt, Todd P. Chang, Martin Pusic,

Designing and Conducting Simulation-Based Research

ServicesUpdated Information &

http://pediatrics.aappublications.org/content/133/6/1091including high resolution figures, can be found at:

References

1http://pediatrics.aappublications.org/content/133/6/1091.full#ref-list-This article cites 54 articles, 6 of which you can access for free at:

Subspecialty Collections

ology_subhttp://classic.pediatrics.aappublications.org/cgi/collection/gastroenterGastroenterologyfollowing collection(s): This article, along with others on similar topics, appears in the

Permissions & Licensing

https://shop.aap.org/licensing-permissions/in its entirety can be found online at: Information about reproducing this article in parts (figures, tables) or

Reprintshttp://classic.pediatrics.aappublications.org/content/reprintsInformation about ordering reprints can be found online:

ISSN: . 60007. Copyright © 2014 by the American Academy of Pediatrics. All rights reserved. Print American Academy of Pediatrics, 141 Northwest Point Boulevard, Elk Grove Village, Illinois,has been published continuously since . Pediatrics is owned, published, and trademarked by the Pediatrics is the official journal of the American Academy of Pediatrics. A monthly publication, it

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from

DOI: 10.1542/peds.2013-3267 originally published online May 12, 2014; 2014;133;1091Pediatrics 

Vinay Nadkarni and David KesslerAdam Cheng, Marc Auerbach, Elizabeth A. Hunt, Todd P. Chang, Martin Pusic,

Designing and Conducting Simulation-Based Research

http://pediatrics.aappublications.org/content/133/6/1091located on the World Wide Web at:

The online version of this article, along with updated information and services, is

ISSN: . 60007. Copyright © 2014 by the American Academy of Pediatrics. All rights reserved. Print American Academy of Pediatrics, 141 Northwest Point Boulevard, Elk Grove Village, Illinois,has been published continuously since . Pediatrics is owned, published, and trademarked by the Pediatrics is the official journal of the American Academy of Pediatrics. A monthly publication, it

by guest on May 18, 2018http://pediatrics.aappublications.org/Downloaded from