An Augmented Reality Delivery Simulator forMedical Training

Tobias Sielhorst1, Tobias Obst2,Rainer Burgkart2, Robert Riener3, and Nassir Navab1

1 Lehrstuhl fur Informatikanwendungen in der Medizin, Institut fur Informatik, TUMunchen, Germany Tobias.Sielhorst@cs.tum.edu

2 Klinik fur Orthopadie und Sportorthopadie r.d. Isar, TU Munchen, Munich,Germany Tobias.Obst@lrz.tu-muenchen.de

3 Automatic Control Lab, ETH, and Spinal Cord Injury Center, University HospitalBalgrist, Zurich, Switzerland riener@control.ee.ethz.ch

Abstract. This paper presents the extension of a birth simulator formedical training with an augmented reality system. The system presentsan add-on of the user interface for our previous work on a mixed realitydelivery simulator system [1]. This simulation system comprised directhaptic and auditory feedback, and provided important physiological dataincluding values of blood pressure, heart rates, pain and oxygen supply,necessary for training physicians. Major drawback of the system wasthe indirect viewing of both the virtual models and the final deliveryprocess. The current paper extends the existing system by bringing inthe in-situ visualization. This plays an important role in increasing theefficiency of the training, since the physician now concentrates on thevaginal delivery rather than the remote computer screen. In addition,forceps are modeled and an external optical tracking system is integratedin order to provide visual feedback while training with the simulator forcomplicated procedures such as forceps delivery.

1 Introduction

1.1 Problem statement

There is a trend from vaginal delivery to cesarean sections in German-speakingcountries [2]. Reasons for this change are earlier decisions for cesarean sectionsand sections on demand. This can be explained with improved techniques aswell as new medication. Still sections have a higher risk of complications whichcannot be neglected. For an emergency section the risk of complications is 12times higher compared to a vaginal delivery [3], [4]. In a scheduled cesareansection the risk of complications is still 1.5 to 2 times higher.

Despite several technical improvements the perinatal mortality is nearly con-stant since 1980. A reason might be found in the traditional way of medicaleducation where practice follows theory seamlessly. The delivery simulator issupposed to fill that gap. Furthermore it gives the chance to experienced doctors

to refresh their knowledge. The long term goal of the proposed delivery simula-tor is offering a device for improved training in order to reduce the amount ofcesarean sections as well as the number of perinatal deaths.

1.2 Delivery Simulator

The ’Klinik fur Orthopadie und Sportorthopadie r.d. Isar’ has developed a birthsimulator consisting of a haptic device in a body phantom and software that sim-ulates biomechanical and physiological functions. The position of the 3D modelis visualized on a screen and audio output is generated. Still in development pro-cess, it already represents a delivery simulator that provides multimodal func-tionality [1, 5]. Therefore the birth simulator includes already 3D visualization.

1.3 Augmented Reality

Augmented Reality (AR) is a relative new means of visualization. The idea ofaugmented reality is inserting virtual objects into the normal field of view. Sincethe early 90’s AR researchers have tried to apply this technology to medicalapplications [6]. Different medical AR prototype systems were built and testedin the late 90’s. Blackewell et al. [7] proposed a semi transparent display foraugmentation of orthopedic surgery. Edwards et al.[8] proposed an integrated ARsolution for operating microscopes. Birkfellner et al. [9] introduced the varioscopeAR, an augmented head-mounted operating microscope, for oral implantology.A major strength of augmented reality compared to visualization on an externalmonitor is the fact that it is able to show virtual objects in place. Users neednot take a look away from the place of interest to the screen.

According to [10] Virtual Reality has a high potential for medical teaching.The advantages of virtual endoscopy for medical training has been recognized[11]. Beier et al [12] propose a virtual reality platform for medical education. Thestrength of augmented reality compared to Virtual Reality is the perception ofthe real scene. The system only overlays those pieces of information that arenecessary. This eliminates the need for the user to immerse in a totally virtualenvironment and supports the intuitive integration of the information into thesetup. This information can be given e.g. by visualization of objects that areactually hidden under the real surface giving the impression of a view inside.

The augmented reality system we use is a research system that is calledRAMP. It has been developed by Siemens Corporate Research (SCR) for realtime augmentation in medical procedures [13]. It is optimized for accurate aug-mentation regarding relative errors between the real and the virtual scene. Itsaccuracy, its high resolution, high update rate and its little lag is currently stateof the art.

1.4 Integration Objectives

The contribution of this paper is the novel combination of an augmented re-ality device with a physical simulator. By this means the advantages of a real

device and Virtual Reality simulation can be combined and the visualization isbrought into the direct view of the physicians in training. The physical presenceof the haptic device can be used for feedback, while 3D visualization can be alsoprovided on the same spot.

The objective in development of simulators is high realism in order to bringusers as close as possible to the situation to learn. The birth simulator is intendedto be used at two stages. At the first stage the students are learning what hap-pens inside the patient and what to do. At the second stage the students arepracticing the delivery at the phantom under realistic circumstances. Note thatno visualization is needed at that stage because there is no visualization in thereal delivery room. In this paper we attack the first stage of learning where vi-sualization is needed and the emphasis is on the process of delivery itself ratherthan the full stressful situation in the delivery room. Realism is an importantrequirement for simulators to establish a close link between the abstract knowl-edge and practical use. For didactic reasons simulators are not intended to betotally realistic all the time. In order to focus on specific things to learn, certainaspects of a real scenario are hidden while the interesting ones are preserved.These should be as realistic as possible. For a better understanding of the pro-cess we would like to show the process of the delivery inside the phantom. Thebirth is a complex process including the movement of the baby.

The aim of the integration of both systems is to provide 3D visualizationat the place of interest. The birth simulator provides a virtual view inside themother on an ordinary screen. 3D visualization of the birth process on a displaydevice that supports 3D perception by stereo vision and kinetic depths cueshelps understanding the spatial structure of the process. Stereo vision and visualfeedback from head movements cannot be displayed on an ordinary display.

By this means the simulation is intended to be more intuitive and the scopeof realism is better adjusted to the learning process. The user interface is muchmore intuitive because the point of view can be chosen the most intuitive waythat is possible - by moving the head to the place of interest as opposed tonavigating by window buttons and a mouse. This intuitive interface lets theuser focus on learning rather than learning a user interface. The visualizationthat used to be on a screen next to the phantom is displayed right on top ofthe phantom into the field of view. The users can focus their view on the bodyphantom. Of course, the view inside the mother is not realistic, but this aspectof reality is not desired. At this point there is an emphasis on a view inside whilepreserving a realistic haptic impression. The haptic impression is provided bythe physical simulator and the view inside is possible because of the AR system.

Augmented reality not only adds objects into the scene. It is also able to givethe impression that real objects are removed or made transparent. This can beused for a teaching effect by taking a look inside the body. The view inside couldbe also achieved by removing the skin from the phantom physically, but thisremoves also the skin as an obstacle for real tools. Real objects like a forcepscould be visualized under the surface of the skin of the phantom in order tocheck the position of its ends, if they are tracked and modeled. This is possible

while the skin is still in the way and restricts the position of the forceps. Thisprovides a fine haptic impression. Without making the task easier the user mayhave a look inside to gain insight.

2 Description of the Three System Parts

2.1 Delivery simulator

The delivery simulator consists of a hardware and a software model. The hard-ware model is a female body phantom with the baby’s head inside. The baby isreduced to its head, which is acceptable for most cases. These use cases wouldinclude uncomplicated and many pathological deliveries, because the head is usu-ally the most important birth obstacle. The head of the baby has a force\torquesensor for haptic interaction with the user.

The software model is divided into two parts. The physiological part providesin real time (4ms) values of blood pressure, heart rates, pain and oxygen supply,which is given to the user. The calculated values generated using the contractionon oxytocine, fatigue of mother and fetus, oxygen supply to the child, and in-dividual boundary conditions. These boundary conditions can be e.g. mother’soxytocine production, stress and pain sensitivity, heart volume, as well as manyfetal parameters. The biomechanical values are calculated into the physiologicalmodel, too, where the forces may cause e.g. pain or blood loss.

The biomechanical model provides values in real time (4ms) for the positionof the baby’s head based on contraction forces, friction in the birth channel,tissue forces, and the user applied forces. The contraction forces are updated bythe physiological model.

These three components are realized as three processes communicating viaTCP/IP (see figure 1). As the fourth component there is the visualization com-ponent that gives a 3D visualization of the head position given by the positionof the baby’s head.

2.2 Augmented Reality System

The augmented reality system displays its images in a high resolution headmounted display (HMD). Since the video see-through HMD has opaque displaysthe images of the real scene are taken by two color cameras. RAMP makes useof visual tracking in order to provide the HMD’s position for accurate renderingof virtual objects. RAMP’s tracking device uses retro reflective markers, an infrared flash and a camera in order to provide highest accuracy and robustness bycontrolling lighting conditions without interfering with the user. The system hasa performance of around 30 frames per second for each eye in order to give anatural impression. The system lag is about 100ms which is true for both, realand virtual, objects since the camera images are synchronized with the trackingdata. Therefore the virtual objects appear always on the expected place evenif the HMD moves fast which adds realism on the virtual objects. RAMP is

Force/Torque SensorRT Linux

Biomechanical modelLinux

Physiological modelLinux

Delivery Simulator

Virtual baby‘shead position

Contractionforces

Force on baby‘s head

Internal forcesSingle CameraHead TrackingWindows 2000

VisualizationWindows 2000

User‘s headposition

RAMP

Multiple CameraInstrument TrackingWindows 2000

Instruments’position

Fig. 1. Data flow diagram: Rectangles depict processes, arrows depict TCP/IP com-munication

optimized to support short range tasks in arm range. The concept of video see-through HMDs for augmentation is well adapted to a training task because itdoes not need any user calibration and a duplication of the user’s view can bedisplayed easily on another display (see figure 5) or even stored for a later review.

2.3 External Tracking System

There are different options for tracking the forceps. The most obvious idea wouldbe to take the same tracking system as the AR-system. We decided not to doso for two reasons. First, there is the line of sight problem. The targets for thesingle camera tracking of the AR system are likely to be occluded since it needsabout eight markers [14] for sufficiently accurate, reliable and robust tracking ofeach of the two forceps parts.

The external tracking system is a stereo camera infra red system by thecompany A.R.T. 4 It can use the same kind of markers as the tracking system ofRAMP but it needs smaller tracking targets since it is not a single, but multiplecamera system. Its update rate is 60Hz. The inside-out tracking system is stillnecessary in this setup because it allows for a high accuracy in the rotationparameter which is significant for the AR system.

4 A.R.T GmbH Herrsching, Germany

THMD2Frame

TExt2Frame

TExt2Forceps

TFrame2Forceps

Fig. 2. The inside-out tracking system tracks the target for head pose recovery relativeto the phantom and the outside-in tracking system tracks the instruments relative tophantom. The marker set in the figure is attached to the phantom.

3 Integration methods

The phantom of the birth simulator has been attached reproducibly to the setof markers that is tracked by the augmented reality system. The registration ofthe virtual objects and the real phantom has been realized manually. The task ofthe whole delivery simulation is distributed on different computers with differentoperating systems. The data flow is established by TCP/IP communication (seefigure 1).

The registration between the external tracking and the AR-system has beenestablished by using the same tracking target for the external tracking for gen-eration of a reference coordinate system as the AR-system does (see figure 2).Since both use the same model data and generate the same coordinate systemsout of the model data, the calculation from the coordinate system of the exter-nal tracker to the reference coordinate system of RAMP can be done with thissimple formula

TFrame2Forceps = TExt2Forceps · T−1Ext2Frame

where T denotes the transformation between two coordinate systems inhomo-geneous including translation and rotation. By this means external cameras the

simulator and, of course, the HMD can be moved during the simulation withoutany need for recalibration.

For a realistic look and feel of the forceps we visualize a model that hasbeen modeled with the professional 3D modeling software MAYA by Alias. Itgenerates a VRML model that can be inserted into the scene of the augmentedspace. The registration of the model to the real object is done manually.

4 Results

The visualization of the virtual head position can be shown in the HMD inreal time (about 30fps) and in place. The update rate of the head positionfor visualization is slowed down to approximately 100ms due to heavy networktraffic in our setup that makes use of a remote X-Server in order to access theuser controls of the delivery simulator.

Different visualizations are available to obtain a view inside, with and withoutthe hip bone for a better orientation or with and without the virtual skin to seewhether the virtual head is already outside or not.

We experienced in this project that is necessary to combine different oper-ating systems (figure 1) because each subsystem has its own specific needs thatare implemented best on a certain operating system. If the data for commu-nication can be kept low as in this project it does not slow down the overallperformance of the system and it can be implemented easily. The update rate ofthe augmentation remains at 30 fps and the lag at 100 ms.

By using augmented reality we can provide virtual views like a look into thebody without missing the presence of the phantom. The phantom gives muchbetter haptic feedback than today’s haptic devices could provide. The qualitativeresults are promising. At this stage we are ready for further investigation in orderto collect quantitative results from students and doctors. We are also going toinvestigate the overall error of the combined tracking system.

Fig. 3. Screen shot with two views as seen on the HMD: Hip bone and head overlaidon phantom.

Fig. 4. Screen shot with two views as seen on the HMD: Skin of the phantom hideship bone and back of the head

5 Discussion

The current integration shows the feasibility of real time augmentation for med-ical training simulators. As a next step the augmented delivery simulator has tobe evaluated as to whether the augmentation can improve the learning effect.From the technical side it would be interesting to see how accurate the systemdisplays the augmented objects. Also the combination of the two evaluationsmight improve the understanding of AR in teaching: What role does accuracyplay in successful teaching?

There are still other open questions concerning user interaction. Augmentedreality has different needs for user interfaces than usual displays. For example,the biological functions used to be displayed in a window on the same screen asthe visualization. The augmented reality device displays the visualization in thefull field of view. Introducing an ordinary window into the field of view harmsthe 3D impression of the view and reduces the feeling of realism.

Another area for improvement is in adapting input devices to augmentedreality. The analgesics used to be applied by a mouse click on the window. Amouse returns 2D coordinates while the augmented reality device provides a 3Ddisplay. Apart from the fact that normal windows should not be used in the fieldof view, a mouse is not an adequate pointing device for augmented reality.

With this work we hope to start into the largely unexplored field of ARteaching.

6 Acknowledgement

We would like to thank Frank Sauer, Ali Khamene, and Sebastian Vogt fromSiemens Corporate Research in Princeton, USA for the courtesy of providing uswith the AR-system RAMP. We would like to thank A.R.T. GmbH, Herrsching,Germany for the courtesy of their multiple camera tracking system.

Fig. 5. Photo of the setup: The screen shows a duplication of the user’s view (left eye)

References

1. Obst, T., Burgkart, R., Ruckhaberle, E., Riener, R.: The delivery simulator: Anew application of medical vr. In: Proceedings of Medicine Meets Virtual Reality.(2004)

2. Haller, U., Hepp, H., Winter, R.: Caesarean section on demand or elective. caesarensection:request for rethinking. Gynakologisch Geburtshilfliche Rundschau (2002)

3. Schneider, K.: Caesarean section on demand - must risk of morbidity be considered?Gynakologisch Geburtshilfliche Rundschau (2002)

4. Hickl, E.: Change in indications for caesarean section. Gynakologisch Geburtshil-fliche Rundschau (2002)

5. Obst, T., Burgkart, R., Ruckhaberle, E., Riener, R.: Geburtensimulator mit mul-timodaler interaktion - delivery simulator with multimodal interaction. Automa-tisierungstechnik 54 (2004) 280–287

6. Bajura, M., Fuchs, H., Ohbuchi, R.: Merging virtual objects with the real world:seeing ultrasound imagery within the patient. In: Proceedings of the 19th annualconference on Computer graphics and interactive techniques, ACM Press (1992)203–210

7. Blackwell, M., Nikou, C., Gioia, A.M.D., Kanade, T.: An image overlay system formedical data visualization. In: Proceedings of the 1998 Medical Imaging Computingand Computer Assisted Intervention Conference (MICCAI ’98). (1998)

8. King, A.P., Edwards, P.J., C. R. Maurer, J., de Cunha, D.A., Gaston, R.P., Clark-son, M., Hill, D.L.G., Hawkes, D.J., Fenlon, M.R., Strong, A.J., Cox, T.C.S., M.,Gleeson, J.: Stereo augmented reality in the surgical microscope. Presence: Tele-operators and Virtual Environments 9 (2000) 360–368

9. Birkfellner, W., Figl, M., Huber, K., Watzinger, F., Wanschitz, F., Hummel, J.,Hanel, R., Greimel, W., Homolka, P., Ewers, R., Bergmann, H.: A head-mountedoperating binocular for augmented reality visualization in medicineUdesign andinitial evaluation. IEEE Transaction on Medical Imaging 21 (2002) 991–997

10. Riva, G.: Medical applications of virtual environments. Yearbook of MedicalInformatics (2003)

11. Geiger, B., Kikinis, R.: Simulation of endoscopy. Simulation of Endoscopy. Pro-ceedings First International Conference on Visition, Virtual Reality, and Roboticsin Medicine (1995)

12. Beier, K.P., Freer, J.A., Levine, H., Pletcher, T.A., Russel, W., Treloar, D.J., Lu-bitz, D.K., Wilkerson, W., Wolf, E.: An immersive virtual reality platform formedical education: Introduction to the medical readiness trainer. In: Proceedingsof the 33rd Hawaii International Conference on System Sciences. (2000)

13. Sauer, F., Khamene, A., Vogt, S.: An augmented reality navigation system witha single-camera tracker: System design and needle biopsy phantom trial. In: Pro-ceedings of the Second International Conference on Medical Image Computing andComputer-Assisted Intervention (MICCAI). (2002)

14. Vogt, S., Khamene, A., Sauer, F., Niemann, H.: Single camera tracking of markerclusters: Multiparameter cluster optimization and experimental verification. In:IEEE and ACM International Symposium on Mixed and Augmented Reality.(2002) 127–136