SIBGRAPI’ 99 - INTERNATIONAL SYMPOSIUM ON COMPUTERGRAPHICS, IMAGE PROCESSING AND VISION

COMPUTER GRAPHICS&

MEDICINE

by

LUCIANA PORCHER NEDELISABEL HARB MANSSOUR

CARLA MARIA DAL SASSO FREITAS

Instituto de InformáticaUniversidade Federal do Rio Grande do Sul (UFRGS)

CampinasOctober, 1999

[BLANK]

1

Contents

CONTENTS.......................................................................................................................1

INTRODUCTION .............................................................................................................5

MODULE 1: IMAGE ACQUISITION AND PROCESSING......................................7

1. Introduction ................................................................................................................7

2. Image Acquisition Modali ties....................................................................................7

3. Image Processing Techniques .................................................................................10

4. Registration...............................................................................................................11

4.1. Registration Classification Cr iteria..................................................................12

4.2. Image Registration Techniques........................................................................15

5. Segmentation.............................................................................................................18

6. References.................................................................................................................20

MODULE 2: VISUALIZATION AND REPRESENTATION MODELS...............25

1. Introduction ..............................................................................................................25

2. Interactive Volume Visualization ...........................................................................25

2.1. Three-Dimensional Reconstruction.....................................................................28

2.2. Surface Visualization...........................................................................................28

2.3. Direct Volume Visualization...............................................................................30

2.3.1. Acceleration Techniques.............................................................................32

2.4. Hybrid Visualization............................................................................................34

3. Human Bodies Modeling Methods..........................................................................36

3.1. Body Modeling Techniques.................................................................................36

3.1.1. Stick Figure Models.....................................................................................36

3.1.2. Surface Models............................................................................................37

3.1.3. Volume Models...........................................................................................38

3.1.4. Multi -Layered Models.................................................................................39

4. Case Study: Vir tual Cadaver (The Visible Human Project) ................................42

5. References.................................................................................................................47

MODULE 3: SIMULATING VIRTUAL HUMAN BODIES.....................................55

1. Introduction ..............................................................................................................55

2

2. Modeling Anatomically-Based Human Bodies......................................................55

2.1. Skeleton Model ....................................................................................................55

2.1.1. Overview of the BODY Structure...............................................................55

2.1.2. General Functionality ..................................................................................56

2.1.3. Anatomic Template.....................................................................................62

2.1.4. Why Representing Bones?...........................................................................64

2.2. Muscles Simulation..............................................................................................66

2.2.1. Action Lines ................................................................................................66

2.2.2. Muscle Shape Design..................................................................................68

2.2.3. Deformation Model .....................................................................................69

2.2.4. Motion Simulation.......................................................................................73

2.2.5. Animation Examples...................................................................................74

2.3. Skin Surface Generation......................................................................................75

2.3.1. Body Division..............................................................................................75

2.3.2. Cross-Sectional Configuration ....................................................................76

2.3.3. Grouping Primitives....................................................................................76

2.3.4. Rapid Tesselation ........................................................................................77

2.4. Integration of the Parts.........................................................................................77

2.4.1. Volume Primitives.......................................................................................78

2.4.2. The Extremities............................................................................................78

2.4.3. System Organization....................................................................................79

2.4.4. How to Build an Anatomically-Based Body?.............................................81

2.4.5. Results .........................................................................................................84

3. Facial Modeling........................................................................................................87

3.1. Motion Simulation...............................................................................................88

3.2. Texture Mapping..................................................................................................89

4. Hands.........................................................................................................................89

4.1. Hand Topography................................................................................................91

4.2. Hand Simulation Model .......................................................................................92

5. Hair ............................................................................................................................94

5.1. Hair Animation....................................................................................................95

5.2. Colli sion Detection..............................................................................................96

5.3. Hair Rendering.....................................................................................................96

6. References.................................................................................................................98

MODULE 4: SURGERY SIMULATION...................................................................103

1. Introduction ............................................................................................................103

3

2. Overview .................................................................................................................103

3. Classification...........................................................................................................104

3.1. Navigation Systems..........................................................................................106

3.2. Modeling Techniques.......................................................................................107

3.3. Tissue Cutt ing ..................................................................................................110

3.4. Bleeding.............................................................................................................111

4. Technologies............................................................................................................113

5. Minimally Invasive Surgery Simulation (MIS) ...................................................116

6. Using Vir tual Humans and Surgery Simulation Systems in MedicalEducation ....................................................................................................................119

7. References...............................................................................................................124

MODULE 5: FINAL CONSIDERATIONS................................................................129

1. Introduction ............................................................................................................129

2. Vir tual Humans: Psychiatry and Behavior .........................................................129

3. Augmented Reali ty Applications..........................................................................130

4. Telemedicine: Collaborative Visualization ..........................................................134

5. Conclusions.............................................................................................................139

6. References...............................................................................................................142

4

5

Introduction

Since the cost of the graphic workstations became affordable, the use ofcomputer graphics techniques applied to different domains of applications isincreasing. One of these domains is medicine and the authors are specially concernedabout it because it is strictly related to li fe and nothing is more important than li fe.

This tutorial aims to introduce some computer graphic techniques applied tomedicine, by presenting the steps involved in the generation of a virtual human model.The text is divided in five main modules. Due to the importance of image acquisitionand processing techniques in dealing with the majority of medical data, the firstmodule presents briefly these subjects. We survey image acquisition modaliti es andimage processing techniques, emphasizing registration and segmentation of images.

Module 2 represents the second step in the modeling of virtual human bodiesfrom medical images. The segmentation methods discussed in the previous modulegive us the basic elements to reconstruct a human body. We discuss some interactivevolume visualization methods as well as the geometric and topological modeling ofthe basic elements that constitute the body (i.e., bones, muscles, skin, tendons, veins,fat, etc.). A revision of the first human body modeling methods is also presented.Concluding this module, a case study is presented, discribing the Visible HumanProject, in development by the Center for Human Simulation (CHS) at the Universityof Colorado Health Sciences Center.

Module 3 proposes a method to simulate human bodies based on anatomyconcepts, because we believe that the closer a model is to reality, the better will be theresults. Using this approach we describe a human representation method divided intothree layers: the rigid body conception from a real skeleton, the muscle design anddeformation and the skin generation. Some integration aspects are also presented anddiscussed. To conclude, we briefly describe a method to model and animate faces, ahands' deformation method and a model to represent hair.

Surgery simulation is nowadays one of the most popular applications ofcomputer graphics to medicine, involving several domains of computer graphics, i.e.,three-dimensional reconstruction, real-time rendering, animation, deformation,interaction, navigation an so on. Because of that, Module 4 presents a survey onsurgery simulation, including a classification of surgery simulation systems, thetechnology needed to build and use these systems, and two special sections focusingon minimally invasive surgeries and the use of surgery simulation systems foreducation and training.

The last module presents other applications for the use of Computer Graphicsand virtual human bodies in medicine, showing part of the large market that still couldbe explored. A brief description of the use of virtual humans in psychiatry is done,some augmented reality applications are presented, and the new field of collaborativevisualization is also described. Finally, some comments trying to enumerate theproblems still t o be solved and the challenges of this area are also drawn.

6

The authors are grateful to Marcelo Rodrigo Maciel Silva for many imagesand for his valuable work in implementing RenderVox (a software for interactivevolume visualization) and also to LIG - EPFL (Computer Graphics Lab) and Miralabteams for some examples and images about virtual human modeling.

7

MODULE 1:Image Acquisition and Processing

1. Introduction

In many medical specialties, diagnoses relay more and more on imageanalysis. With the evolution of image acquisition modaliti es, both in resolution andtissue distinctiveness, the capacity and fidelity of image diagnosis were extended[Rhodes 97]. Traditionally, image analysis has been the primary role of radiologists,who use their clinical knowledge, perceptual abiliti es, and information about specificpatients to interpret medical images. Radiological examination usually is requested forfour reasons: visualization, quantifi cation, localization, and screening.

Physicians often request an imaging procedure to confirm specific diseasehypothesis. The images are then used to visualize body structure or function, and thusdetermine whether disease features are present. Some diseases or therapy proceduresrequire quantifying measurable parameters, li ke the volume, shape or size of astructure. For certain purposes, such as radiation therapy or surgery, the localizationof a lesion or pathological change is the primary reason for a study. Finally, becausecareful interpretation of radiological images is time-consuming, systems that screenseveral images looking for abnormal features, thus providing assistance in medicalanalysis and interpretation.

This module presents an introduction of image acquisition and processingtechniques due to the importance of these subjects in dealing with the majority ofmedical data. Image acquisition modaliti es and image processing techniques aresurveyed and registration and segmentation of images are emphasized.

2. Image Acquisition Modali ties

The several existing medical image acquisition modaliti es can be divided intotwo broad classes based on the two basic ways of obtaining such images: invasive andnon-invasive methods. Non-invasive methods are, for example, X-ray, ultrasound,computed tomography and magnetic resonance. The invasive methods are thosewhere it is necessary to insert some instrument or liquid inside the human body toobtain the images. Some examples of this modality are angiography and nuclearmedicine [Greenes 90].

Also concerning imaging modaliti es, two global categories can be identified:anatomical and functional. Anatomical modaliti es depict primarily morphology, soit’s used, for example, to delineate the correct spatial extent of a tumor and thesurrounding normal structures. These modaliti es include X-ray, CT (ComputedTomography), MRI (Magnetic Resonance Imaging), US (Ultrasound), portal images(an X-ray image taken using a high energy) and (video) sequences obtained by

8

various catheter “scopes” , e.g., by laparascopy. Some prominent derivative techniquesare so detached from the original modaliti es that they appear under a separate name,e.g., MRA (Magnetic Resonance Angiography), DAS (Digital SubtractionAngiography, derived from X-ray), CTA (Computed Tomography Angiography), andDoppler (derived from US).

The second category, depicting primarily functional and metabolicinformation, include (planar) scintilography, SPECT (Single Photon EmissionComputed Tomography) and PET (Positron Emission Tomography), which togethermake up the nuclear medicine imaging modaliti es, and fMRI (functional MRI). Infact, these modaliti es show different, complementary and/or partially overlappingaspects of the anatomy examined or functional aspects, giving few anatomicalpointers [Bansdal 98, Maintz 98, Pluim 99].

Film-based radiography is the primary modality of medical imaging. TypicalX-ray images are produced by an X-ray beam - a form of ionizing radiation - form aX-ray source through the patient's body and onto a X-ray sensitive-film. The differentbody tissues absorb this energy in different amounts, so the rays produce differentshadows in the X-ray film. Depending on the equipment, to obtain the data is a realcrucial task because some are not in digital form. Digital radiography is the sameacquisition modality but the images are recorded digitally instead of in film form.

Computed-assisted tomography (CT) uses the same basic principle as X-raysbut produces images that represent slices of the human body (or whatever is beingimaged), i.e., they show internal parts of the body as we were observing a cut plane.Also, the quality of the data obtained from a CT is much higher than that produced bya conventional X-ray (Figure M1.1).

Figure M1.1: A subset of images obtained with computed-assisted tomography

First generation CT scanners used a pencil beam defined by a well -collimatedX-ray source and a single detector, which translate and rotate through 180 degrees inabout 300 seconds. Second generation scanners use a small li near array of detectors tospeed up data acquisition to about 20 seconds using fan-beam geometry. Third-generation scanners use a large, usually circular of detectors, and the source anddetector can rotate through 360 degrees in about 5 seconds. Fourth-generation systemsuse a fixed, circular ring of several detectors, and an X-ray source inside or outsidethe ring that can rotate 360 degrees in less than 5 seconds. The results of a completerotation are used to reconstruct a serial section, thus providing information describingthe interior of the body.

Magnetic Resonance Imaging (MRI) is mainly applied to soft tissues. Allatomic nuclei within the body have a net magnetic moment, which means that they actli ke tiny magnets. When the patient is placed inside a huge magnet that generates anintense magnetic field, all the nuclei li ne up in the direction of the field, spinningabout the axis of the field, with a frequency depending on the type of nucleus and onthe strength of the magnetic field. Measuring differences in the orientation of nucleiwhen radio pulses are applied and then removed produces the image. In this way, MR

9

images depict differences among tissues. An advantage of MRI in relation to CT isthat it allows revealing characteristics not shown in conventional CT. Like CT, MRIshows the internal parts of a body (Figure M1.2).

Figure M1.2: A subset of images obtained with magnetic resonance imaging

Ultrasound (or echographic) images are another common imaging modality.This imaging modality uses pulses of high-frequency sound waves, rather thanionizing radiation. An emitter (pointer) moved by an operator on the patient's bodycauses the image to be taken in real-time. As each sound wave encounters tissueinterfaces, a portion of the wave is reflected and a portion continues. The timerequired for the echo to return is proportional to the distance into the body at which itis reflected; the amplitude of a returning echo depend on the acoustical properties ofthe tissues and is represented in the image as brightness. The system constructs two-dimensional images by displaying the echoes from pulses of multiple adjacent one-dimensional paths (Figure M1.3).

Figure M1.3: Image obtained with ultrasound imaging modali ty

10

In Nuclear Medicine imaging, a radioactive isotope is chemically attached to abiologically active compound, then injected into the peripheral circulation. Thecompound is stored or processed in the proper organs (such as the thyroid or li ver).The isotopes emit radiation locally, which is measured by a special detector. The levelof radioactivity at each point is then stored digitally. This type of emissiontomography can be PET (Positron-Emission Tomography) or SPECT (Single-PhotonEmission Tomography). So, while CT shows absorption properties, representinganatomical structures, emission tomography methods depicts the spatial distributionof radiation and then are adequate for studying metabolic activities.The physical andbiological characteristics of the isotopes allow the use of small amounts of radiationto obtain scintilographic images, which can show physiological processes either in aspecific region or through the whole body (Figure M1.4).

Figure M1.4: Images obtained with nuclear medicine methods

3. Image Processing Techniques

Once the images have been acquired and before they are analyzed within thetasks mentioned in the Section 2, it may be necessary to perform some operations onthem in order to meet certain quality criteria. Well -known image processingtechniques [Jain 89, Russ 95] are used:� Image representation and compression deals with the characterization of

the quantity each pixel represents and how well the images can berepresented in compressed form for storing and transmission purposes;� Image restoration techniques are concerned to correct images by removingor minimizing degradations, in an image. This includes deblurring ofimages that present degradations due to sensor limitations or environmentinterference, noise filtering, and correction of geometric distortions;� Image enhancement, where the goal is to better represent some featureseither for visualization or analysis purposes. Some examples include

11

contrast enhancement and edge detection, pseudocoloring, noise filtering,sharpening, and magnifying;� Extraction of features or substructures represented in an image, orquantitative measurements to produce a description of the image.

Digital filters, operators, transformations, and algorithms accomplish thesetechniques. Digital filters can be applied to remove signal noise, enhance edges, andotherwise sharpen blurry images. Low-pass filters usually introduce blurring inimages and are used to attenuate features for visualization purposes. On the otherhand, high-pass filters are used to sharpen images and then enhance edges that mightrepresent structures that need to be detected. Edges can be better detected employingoperators based on derivatives like Sobel, Kirsch and gradient operators.

The enhancement and extraction of features from images often requirearithmetic operations to be executed on images. Subtracting images is particularlyuseful when dealing with images that are obtained with substances that produces highcontrast in some regions. Additionally, for comparison purposes, or for reconstructinga volume from a series of images, it may be necessary to align them. This involvesgeometric operations used in a process known as registration (see Section 4).

A very important process, fundamental for the reconstruction of organs andbody parts from volumes of images is segmentation. It is the first processing stage toextract regions of interest for another task (visualization, measurement, and analysis).There is no unique solution to this problem, but instead classes of tools (for example,deformable models, mathematical morphology, scale-space analysis, etc.), which canbe often combined together to reach a given goal.

Due to their importance, registration and segmentation are commented with alittl e more detail i n the following sections.

4. Registration

Registration is a fundamental task in image processing used to match two ormore images, or volumes, obtained, for example, at different times, from differentsensors and scanners, or from different viewpoints. Simply, it consists in a processthat maps pixels from one image, called the reference image, to pixels in anotherimage, called the test image. So, registration is often necessary for:� Integrating information taken from different sensors;� Finding changes in images taken at different times or under different

conditions;� Inferring three-dimensional information from images in which either thecamera or the objects in the scene has moved;� Model-based object recognition [Bansal 98, Brown 92].

Although many registration techniques have been developed over the years forvarious types of data and problems, in this section we will only consider the medicalimage registration techniques. As we know, medical imaging is an essentialcomponent of a large number of applications that occurs throughout the clinical trackof events, as clinical diagnosis and planning of surgical and radio-therapeuticprocedures. The several acquired images could be two or three-dimensional, leadingto the development of two or three-dimensional registration techniques. Moreover, the

12

precision of the several registration methods described in the literature is partiallyrelated to the precision of the acquiring equipment [Vemuri 97, Maintz 98].

Many medical systems which works with images evaluation, volumevisualization of an image set, CAS (Computer-Assisted Surgery), and augmentedreality, require the registration of images, or a closely related operation, as anintermediate step. An example where image registration is a significant componentincludes the alignment of images acquired from just one scanner. In this case, thealignment became necessary because during the data acquisition the patient couldmove and then the sequence of acquired images are not aligned (e.g. Figure M1.5).Another example involves utili zation of different scanners, when the alignment ofimages is done to allow a better characteristic analysis. This is very useful for clinicaldiagnosis, treatment planning and delivery of a therapy, when images from differentmodaliti es are often acquired as they provide complementary information about adisease and also can give graphical verification of a delivery therapy. Then, the simpletask of pili ng up the images is not suff icient for medical data visualization [Brown 92,Mahoney 96, Westermann 96, Bansal. 98].

Figure M1.5: Example of acquired images when the patient moves

During registration algorithm development, several common characteristicscan be enumerated. For example, if it is based on artificial objects introduced into theimage or not, if it’s based on rigid or aff ine transformations, or if it is monomodal(images from the same modality) or multimodal (images from different modaliti es).

Maintz and Viergever classified the medical image registration methods[Maintz 98, Maintz 98a], according to a model based on nine salient criteria, whichwill be briefly explained in the following section. Section 4.2 presents some of themost used image registration techniques.

4.1. Registration Classification Cr iter ia

The first criterion is dimensionali ty, i.e., the number of spatial and timedimensions of the image to be registered, that could be 2D/2D, 2D/3D or 3D/3D. Themain criterion here is whether all dimensions are spatial, or if time is an addeddimension. Time series involve more than two images that are usually acquired, forexample, to monitoring the bone growth in children or tumor growth methods [Maintz98, Maintz 98a]. 2D/3D-image registration is often used to help in guidinginterventions. In this case, the common “real time” modaliti es combined with CT andMRI are ultrasound or x-ray fluoroscopy [Penney et al. 98a]. When the work involves3D/3D dimensionality, it’s also possible to divide the registration methods into twogroups, volume based and surface based. So, any technique which performsregistration by making use of a relationship between voxel intensities within theimages is volume based, and any technique which works by minimizing a distancemeasure between two corresponding surfaces in the images to be matched is surfacebased [West 99].

13

For sure, the nature of registration basis is the most important classificationcriterion, yielding methods characterized as:� Extrinsic: based on artificial objects introduced into the image,� Intrinsic: based on patient generated image information only, and� Non-image based: using calibrated scanner coordinate systems.

Although some recent studies had shown that intrinsic methods are starting tohave a better performance, extrinsic methods are faster and depends of the markersobtained by the insertion of artificial objects attached to the patient (e.g. screwmarkers). Non-invasive markers, as fiducial objects (skin markers), can also be used,but as a rule are less accurate. Intrinsic methods are more attractive, since they don’ tintroduce any object in the image during acquisition and could consider the entireimage content. In this case, the registration can be based in one of three alternatives:� On a limited set of identified salient points (landmarks);� On the alignment of segmented binary structures (segmentation based),

most commonly object surfaces, where we can stand out the popularity ofHead-hat [Pelizarri 89] and Chamfer [Borgefors 88] techniques;� Directly onto measurements computed from the image gray values (voxelproperty based).

The last one, non-image based is possible if the imaging coordinate systems ofthe two scanners are somehow calibrated one to each other. This usually requires thescanners to be brought into the same physical location, and the assumption that thepatient remains motionless between both acquisitions [Maintz 98, Maintz 98a,Zuiderveld 96].

A fundamental characteristic of any image registration technique is the type ofspatial transformation or mapping used to properly overlay two images. According tothe nature of transformation, an image coordinate transformation could be rigid,aff ine, projective or curved. Rigid transformations account for object or sensormovement in which objects in the images retain their relative shape and size. In otherwords, this kind of transformation is composed of a combination of rotations andtranslations. Aff ine transformations are more general than rigid and can thereforetolerate more complicated distortions while still maintaining some nice mathematicalproperties. So parallel li nes are mapped onto parallel li nes. Projective, that maps linesonto lines, and curved transformations, that map lines onto curves, accounts fordistortions [Brown 92, Maintz 98].

Finally, considering the domain of transformation, a transformation is calledglobal i f it applies to the entire image, and local i f subsections of the image have theirown transformations defined. Figure M1.6 shows examples of all these transformationtypes [Maintz 98].

Concerning the degree of user interaction, three levels could be recognized:interactive, semi-automatic and automatic. In the first level the user does theregistration himself. On the semi-automatic level, the interaction required can be oftwo different natures: or the user needs to initialize the algorithm (e.g. making theinitial geometrical transformation of an image), or steer the algorithm (e.g. byrejecting or accepting suggested registration hypotheses). Automatic is when the useronly supplies the algorithm with the image data and possibly information on theimage acquisition.

14

Rigid

Original LocalGlobal

Aff ine

Projective

Curved

Figure M1.6: Examples of two-dimensional transformations [Maintz 98]

Optimization procedure, the sixth criterion, uses parameters to build up theregistration transformation that can either be computed directly, i.e., determined in anexplicit fashion from the available data, or searched for, i.e., determined byoptimizing some function defined in the parameter space.

There are many image acquisition modaliti es that could be encountered incurrent literature, such as CT, MRI, PET, SPECT and X-ray. The variety ofmodali ties involved in the registration procedure allows the recognition of fourclasses of registration tasks: monomodal, multimodal, modality to model and patientto modality. In monomodal applications, the images to be registered belong to thesame modality, as opposed to multimodal registration tasks, where the images to beregistered stem from two different modaliti es. In modality to model and patient tomodality registration only one image is involved and the other “modality” is either amodel (e.g. mathematical models of anatomy or physiology) or the patient himself.Examples of these classes of registration tasks are presented in Figure M1.7.

Monomodal

[Silva 97]

Multimodal

MRI PET Registered

[Goshtasby 99]

Modali ty to Model

+Mathematically definedcompartimental model

of gross brain structures[Maintz 98]

Patient to Modali ty

[Azuma 99]

Figure M1.7: Classification according to image modali ties involved inregistration

15

Considering the subject criteria, when all the images involved in a registrationtask are acquired from a single patient, it’s referred as intra-subject registration. Inter-subject registration occurs if the registration is accomplished using two images ofdifferent patients (or a patient and a model). If one image is acquired from a singlepatient, and the other image is somehow constructed from an image informationdatabase obtained using imaging of many subjects, it’s named “atlas registration” .

Finally, the last criterion considers the par t of the body imaged. Since this isrelated to all body's parts, we can find specialized methods for: head (brain, eye ordental), thorax (entire, cardiac or breast), abdomen (general, kidney or li ver), pelvisand perineum, limbs (general, femur, humerus, or hand) and spine and vertebrae.Table M1.1, presented by [Maintz 98] summarizes these nine criteria for medicalimage registration classification.

4.2. Image Registration Techniques

Several image registration techniques have been studied and developed in thelast few years [W'estermann 96, Zuiderveld 96, Bro-Nielsen 97, Vemuri 97, Gaens98, Penney 98, West 99]. The majority of them try to improve final results inperformance, accuracy and reliabilit y, using different approaches (e.g. cross-correlation, Fourier method, surface registration and mutual information) [Brown 92,Maintz 98]. Analyzing recently published techniques, it’s possible to notice apreference in the development of voxel property based registration methods thatoperates directly on the image gray values. In these case, two distinct approaches arefound: the first is to immediately reduce the image gray value content to arepresentative set of scalars and orientations, and the second is to use the full i magecontent throughout the registration process.

Principal axes and moments based methods are the most common examples ofreductive registration methods. In the second one, geometric moment’s alignment isdone, which usually needs the segmentation of some correspondent structures, unlesssimilar structures are generated by the acquisition modaliti es. Within principal axesthe image principal orientations (principal axes) are computed and registration is thenperformed by aligning the principal orientations. The result is usually not veryaccurate, and the method, that is easy to implement, is not equipped to handledifferences in scanned volume well .

The most interesting methods currently in use are the voxel property basedmethods using the full i mage content, since they allowed important progress in imageregistration. These methods do not start with reducing the gray valued image torelatively sparse extracted information, but use all the available informationthroughout the registration process. Its general principle consists of quantifying thequality of matching with respect to a similarity measure of the images’ overlappingvoxels. So, it is a very flexible registration method, with a high cost, that can beapplied in almost any medical application area [Roche 98, Maintz 98]. Among severalproposed methods in the literature [Penney 98, Roche 98], cross-correlation andmutual information similarity measures are the most used techniques.

16

Table M1.1: Cr iter ia for registration classificationCr iteria Subdivisions

Spatial dimensions: 2D/2D, 2D/3D, 3D/3DDimensionality

Time series with spatial dimensions: 2D/2D, 2D/3D, 3D/3DStereotactic frame

InvasiveFiducials (screw markers)Mould, frame, dental adapter, etc.

ExtrinsicNon-invasive

Fiducials (skin markers)Anatomical

Landmark basedGeometricalRigid models (points,curves,surfaces)

Segmentation basedDeformable modelsReduction to scalars/vectors

Intrinsic

Voxel property basedUsing full i mage content

Registration basis

Non-image basedRigidAffineProjective

Nature of transformation

CurvedLocal

Domain of transformationGlobal

Initialization suppliedInteractive

No initialization suppliedUser initializationUser steering/correctingSemi-automaticBoth

Degree of user interaction

AutomaticParameters computed

Optimization procedureParameters searched forMonomodalMultimodalModali ty to model

Modalities involved

Patient to modali tyIntrasubjectIntersubjectSubjectAtlas

Brain or skullEyeHeadDentalEntireCardiacThoraxBreastGeneralKidneyAbdomenLiverGeneralFemurHumerusLimbs

HandPelvis and perineum

Part of the body imaged

Spine and vertebrae

Similarity measurements or matches metric give a measure of the degree ofsimilarity between an image and a template (subvolume), or between two images[Brown 92]. This measure of similarity, that is image dependent, could be divided in:� Signal-based: sum of absolute differences;� Spatial property: spatial transform coeff icients;

17

� Statistical: correlation coeff icient, K-S test;� Geometric: distance metrics;� Information theory: entropy, mutual information [Goshtasby 99].

Cross-correlation, a basic statistical approach to registration, is an example ofsimilarity measure often used for template matching or pattern recognition.Considering a template T and an image I, where T is small compared to I, a two-dimensional normalized cross-correlation function, C(u,v), measures the similarity foreach translation. If the template matches the image exactly, except for an intensityscale factor, at a translation of (i,j), the cross-correlation will have its peak at C(i,j).Thus, by computing C over all possible translations, it is possible to find the degree ofsimilarity for any template-sized window in the image. It’s important to notice that thecross-correlation must be normalized since local image intensity would otherwiseinfluence the measure. This method is generally useful for images that are misalignedby small rigid or aff ine transformations [Brown 92].

Recently, especially for multimodal applications, a new information-theoreticapproach has been presented for registration. In this case, registration is achieved byadjustment of the relative position and orientation until the mutual informationbetween the images is maximized [Colli gnon 95, Viola 95, and Wells 96]. Thisapproach, which works directly with image data, has been largely used [Maes 97,Meyer 97, Gaens 98, Leventon 98, Maintz 98a].

Mutual information is a measure from the field of information theoryintroduced for matching of medical images. The mutual information of two images isexpressed in terms of the entropy of the images. Entropy could be a measure ofuncertainty, or a measure of the crispness of the histogram of the gray values, where asingle, sharp peak in the histogram (i.e. a homogeneous image) yields low entropy.For two images it’s possible to build a two-dimensional joint histogram to estimatethe joint probabilit y distribution of the two images (Figure M1.8). It is assumed that iftwo images are matched, the entropy of their joint histogram will be minimized,because of the peaks of corresponding gray values are sharpest. To register the imagesthe mutual information has to be maximized. Looking at the definition, this meansthat the joint entropy is minimized (i.e. the joint histogram has sharp peaks) and, atthe same time, the marginal entropy of the images is maximized [Pluim 99].

Figure M1.8: Joint histogram example [Pluim 99]

For new techniques validation, or to allow the accuracy quantification of theregistration algorithms, usually a “gold standard” registration, i.e. a bone markermatch, is done. In this case, a registration that has been accurately calculated usingfiducial markers is done and the result is compared with the images generated by, for

18

example, a voxel property based method. Sometimes a phantom could also be used asa source for the image acquisition. For example, Penney et al. made experimentsusing a phantom consisted of f ive lumbar vertebrae and pelvis encased in acrylic,which is approximately tissue equivalent at diagnostic X-ray energies [Penney 98a,Pluim 99].

However, although it’s possible to consider a gold standard, one of the mostdiff icult tasks related to registration still i s the quantification of how accurate is thereached result. Validation studies are only now emerging and usually the procedure toget a gold standard couldn’ t be done regarding clinical practice, since it’s onlypossible to supply a measure of accuracy by reference to control phantom studies,simulations or other registration methods. With the goal of minimizing the problem ofvalidating registration, a li st of items is usually analyzed.

This li st includes precision, accuracy, robustness/stabilit y, reliabilit y, resourcerequirements, algorithm complexity, assumption verification and clinical use.Precision is related with the typical systematic error that can be obtained when theregistration algorithm is supplied with idealized input, and its measure can beobtained concerning the entire registration system, or applying to specificcomponents, as the patient or the acquisition. Accuracy refers to the actual, “ true”error occurring at a specific image location. Robustness or stabilit y refers to the basicrequirement that small variations in the input should result in small variations in theoutput. Reliabili ty is the requirement that the algorithm should behave as expected,given a reasonable range of possible clinical input. Resource requirements concernthe material and effort involved in the registration process. The algor ithmcomplexity and related computation time should be adapted to the time and resourceconstraints of the clinical environment. The assumptions on reali ty made in theparadigm and optimization modeling should be verified to hold up suff iciently inpractice. Finally, the clinical use should be verified, which means if the registrationprovides for the clinical need and if its use outweigh available alternatives [Maintz98].

Analyzing current literature it’s possible to observe that recently intrinsicmethods are gaining more importance and are being extensively used. Amongintrinsic methods, the surface based methods appear most frequently, closely followedby “ full image content” voxel property based methods. According to Maintz andViergever [Maintz 98], surface based methods are, on the average, still faster thanvoxel property based methods, but they cannot cope with shift of relevant anatomyrelative to the surface used in the registration. They also aff irm that many methodscan still be considered barred from meaningful clinical application by the fact thatthey are as yet improperly validated. Although the proper verification methods areknown in most cases, for most applications the painstaking work of conducting themany experiments involved is only now starting.

5. Segmentation

Segmentation is any technique that takes an image and separates its elementsin connected regions, which present the same desired property or characteristic.These regions can be characterized by pixels, boundary or texture, thus defining threemain classes of segmentation methods: region-based segmentation, contour methodsand texture-based segmentation.

Different information is obtained from these segmentation approaches. Whilecontour based methods deal with geometric characteristics of the elements, region-

19

based segmentation techniques are concerned with the content of the image pixels andtexture-based segmentation considers image properties like regularity and repetitionof features.

Generally, segmentation algorithms are based on the homogeneity ordiscontinuity of the image function. Homogeneity is related to the inner part of theelements while discontinuity is associated with the region borders, and contrast. Somealgorithms combine both sources of information or consider such data in differentscales. Segmentation is one of the hardest problems in medical image analysis[Greenes 90] and there is no better method because regions often do not havecontinuous borders or homogeneous interior.

Considering user actions during segmentation processes, we can distinguishmanual, automatic and interactive segmentation methods. Manual segmentation isany technique where the user decides which pixels belong to the element beingseparated from the image. This technique can be associated to painting or drawingprocedures onto the image. Automatic segmentation methods are based on the supplyof some parameters to an algorithm that processes the image and determines whichpixels pertain to the desired element. The simplest form of automatic segmentationuses a simple threshold operation where a pixel value or interval is used as thedecision criterion to separate a region in the image. Interactive segmentation methodsrely on user orientation during the processing to determine which pixels should bepart of the region (or boundary). It should be noticed that the classifications ofsegmentation methods considering element representation or user intervention areorthogonal, i.e., we can have an automatic, region-based method as well as anautomatic, texture-based method.

Manual segmentation is still employed in many cases. However, a lot ofefforts have been dedicated to improve either automatic or interactive methods.Mathematical morphology is a powerful tool to segment images. Examples of its usein medical images can be found in [Hohne 92]. Interactive methods are gainingacceptance due to the problems in finding a general solution to automaticsegmentation. The main causes of failure of automatic methods are [Olabarriaga 97,Smeulders 97]:� Insuff icient information in the image, due to high noise rates, low contrast

or unexpected textures� Absence of pictorial information, which might occur when an element iscovered by other. The algorithm can not guess where the object border isbut a user could;� Pictorial information is obscured by a neighbor object, i.e., the edge of anobject has high contrast than that of the desired element;� Pictorial information differs from the intended computational model. Thisoccurs when the segmentation algorithm is trying to identify an elementwith a pre-defined shape.

Interactive segmentation methods like those based on deformable contoursmodels [McInerney 96] seems to be one of the best approaches when automaticmethods fail .

Depending on the desired level of accuracy of the segmentation results, onecould use methods like the marching lines [Thirion 96] or marching cubes [Lorensen87] algorithms. The first one extract lines defined at the intersection of 2 isosurfaces,

20

with subvoxel accuracy, whilst marching cube is a well -known algorithm forextracting isosurfaces from volume data based on a threshold parameter.

6. References

[Azuma 99] Azuma, R. T. Registration Errors in Augmented Reali ty.Available at http://www.cs.unc.edu/~azuma/azuma_AR.html(Sep. 1999).

[Bansal 98] Bansal, R.; Staib, L. H.; Chen, Z. et al. A Novel Approach for theRegistration of 2D Portal and 3D CT Images for Treatment SetupVerification in Radiotherapy. In: INTERNATIONALCONFERENCE ON MEDICAL IMAGE COMPUTING ANDCOMPUTER-ASSISTED INTERVENTION, MICCAI, 1., 1998,Cambridge, MA. Proceedings... Berlin: Springer-Verlag, 1998.p. 1075-1086. (Lecture Notes in Computer Science, v. 1496).

[Borgefors 88] Borgefors, G. Hierarchical Chamfer Matching: a Parametric EdgeMatching Algorithm. IEEE Transactions on Pattern Analysisand Machine Intelli gence, v. 10, p. 849-865, 1988.

[Brown 92] Brown, L. G. A Survey of Image Registration Techniques. AcmComputing Surveys, ACM Press, New York, v. 24, n. 4, p. 325-376, Dec. 1992.

[Bro-Nielsen 97] Bro-Nielsen, M.; Gramkow, C.; Kreiborg, S. Non-rigid ImageRegistration Using Bone Growth Model. In: FIRST JOINTCONFERENCE COMPUTER VISION, VIRTUAL REALITYAND ROBOTICS IN MEDICINE AND MEDICIALROBOTICS AND COMPUTER-ASSISTED SURGERY,CVRMed-MRCAS, 1., 1997, Grenoble, France. Proceedings...Berlin: Springer-Verlag, 1997. p. 3-12. (Lecture Notes inComputer Science, v. 1205).

[Collignon 95] Colli gnon, A.; Maes, F.; Delaere, D. et al. Automated multi -modali ty image registration based on information theory. In:BIZAIS, Y.; BARILLOT, C.; DI PAOLA, R. (Eds.).Information Processing in Medical Imaging. Dordrecht:Kluwer Academic Publishers, 1995, p. 263-274.

[Gaens 98] Gaens, T.; Maes, F.; Vandermeulen, D. et al. Non-rigidMultimodal Image Registration Using Mutual Information. In:INTERNATIONAL CONFERENCE ON MEDICAL IMAGECOMPUTING AND COMPUTER-ASSISTEDINTERVENTION, MICCAI, 1., 1998, Cambridge, MA.Proceedings... Berlin: Springer-Verlag, 1998. p. 1099-1106.(Lecture Notes in Computer Science, v. 1496).

[Goshtasby 99] Goshtasby, A.; Szeliski, R.; Wolberg, G. 2D and 3D ImageRegistration and Image Warping. In: INTERNATIONALCONFERENCE ON COMPUTER GRAPHICS ANDINTERACTIVE TECHNIQUES - SIGGRAPH, 26., 1999, LosAngeles, CA. Course Notes 02. [S.l.: s.n.], 1999.

21

[Greenes 90] Greenes, R. A.; Brinkley, J. F. Radiology Systems. In:SHORTLIFFE, Edward H.; PERREAULT, Leslie E. (Eds.).Medical Informatics. USA: Addison-Wesley, 1990. p. 324-365.

[Hohne 92] Höhne, K.H.; hanson, W. Interactive 3-D Segmentation of MRIand CT volumes using morphological operations. Journanl ofAssisted Tomography, v. 16, n.2, pp. 285-294, March/April1992.

[Jain 89] Jain, A.K. Fundamentals of Digital Image Processing. PrenticeHall , Englewood Cli ffs, NJ, 1989.

[Leventon 98] Leventon, M. E.; Grimson, W. E. L. Multi -modal VolumeRegistration Using Joint Intensity Distributions. In:INTERNATIONAL CONFERENCE ON MEDICAL IMAGECOMPUTING AND COMPUTER-ASSISTEDINTERVENTION, MICCAI, 1., 1998, Cambridge, MA.Proceedings... Berlin: Springer-Verlag, 1998. p. 1057-1066.(Lecture Notes in Computer Science, v. 1496).

[Lorensen 87] Lorensen, W. E.; Cline, H.E. Marching Cubes: a high resolution3D surface reconstruction algorithm; Computer Graphics, v. 21,n.4, July 1987.

[Maes 97] Maes, F.; Colli gnon, A.; Vandermeulen, D. et al. Multimodali tyImage Registration by Maximization of Mutual Information.IEEE Transactions on Medical Imaging, New York, v. 16, n.2, p. 187-198, April 1997.

[Maintz 98] Maintz, J. B. A.; Viergever, M. A. A Survey of Medical ImageRegistration. Medical Image Analysis, Oxford, v. 2, n. 1, p. 1-36, 1998.

[Maintz 98a] Maintz, J. B. A.; Meijering, E. H. W.; Viergever, M. A. A.General Multimodal Elastic Registration Based on MutualInformation. In: MEDICAL IMAGING 1998 - IMAGEPROCESSING, SPIE, 1998, San Diego, Cali fornia.Proceedings... 1998. p. 144-154.

[Mahoney 96] Mahoney, D. P. The Art and Science of Medical Visualization.Computer Graphics Wor ld, Tulsa, OK, v. 19, n. 7, p. 25-32,July 1996.

[McInerney 96] McInerney, T.; terzoppoulos, D. Defomable models in medicalImage Analysis: a survey. Medical Image Analysis, v. 1, n.2.pp. 91-108, 1996.

[Meyer 97] Meyer, C. R.; Boes, J. L.; KimM, B. Demonstration of Accuracyand Clinical Versatili ty of Mutual Information for AutomaticMultimodali ty Image Fusion Using Aff ine and Thin-plate SplineWarped Geometric Deformations. Medical Image Analysis,Oxford, v. 1, n. 3, p. 195-206, 1997.

[Olabarriaga 97] Olabarriaga, S.D.; Smeulders, A.W.M. Setting the Mind forIntell igent Interactive Segmentation: Overview, Requirements,and Framework. In: DUNCAN, J.; GINDI, G. (eds.)

22

INTERNATIONAL CONFERENCE ON INFORMATIONPROCESSING IN MEDICAL IMAGING (IPMI'97), 15., 1997,Poultney VT. Proceedings…(Lecture Notes in Computer Science1320).

[Pelizarri 89] Pelizarri, C. A., Chen, G. T. Spelbring, D. R. et al. AccurateThree-dimensional Registration of CT, PET and/or MR Imagesof the Brain. J. Comput. Assisted Tomogr., v. 13, p. 20-26,Jan./Feb. 1989.

[Penney 98] Penney, G. P.; Weese, J.; Little, J. A. et al. A Comparison ofSimilarity Measures for Use in 2D-3D Medical ImageRegistration. IEEE Transactions on Medical Imaging, NewYork, v. 17, n. 4, p. 586-595, Aug. 1998.

[Penney 98a] Penney, G. P.; Weese, J.; Little, J. A. et al. A Comparison ofSimilarity Measures for Use in 2D-3D Medical ImageRegistration. In: INTERNATIONAL CONFERENCE ONMEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION, MICCAI, 1., 1998, Cambridge,MA. Proceedings... Berlin: Springer-Verlag, 1998. p. 1153-1161. (Lecture Notes in Computer Science, v. 1496).

[Pluim 99] Pluim, J.; Maintz, T. Registration of multimodal medical images.Available athttp://www.cv.ruu.nl/Research/Registration/registration-frame.html (Aug. 1999).

[Rhodes 97] M. L. Rhodes. Computer Graphics and Medicine: A ComplexPartnership. IEEE Computer Graphics & Applications, LosAlamitos, CA, v. 17, n. 1, p. 22-28, Jan.-Feb. 1997.

[Roche 98] Roche, A.; Maladain, G.; Pennec, X. et al. The CorrelationRation as a New Similarity Measure for Multimodal ImageRegistration. In: INTERNATIONAL CONFERENCE ONMEDICAL IMAGE COMPUTING AND COMPUTER-ASSISTED INTERVENTION, MICCAI, 1., 1998, Cambridge,MA. Proceedings... Berlin: Springer-Verlag, 1998. p. 1115-1124. (Lecture Notes in Computer Science, v. 1496).

[Russ 95] Russ, J.C. The Image Processing Handbook - 2nd Edition.CRC Press, Boca Raton, FL, 1995.

[Smeulders 97] Smeulders, A.W.; Olabarriaga, S.D. et al. Design Considerationsfor Interactive Segmentation. In: JAIN, R.; SANTINI, S. (eds.)VISUAL INFORMATION SYSTEMS., 1997. Proceedings…San Diego, Knowledge Systems Institute, 1997, pp. 5-12.

[Thirion 96] Thirion, J-P.; Gourdon, A. The 3d marching lines algorithm.Graphical Models and Image Processing, v. 58, n.6, pp. 503-509, November 1996.

[Vemuri 97] Vemuri, B. C.; Huang, S.; Sahni, S. et al. A Robust and Eff icientAlgorithm for Image Registration. In: INTERNATIONALCONFERENCE ON INFORMATION PROCESSING INMEDICAL IMAGING, IPMI, 15., 1997, Poultney, Vermont.

23

Proceedings... Berlin: Springer-Verlag, 1997. p. 465-470.(Lecture Notes in Computer Science, v. 1230).

[Viola 95] Viola, P.; Wells III, W. M. Alignment by maximization of mutualinformation. In: INTERNATIONAL CONFERENCE ONCOMPUTER VISION (E. Grimson, S. Shafer, A. Blake and K.Sugihara, eds.), Proceedings... Los Alamitos, CA: IEEEComputer Society Press, 1995, p. 16-23.

[Wells 96] Wells III, W. M.; Viola, P.; Atsumi, H. et al. Multi -ModalVolume Registration by Maximization of Mutual Information.Medical Image Analysis, Oxford, v. 1, n. 1, p 35-51, 1996.Available by ftp.ai.mit.edu:/pub/sw/papers/mia-95-dist-color.ps.gz (Ago. 1999).

[West 99] West, J.; Fitzpatrick, J. M.; Wang, M. Y. et al. RetrospectiveIntermodali ty Registration Techniques for Images of the Head:Surface-Based Versus Volume-Based. IEEE Transactions onMedical Imaging, New York, v. 18, n. 2, p. 144-150, Feb. 1999.

[Westermann 96] Westermann, B.; Hauser, R. Non-Invasive 3-D PatientRegistration for Image-Guided Skull Base Surgery.Computers& Graphics, Oxford, v. 20, n. 6, p. 793-799, 1996.

[Zuiderveld 96] Zuiderveld, Karel J.; Koning, Anton H. J.; Stokking, Rik et al.Multimodali ty Visualization of Medical Volume Data.Computers& Graphics, Oxford, v. 20, n. 6, p. 775-791, 1996.

24

25

MODULE 2: Visualization and Representation

Models

1. Introduction

Medical data obtained from images are used both to represent body's parts andcomplete virtual humans. Usually for visualization and quantification purposes ofspecific parts a volume of medical data suff ice. However, the simulation of morecomplex procedures requires detailed representation models. The following sectionspresent an overview of volume visualization as well as models used to representvirtual bodies.

2. Interactive Volume Visualization

Volume Visualization denotes the set of techniques used in the presentation ofvolume data, i.e., data associated to positions (often regularly) spaced in some 3Ddomain. Its main goal is to show the interior of volumetric objects to allow theexploration of its complex structure and to facilit ate its comprehension. In general,volume visualization is a projection process of a multidimensional data set in a plane.It was firstly used in the beginning of the 70’s [Greenleaf 70].

For visualizing volumetric medical data, sequences of 2D images are piled upto recreate the three-dimensional structure (Figure M2.1). Usually, in this stepinterpolation of adjacent slices is needed for the generation of new slices (FigureM2.1b), since one of the problems resulting from image acquisition is the spacebetween slices. This problem occurs because the sampling interval between slices isnormally greater than the generated image resolution, and then the volume voxels arenot cubic. After interpolation this size distortion is corrected, so that the visualizationalgorithm could generate correct proportion projections. However, although the newvolume has more slices, the data set isn't better, because the new slices are producedfrom sampled data.

(a) (b)

Figure M2.1: Slice interpolation

26

It’s important to notice that one of the biggest problems in the visualizationand interaction with volumetric data some years ago was the relation between thehuge data quantity and the computer storage capacity. However, technologicaladvances in hardware and software broadened the application of those techniques, andrecently, real time visualization and virtual reality equipment have enabled surgicalprocedures simulation imposing new goals to visualization algorithms. Today, we canmeasure the interest in volume visualization by the extensive literature dedicated tovolume data manipulation, associated visualization techniques and relatedapplications [Kaufman 91, Lichtenbelt 98]. There are several algorithms andtechniques especially for medical data, and we couldn't say that one technique isbetter than others are. Each technique is suitable for a specific kind of application.

Volume visualization techniques can be classified in two main groups: surfacevisualization, used in three-dimensional reconstruction, and direct volumevisualization. The algorithms belonging to the first group, described in Section 2.1.2,usually separate the volume subset that represents a specific anatomical surface usinga segmentation technique. During the process, this volume subset is approximated bya set of polygons and exhibited with conventional computer graphics techniques. Thesecond group of volume visualization techniques (Section 2.1.3) is based on transferfunctions that establish the relation between voxel values (denoting, for example,tissue density) and color/opacities. The visualization of the structure of interest insidethe volume is accomplished by visiting all (or almost all ) voxels and applying thetransfer function to build the image.

Initially, volume visualization algorithms were only developed to show theinterior of the volume, to allow the identification of its inner regions and structuresand to facilit ate the comprehension of its complex structure. Later on, user interactionduring the visualization process became important in order to allow the user to changeparameters and perform a dynamic navigation process. In this context, interactivevisualization techniques advanced in order to provide a better comprehension of thegreat volume of medical data generated by the several image-based instrumentsavailable today.

In fact, the need of interactive data visualization, that is, the abilit y to changethe point-of-view and the region of interest, exists for a long time. Besides thepossibilit y of selecting different regions and structures, manipulating and animatingthe reconstructed spatial structure, some other examples of interactive visualizationtools are: cross sections, cut planes, cut volumes, and classification tables. Theselection of regions of interest is usually accomplished by submitting the volume to asegmentation process that isolates the desired structures. The simplest form of doingthis is to specify an interval of voxel values, thus indicating the tissue density thatcorresponds to the desired structure (Figure M2.2b). Cross sections (Figure M2.2a),cut volumes (Figure M2.3a), and cut planes (Figure M2.3b) are used to show sectionsof the volume in a different orientation from the one used in the original imageacquisition process. This produces a new image without having to submit the patientto another imaging procedure. Classification tables, namely color tables and opacitytables, are used to assign colors and transparency levels to different intervals of voxelvalues (e.g. Figure M2.4), changing the image significantly [Silva 97]. These are theways of displaying different structures at the same time in direct volume visualizationtechniques.

27

Figure M2.2: Cross-section (a) and selection of region of interest (b) [Silva 97]

Figure M2.3: Cut volume and cut plane, respectively [Silva 97]

Figure M2.4: Use of classification table to separate regions of interest [Silva 97]

A complex application of interactive manipulation of medical data is surgerysimulation, which has a great utilit y in helping the training of new physicians.Simulators allow the achievement of virtual surgeries emphasizing real timeinteraction between the user and medical instruments, surgical techniques and modelsthat represent several anatomical structures and physiologic processes. Module 4 isconcerned with this important class of application.

28

2.1. Three-Dimensional Reconstruction

Reconstruction is the abstract "rebuilding" of something that has been tornapart. In medical imaging context it has three basic elements: data acquisition, three-dimensional model reconstruction and visualization. So, the majority of inputs for thethree-dimensional reconstruction are medical data consisting of slices (images) takenat regular intervals throughout a portion of the body (i.e., the data are acquired onepiece at a time). As reported before, these data are derived from a variety of medicalimage modaliti es such as MRI or CT. After reconstructing the parts imaged, one isable to view or visualize all the data as if it were put together again [Olabarriaga 89,Montgomery 99].

Considering the reconstruction step, three basic structures could be used torepresent a selected portion of the body: contours, surface and volume. The first one isan immediate representation and involves a segmentation technique to separate thevarious tissues in each slice. So, the contour visualization set allows the approximatedetermination of the space portion that an object, or organ, takes part. This simplestructure, however, has limited visualization possibiliti es. Another way to represent athree-dimensional object is to show its external portion. Often, in this case a surfacevisualization algorithm is used to create a three-dimensional representation of thestructures. This surface can be described using a polygonal mesh extracted, forexample, from a contour set. The third alternative is the representation of the volumethrough its voxels, when the object is stored as a cubic array of elements that describethe characteristics of the specific region in 3D space occupied by it.

Finally, once the model is obtained, another algorithms are used to provide thevisualization and animation of the selected portion of a human body. This is the mostimportant step of a system, because it has to provide the observation from differentpoint of views, as well as support to several interaction techniques, with the main goalof facilit ate the model reconstruction comprehension [Olabarriaga 89].

2.2. Surface Visualization

Surface volume visualization technique uses geometrical primitives, aspolygonal meshes or contour lines, to present the volume, which means that its basicrepresentation is a contour. Usually, surface rendering techniques rely onsegmentation and reconstruction processes to obtain these geometric primitives thatcorrespond to the structures to be rendered with traditional computer graphicstechniques and then can take advantage of common graphics acceleration hardware tobe employed in real-time applications. Among the published algorithms that use thisapproach we can mention the contour connecting [Fuchs 77, Keppel 75] and marchingcubes [Lorensen 87] algorithms. Some problems that are found in these algorithms arethe occasional false surface pieces, the wrong manipulation of littl e datacharacteristics, and the diff icult surface representation for some human bodystructures. The great advantages, however, are the speed and the small storage spacerequired.

To exempli fy how the surface volume visualization algorithms work, themarching cubes algorithm will be described in this section. Lorensen and Cline[Lorensen 87] proposed this method to create a triangle mesh model from volume datawith constant density (isosurface).

The user has to specify a threshold value, which determines a density valuethat defines the region inside the volume that has to be emphasized. Once this value

29

has been determined, the volume data is processed in slice sequence, extracting theisosurface. The pixels in the slices are treated as isolated points in the space, with adistance from each other. A logic cube, or cell , is formed by eight pixels, four fromeach adjacent slice (Figure M2.5) [Lorensen 87]. The algorithm moves sequentially,one cube at each time, identifying the intersection of the surface with the cube (that’sbecause the name is Marching Cubes).

Figure M2.5: Logic cube

At this moment, each vertex receives a flag "1" if its value is above thespecified threshold that identifies the interest surface. If the vertex value is below thethreshold, the vertex receives flag "0". The surface intersection with the cube islocated in the edges where one vertex is inside the surface and the other is outside.This information is suff icient to obtain a surface approximation inside the cube. Aseach one of the eight vertices from the cube could have one of the two values (zero orone), there are 256 possible ways to approximate the surface inside the cube. Thismeans that the 256 cases could be easily implemented as a table to optimize algorithmperformance. However, considering the two different symmetries of the cube, the 256cases could be easily reduced to 14 basic cases. First, the topology of the triangulatedsurface is unchanged if the relationship of the surface values to the cubes is reversed(i.e. vertex values are inverted). Complementary cases, where vertices greater than thesurface value are interchanged with those with smaller value, are equivalent. Thus,only cases with zero to four vertices greater than the surface value need to beconsidered, reducing the number of cases to 128. Second, the rotational symmetryreduce the remaining 128 cases to only 14 by inspection. If all vertices flags havevalue zero, the surface doesn’ t cross the cube. On the other hand, if all vertices havevalue one, the cube is inside the object and doesn’ t need to be considered too. FigureM2.6 shows the 14 topological configurations for a surface inside a cube.

Vertex values are successive placed in a byte that is treated as an index to thetopologies table. The index determines the edges that have intersection with thesurface. These intersections establish the triangle vertex positions that approximatethe surface form. The topology table only shows the edges where the triangle vertex isover. The vertex correct positions are calculated by linear interpolation.

In the last algorithm step, a unit normal for each triangle vertex is calculated.For that, the normal vectors of each cube vertex are processed according to thedirectional gradients. So, the triangle vertex normal vectors are calculated by vertexnormal vectors linear interpolation. It means that during any moment of algorithmexecution it’s necessary to have only four slices of the volume data in memory. Forthe evaluation of surface intersection with the logical cubes, only two slices areneeded, but the normal vector calculation needs two more slices.

A problem found in this algorithm is that sometimes it connects a wrong set ofthree points during the triangle generation, resulting in wrong isosurface elements.

30

This occurs because some ambiguities could happen in some cube arrangements.These ambiguities occur when two adjacent cubes have two diagonally oppositevertex with value one and another two diagonally opposite vertex with value zero.The triangulation in these cubes could generate adjacent faces or not. This problemcould be solved if the algorithm is consistent in its choice and the complementarycubes, as in the original algorithm, aren’ t considered equivalent. These leads to eightnew cases for the topology table, as shown in Figure M2.7. Another problem of theMarching Cubes algorithm is that the number of generated triangles could beprohibitive and appropriated optimization methods for these cases had already beenproposed.

Figure M2.6: Cases of sur face intersection with a logic cube [Lorensen 87]

Figure M2.7: Additional topologies

2.3. Direct Volume Visualization

In this class of algorithms, the volume data are directly exhibited, whichmeans that the images are generated through the transformation, shading andprojection of 3D voxels into 2D-pixel space, avoiding the use of geometric primitives.These techniques usually demand great computational processing, but produce imagsewith excellent quality, since all the voxels located in the line of view are used inimage generation, allowing the visualization of parts inside the volume. The mostpopular algorithm of this group is Ray-Casting [Levoy 88, 90] which has been

31

described, implemented, and optimized by several authors [Jones 97]. Anotheralgorithms that could be mentioned are splatting [Westover 89] and V-Buffer [Upson88].

Because of its wide acceptance, the ray-casting algorithm will be described inthis section, as an example of a direct volume visualization method. After Levoy[Levoy 88, 90] described it, several authors developed extensions with the main goalof improving its performance. Traditionally, the algorithm spans the projection planeand fires the rays into the scene. The projection plane corresponds to the imagewindow, and the rays are associated to the pixels for exhibition. Usually parallel rays,orthogonal to the projection plane are traced. This approach generates a volumeorthographic projection (Figure M2.8).

rProjection Plane

(image with NxM pixels)

Data Volume

Object Space

O

Figure M2.8: Ray-casting of volumetr ic data

In this case, the volume corresponds to the scene, while the plane projectioncorresponds to the camera. Camera orientation is defined by a vector (r) and the targetis a point inside the data volume. For orthographic projections, the distance betweenthe camera and the volume doesn’ t change the image generated, since the traced raysare parallel. So, camera parameters have to be provided by the user and planeprojection center, camera position and target are defined by the system.

Parallel rays are traced from the observer position to the volume data.Considering that the image is a rectangle of NxM pixels (plane projection), one ray iscast for each one of these pixels. So, for each ray, sample points are calculatedconsidering a fixed step on the path determined by the ray. The algorithm cancalculate and accumulate both color and opacity values along the ray for obtainingpixel color. Figure M2.9 presents a diagram of Levoy's algorithm that ill ustrates howthe pixel colors are calculated from voxels values and Figure M2.10 presents anexample of an image generated with this algorithm implementation.

Voxels Values

ClassificationShading

Voxel Colors Voxel Opacities

Ray-tracing / Resampling

Pixel Colors

Sample Colors Sample Opacities

Composition

Ray-tracing / Resampling

Figure M2.9: Direct volume rendering algor ithm diagram

32

Figure M2.10: Image generated by using Levoy's direct volumerendering algor ithm [Silva 97]

Although its large utili zation, this algorithm has a poor performance becauseof the great number of operations needed for color pixel determination. So, a lot ofoptimization techniques have been proposed for this algorithm.

2.3.1. Acceleration Techniques

Since each ray casting calculation is independent, the first accelerationtechnique that comes to mind is to use parallelism. Then, because ray-castingalgorithms are adequate for parallel implementation, one way to solve the volumerendering computational cost problem is to use computers with parallel architecture,or to execute the algorithm in several computers, or workstations forcing a distributedparallel memory program implementation. As reported in the literature, good resultsare also obtained with the use of multiprocessor systems with shared memory,compared to distributed memory architectures [Rowlan 94, Zuiderveld 96].

To ill ustrate this kind of acceleration technique, the idea of the parallel ray-casting volume rendering algorithm developed by Rowlan et al. [Rowlan 94] will bebriefly described. This algorithm uses image-ordered work subdivision and distributesthe data and the computational load to individual processors. After the subvolumes aredistributed, all processors perform local ray tracing of their respective subvolumesconcurrently. No interprocess communication takes place during the ray tracingprocess. After each processor generates a subimage, the final image is obtained bycomposing the several subimages.

A new volume rendering method presented by Knittel [Knittel 95] offers highspeed based on a lossy data compression scheme that drastically reduces the memorybandwidth and computing requirements of perspective ray-casting. Starting fromclassified and shaded data sets, two compression methods can be used to compress ablock of 12 voxels into 32 bits. All blocks of the data set are processed redundantly,yielding a data structure, which avoids multiple memory accesses per ray-point. As aside effect, the trili near interpolation of data coded in such a way is very muchsimpli fied. These techniques allow interactive navigation through the volume andmoreover, the algorithm provides depth-cueing and the semi-transparent display ofdifferent materials.

As mentioned before, processing starts with the classification of the data set.The voxels are grouped and tagged according to the material they belong to (e.g.,bone and tissue). Each material is shaded separately according to whether only itssurface should be displayed or its entire region. Thus, one opacity transfer function isused for each material having an upper bound equal to 1. Then the data is passed tothe compression stage where Block Truncation Coding or Color Cell Compressionalgorithms is used [Knittel 95].

33

Another optimization, presented by Haley [Haley 96], describes a newalgorithm for an eff icient incremental rendering of volume data set. The primary goalof this algorithm is to give to average workstations the abilit y to eff iciently rendervolume data received over relatively low bandwidth network links in such a way thatrapid user feedback is maintained. The rendering algorithm makes use of the eff icientShear-Warp Factorization rendering method developed by Lacroute [Haley 96] tosolve these problems, and represents the volumetric data using a hierarchical datastructure which exploits the multiscale nature of an octree data structure. So, thealgorithm builds a compressed hierarchical representation of the volume data using anoctree, which may then be transmitted to a client. On reception of this data the clientmay incrementally classify and render the data using a modified Shear-WarpFactorization algorithm that makes use of a run-length encoding technique andreduces the memory footprint of the original algorithm by a factor of more than two,while maintaining good rendering performance. Due to the fact that the incrementalclassification and rendering times are faster than the network transmission times,multiple incremental renderings may be performed on the data as it arrives, giving theuser the abilit y to rapidly examine overall characteristics of the data. The result of thismethod is a rapid incremental volume renderer that reduces the memory requirementsand is appropriated for utili zation in average workstations to allow an interactiveexploration of the volume models in a network.

Jones [Jones 97] presented a method for a more eff icient volume datasampling. This method introduces the idea of choosing the distance used to obtain thesample along the ray, in such a way that the process could be reduced and thevisualization works for any observer position. It reduces around 50% of the neededoperations for the volume rendering. A observer-dependent, sampling interval iscalculated in such a way that each data sample occurs always in each “cube” face,thus allowing the utili zation of a bili near interpolation instead of a tri-linear one, andreducing the computational complexity. The generated images are a goodrepresentation of the volume, but some problems occur during animation, whenobservers' position varies constantly.

Another traditional acceleration technique is the adaptive termination, whichconsiders that the volume data is processes in “ front to back” order, i.e. it’s traversedfrom the observer position. In this technique, when a totally opaque sample is found,it is considered that the following samples within the same ray path will not influencethe pixel final color, since they are hidden. In this case, the ray could be finished,without the need of new sample calculations along the ray [Jones 97], as ill ustrated inFigure M2.11.

rays

Figure M2.11: Adaptive termination (white cubes represents opaque samples)

34

A lot of optimizations that could be done in the ray-casting algorithm werepresented by Silva [Silva 96], resulting in the development of an exploration volumedata tool with a very good performance [Silva 97]. Among the implementedoptimizations it’s possible to emphasize:� Elimination of slices interpolation.� Avoidance of f iring rays that doesn’ t reach the volume.� Spanning of the projection plane in center-edges direction.� Incremental spanning method in the volume surface that simpli fies the

intersection calculation between rays and volume.

2.4. Hybrid Visualization

As described in the Sections 2.1.2 and 2.1.3, it’s noticeable that while surfacevisualization techniques are faster but present an approximation of the structure ofinterest, direct volume visualization techniques demand greater storage capacity andhigher processing time, but display original data. However, direct volumevisualization has benefited from hardware and software advances and from thedevelopment of specific hardware for supporting real-time visualization, whichovercome the problem of processing time and storage capacity. It’s also important tomention that although both categories are suited for rendering volumetric data indifferent situations, or rendering different types of data, there are many applicationswhere volume data should be either displayed, or manipulated along with geometricmodeled objects. For example, in surgical planning, surgery simulation, and radiationtherapy planning, volume data obtained from 2D images taken with imaging devicesneed to be rendered with geometric models of prosthesis, surgical cuts and radiationbeams.

Techniques for simultaneous displaying volume data and geometric modelshave already been reported in the literature [Kaufman 90, Levoy 90a] either requiringconversion from one representation to the other or based on a hybrid data model. Anoptimization of the ray casting technique for direct volume rendering applied to bothvolume data and geometric model visualization was also developed recently and willbe briefly described here.

In this technique the first ray is fired into the scene from the central point of aprojection plane. The other rays start at adjacent points and are fired following acenter-border direction, allowing stopping the generating rays when the borders of thevolume (or the borders of the objects’ bounding rectangle) are reached. The samplingpoints in the volume along a ray are calculated as usual, considering a fixed step onthe path determined by the ray. Intersection points between the ray and the geometricmodels are calculated following Badouel’s technique [Badouel 90], and inserted in thesame path so the algorithm can accumulate both color and opacity values along theray. Both sampling and intersection points are parametrically represented so it’s easierto obtain the ordered set of samples in a ray. Color and opacity values for each samplepoint are obtained by sampling the voxels values or the geometric surface, which isconsidered as flat-shaded. The parametric representation of sampling and intersectionpoints also allows to correct color and opacity values when an intersection point liesbetween two volume sampling points, as can be seen in Figure M2.12.

35

P0OCOC P1P1

t0t0 t1t1

A

B

C

D

E

Sampling points

Rays

Geometric surface

Volumetricdata

Samplingstep� �

sPA

Figure M2.12: Sampling and intersection points along rays. Values inray C between P0 and P1 must be corrected due to the position ofintersection Oc, inside the inner structure.

Using bounding boxes, or bounding spheres for each object, or indexing thespace with an octree usually do optimize the computation of intersections in raytracing. When using bounding boxes or bounding spheres, each ray is firstly testedagainst the bounding volumes of objects [Whitted 80]. Intersections are thencalculated only for objects whose bounding volumes are intercepted by the rays. Theuse of octrees to index the object space avoids intersection calculation for objects thatare in a subspace out of the path of the ray [Glassner 84]. These techniques proved tobe useful in reducing the total processing time in ray tracing. Since in volumerendering projection can be parallel, and there are no second-generation rays, similarapproaches have been adopted in the technique that is being described. Instead ofbounding volumes, bounding rectangles are computed for each triangular face of thegeometric objects, and instead of subdividing the space in octants, the algorithmsubdivides the viewport in cells.

In the first approach, faces are projected into the viewport, boundingrectangles are determined, and then intersection points between a ray and each faceare calculated only for those rays that are fired from pixels inside the boundingrectangles.

In the latter optimization approach, the viewport is subdivided into squarescells and a list stores pointer to the bounding rectangles, which lie inside each cell .When a ray R is fired from a pixel inside cell C, only the bounding rectanglesbelonging to the cell C list are tested, in order to determine which faces are likely tobe intercepted by the ray. The size of the cells is calculated based on the average sizeof the bounding rectangles of each face as well as the total size of object projectiononto the viewport. Figure M2.13 shows some images produced with the hybrid raycasting described here.

36

Figure M2.13: Images obtained with hybr id ray casting

3. Human Bodies Modeling Methods

The goal of this section is the presentation of the main steps used in the designof a virtual human. A review of the main existing methods for body modeling ispresented. The three-dimensional models developed until now can be classified intofour categories: stick figures models, surface models, volume models and multi -layered models. The face modeling processes are presented as well as a method toreconstruct hands by using FFD (Free-Form Deformation). Some methods to simulatethe hair are quickly described.

3.1. Body Modeling Techniques

The human body can be structured in three levels: the skeleton, the net of themuscles and the surface representing the skin. Most of the systems base their modelson this “anatomic” structure, articulating a geometric model li ke the real skeleton[Steketee 85, Cachola 86, Magnenat 85], and sculpting a surface similar to the skin[Fetter 82]. Some authors instead, use a mixed model dealing with the both levels,skeleton and skin [Armstrong 86].

Other systems try yet to model the human body as a collection of primitivevolumes as spheres elli psoids or cylinders [Badler 79, Herbison-Evans 78]. All thesesystems present several advantages and disadvantages, as we will discuss now.

3.1.1. Stick Figure Models

Systems using stick figure models consist of a hierarchical set of rigidsegments (limbs) connected at joints. These models are called articulated bodies andmay be more or less complex, depending on the number of limbs and joints involved.Each joint can have up to three degrees of freedom (DOF) and the model can be moreor less complex, reaching more than 200 DOFs as the real human skeleton [Zeltzer82]. The main advantage of the stick figure model is that the motion specification isvery easy. It is only necessary to give for each joint a three-dimensionaltransformation matrix corresponding to its three DOFs. On the other hand, this kind ofrepresentation produces rather unrealistic simulations. Its lack of volume makes theperception of the depth diff icult and causes ambiguities in the pictures. Severalmovements are impossible to represent, li ke twists. An example of modeling andanimation of a stick figure with 32 segments and 15 joints (“Hipi” - from “DreamFlight” , 1982) is presented in Figure M2.14.

37

Figure M2.14: Hipi flying (from Dream Flight) [Magnenat 90]

3.1.2. Surface Models

Surface models are conceptually simple, containing a skeleton surrounded bysurfaces composed of planar or curved patches, simulating the skin. Several modelshave been developed since the sixties when Fetter [Fetter 82] introduced his first manmodel composed of only seven segments. A second, more fully articulated man wasintroduced in 1969, then the third man and in 1981, the fourth man and woman, inwhich several levels of complexity of the model were available. Other models arebased on anthropometric data and presented by Dooley [Dooley 82].

The concept of Joint-dependent Local Deformation (JLD) operators has beenintroduced by Magnenat-Thalmann and Thalmann [Magnenat 87] and is based onspecific local deformation operators depending on the joint nature. These JLDoperators control the evolution of surfaces and may be considered as operators onthese surfaces. Each JLD operator will be applied to some uniquely defined part of thesurface, which may be called the domain of the operator. The value of the operatoritself will be determined as a function of the angular values of the specific set of jointsthat defines the operator. Figure M2.15 shows an example of human charactergenerated with JLD operators.

Komatsu proposes a model with a skeleton structure represented by a stickfigure with 14 joints, and free-form surfaces covering the skeleton, li ke a skin[Komatsu 88]. Every part of this skin was defined by a set of biquartic Bézier surfacesand all the parts were smoothly combined. Adjusting the surface control pointsaccording to the joint angles, bending, twisting and swelli ng can be simulated.

38

Figure M2.15: The vir tual Mar ilyn, generated with JLD operators[Magnenat 90].

3.1.3. Volume Models

The volume models approximate the structure and the shape of the body witha collection of primitive volumes, such as cylinders [Evans 76, Potter 75], elli psoids[Herbison-Evans 74], or spheres [Badler 78]. A few cylinders or elli psoids cancapture the surface and longitudinal axis properties of many body parts, although thesimilarities are quite stylized, li ke cartoons. Those models were developed at the earlyof human animation when graphic systems had very limited capabiliti es. They cannotproduce better results than surface models, and suffer from inadequate controlmechanism of a large set of primitives during the animation.

Yoshimoto [Yoshimoto 92] presented a recent and interesting example ofhuman representation by the use of volume primitives. He proposes the use ofmetaballs, consider the volume as a potential function and produce good and naturalresults to create very realistic ballerinas. About 500 metaballs and some elli psoidswere used to design the ballerina’s body and costume shown in Figure M2.16. By theway, the model is static.

Figure M2.16: Ballerina made of metaballs [Yoshimoto 92]

39

3.1.4. Multi-Layered Models

Recently, multi -layered models have been developed to represent humanfigures. In this approach, normally a skeleton is used to support intermediate layersthat simulate the body volume (bones, muscles, fat tissues and so on) and the skinlayer. Sometimes a clothing layer can be also considered. A layer can be defined as aconceptual simulation model, which maps higher level input parameters into lowerlevel outputs. The animator specifies various constraint relationships between thelayers and can control the global motion from a high level.

A good example of layered construction and animation of deformablecharacters are presented by Chadwick et al. [Chadwick 89]. In their approach, thegeometric skin is not simply fitted to an articulated skeleton. This captures the fluidsquash and stretch behavior of the surface geometry by providing volumetric musclebulging, dynamic fatty tissue response, and creasing at the joint. A four layeredmodel, from high to low levels, was designed: the motion specification, referred to asthe behavior layer in the system; the motion foundation, represented by an articulatedskeleton layer; the shape transition, squash and stretch, corresponding to the musclesand fatty tissues layer; and the surface description, surface appearance and geometry,corresponding to the skin and clothing layer.

The skeleton layer provides the foundation for controlli ng the motion of thecharacter and is composed of a tree structure hierarchy of robotics manipulators,robotics joint-link parameters, joint angle constraints, and physical attributes. Themuscle layer is added on top of and attached to the skeleton hierarchy. The foundationfor the muscle and fat are based on Free Form Deformations (FFDs) [Sederberg 86].The control points of the deformations are constrained by the positioning (jointangles) and forces (joint torques) applied to and by the underlying skeleton. Then,these deformations act to glue and deform the geometric skin to the underlyingskeleton. The skin layer represents the actual visible surface geometry to be rendered.Figure M2.17 shows an example of human character generated using this layeredapproach.

Figure M2.17: Bragger Bones, a character designed by the Crittersystem [Chadwick 89]

40

The Body Builder system developed by Shen [Shen 95, 96], consists of ahighly effective multi -layered approach for design and animation of realistic humanbodies. This layered model is based on three levels, similar to those proposed byChadwick et al. [Chadwick 89]. The first layer is composed of an articulated skeletonhierarchy with only articulated line segments whose movements are pre-specified. Itmay be a dynamic skeleton governed by rigid body, physical simulations, or ageometric one, animated by key-framing. The second layer is composed of volumeprimitives designed to simulate the gross behavior of bones, fat and muscles. Theseprimitives fall i nto two categories: blendable volume, which will blend with otherblendable volumes in the same group; unblendable volume, which will not blend withother primitives. The volume primitives, arranged in an anatomically basedapproximation, are attached to the proximal joints of the skeleton.

The third layer is the skin surface of the body, which is automatically derivedfrom the position and shape of the first and second layers. An implicitl y surfacedefined by volume primitives is sampled with ray casting on semi-regular cylindricalgrids. These sample points are used directly as cubic B-spline control points tosmooth out the skin surface. Individual B-spline patches are triangulated, and thesetriangular meshes are stitched together to connect different parts of the human bodyfor final rendering and output. Simple, eff icient algorithms, and realistic, robustdeformations characterize the Body Builder system. The system provides an intuitivetool for animators to eff iciently design a rich variety of human shapes. Figure M2.18shows the various stages in the construction of a layered human body model.

Figure M2.18: Stages in the construction of the human body model[Shen 96].

Scheepers et al. [Scheepers 97] presented an approach to human figuremodeling similar to the one taken in artistic anatomy - by analyzing the relationshipbetween exterior form and the underlying structures responsible for creating it,surface form and shape change may be understood and best represented. Theyconsidered the influence of the musculature on surface form and developed musclemodels, which react automatically to changes in the posture of an underlyingarticulated skeleton. The first layer of the Scheepers’ model is the skeleton. They havedeveloped a procedural model for skeletal support, implemented in AL [May 96], aprocedural modeling and animation language with faciliti es for defining andmanipulating articulated models.

41

Muscles are the second layer and were the first goal of this work. Elli psoidswere used to represent muscle belli es because they consist of natural and convenientprimitives that can be scaled along their three major axes to simulate bulging. Manyskeletal muscles are fusiform and act in straight lines between their points ofattachment. For these muscles, a simple model with relatively few parameters, calledthe fusiform muscle model was used. To simulate the shape of complex muscles thatcannot be modeled with the straight fusiform muscles, a generative approach in whichany number of muscle belli es may be positioned automatically was used, the so-calledmulti -belly muscle model. Implicit versions of the simple geometric modelingprimitives are used to adjust the control points of bicubic patch meshes representingthe skin, the third layer of this model. This technique also allows the modeling of fattytissue between the muscles and the skin - adjusting the radius of influence of theimplicit functions allows different thickness of fatty tissue deposits to be modeled.Figure M2.19 shows an example of muscle reconstruction of the upper arm and torso.

Figure M2.19: Example of Scheepers’ anatomy-based human model[Scheepers 97].

The work developed by Wilhelms and Van Gelder [Wilhelms 97] describes anew modeling and animation approach for animals and humans that is based on actualthree-dimensional representations of individual body components. The body consistsof four types of materials: bones are rigidly embedded in segments; muscles areattached to bones; generalized tissue gives shape to regions where detailed bones andmuscles are not used, and for features such as eyes and nails; an elastic overlyingtriangle-mesh skin is attached to underlying tissues with anchors, but adjusts inresponse to forces from neighboring skin vertices.

The skeleton and generalized tissues were modeled as triangle meshes orelli psoids. To model each muscle, a model based on a deformed cylinder wasdeveloped. Therefore, each muscle is a discretized, deformed cylinder whose axis is acurve that proceeds from the midpoint of the muscle origins to the midpoint of themuscle insertions. The skin is an elastic triangle-mesh surface that is attached tounderlying components but can move relative to them; i.e., a separate, controllablyloose layer over underlying components. Animation involves repetition of thefollowing steps: motion specification at joints; reposition and deformation of theunderlying components accordingly to new positions; mapping of the skin verticesback to world space; application of the iterative relaxation algorithm, adjusting skinvertices to achieve the equili brium of the elastic membrane forces. Figure M2.20

42

shows a monkey model with 85 body segments, 156 bones, 52 muscles, 54generalized components and a skin surface composed of about 150,000 triangles.

Figure M2.20: Anatomical components of a monkey in rest posture:skeleton (white), muscles (red), generalized tissue (purple), and skin(lower right) [Wilhelms 97].

Other multi -layered models were also developed but there are not stressed herebecause they do not present a direct application on humans modeling. However, theirconcepts can be used to model humanoid characters. Gascuel et al. [Gascuel 91], forexample, propose a general model using a rigid skeleton layer, deformable cylindersassociated with each link of the skeleton to model elastic flesh; and a third layer tomodel the propagation of deformations from one flesh primitive to the others. Ageometric skin represented by a B-spline surface covers the deformable elements.Terzopoulos and Waters to implement facial animation [Terzopoulos 91] presented asophisticated example of layered model. In this model, an elastic solid simulationconsisting of a mass-spring lattice of depth three, is attached to a human skull modeland deformed by muscles which take the form of force constraints between points onthe skin surface and the underlying bone. The LEMAN system developed by Turner[Turner 93, 95] is structured into four layers: skeleton layer, muscle layer, fat andconnective tissue layer, and skin layer. The skin surface was implemented as asimulation of a continuous elastic surface discretized using a finite differencetechnique [Terzopoulos 87].

4. Case Study: Vir tual Cadaver (The Visible HumanProject)

The Center for Human Simulation (CHS) at the University of Colorado HealthSciences Center, is a synthesis of human anatomy and computed three-dimensionalimaging [CHS 99]. This synthesis resulted in a three-dimensional, high resolutiondatabase of human male and female anatomy (the Visible Human) as derived fromdirect analysis of anatomical specimens and radiological imaging. The generalpurpose of this Center is to facilit ate the collaboration of anatomists, radiologists,computer scientists, engineers, physicians and educators to promote the application of

43

this and other anatomical data to basic and clinical research, clinical practice andteaching.

Overall , the Center's advances will provide health care professionalseffectively with a thorough understanding of human form, function and development.Furthermore, these advances will enable them to electronically model normal,pathologic and procedural changes in, or manipulations of, human anatomy.Therefore, the major goal of the Center is to develop simulators that provideinteractions with computerized anatomy in virtual space. The Visible Human databasedefines the anatomy of the adult human body in three dimensions and at highresolution. Radiological studies will fuel the extension of these forms to models withcontrollable development, normal functions and pathology.

Anesthesiology simulators, commercially available today, will be integratedwith the complete virtual operating rooms of the Center. Body movements and fluiddynamics will add external and internal kinetics to the computerized anatomy of theVisible Human. Development of these functional models will require the interactionand collaboration of a broad range of experts, including mathematicians, computerscientists, anatomists, bioengineers, physicians, pathologists, anthropologists andothers. Eventually, the application, distribution and management of these models willnecessitate the involvement of medical information specialists, educators and libraryscientists.

As models are developed and databases extended, they will be used ineducation at all l evels, but especially for health care professionals. In the HealthSciences Center environment, this will i nclude not only courses in human anatomyand physiology for students of medicine, dentistry, physical therapy, nursing, CHAPSand dental hygiene but also training in radiology and in surgical disciplines.Moreover, trained health care professionals will be able to develop, teach and practicediagnostic and therapeutic procedures in virtual reality, on electronic normalizedpatients. The teaching tools will be available for all l evels of education and areanticipated to accompany the health care professional throughout their chosen career.

As the Center develops, it will become increasingly active in graduate andpostdoctoral education of young scientists. The goal of this research-training programis to educate the anatomist of the 21st century, a new type of anatomist/computerimaging specialist. In order to achieve this goal, a training program will be developedthat spans the range from anatomy to radiological imaging, computer science, andengineering. This will be an interdisciplinary effort, and will i nvolve faculty from allCU campuses.

The current knowledge explosion in biomedical science necessitates greatereff iciency in training, data search and testing for all health care professionals, butparticularly for physicians. This applies especially to anatomy, which is diff icult tovisualize in three dimensions but provides much of the fundamental vocabulary of thehealth care professional. Increasingly sophisticated presentations of anatomy areimportant as reference standards in conjunction with the development of more andmore abstract radiological imaging techniques. Increasingly complex clinicalprocedures, and the need to teach them, make the development of virtual-realitymodels highly desirable. Analogous to flight simulation, this construct allows thestudent to participate often and repeatedly in rare and emergency procedures. Thegoals and cooperative efforts of the Center for Human Simulation respond directly tothese needs of the Medical School and the missions of the University of ColoradoHealth Sciences Center.

44

The Center for Human Simulation will be based firmly on human anatomy andphysiology, just as medical and dental students – and indeed most health careprofessionals -- begin their education with a foundation in anatomy. As anatomicalmodels are developed they will serve the teaching mission of the Health SciencesCenter, initially in courses based on human anatomy. The first use of Visible Humanmaterial was in the Dental Gross Anatomy course for first-year CU dentistry studentsin the fall of 1994. The Visible Human Male has been used in our Dental SchoolGross Anatomy class since 1995. With the establishment in 1996 of a computerlaboratory, use of the Visible Human has been improved and expanded; it wasintroduced to the Medical Gross Anatomy Course in 1996. Subsequently, theelectronic teaching materials will become available to anatomy courses for otherhealth care professionals and to clinical courses. When the appropriate tools areavailable they will be used also for post-professional training and Continuing MedicalEducation.

In the research arena, collaborations of the anatomical imaging laboratory withmembers of Radiology, Rheumatology, Otolaryngology, Obstetrics and Gynecologyand Anesthesiology already are in progress. For example the presentation, withneuroradiology, on the three-dimensional nature of the amygdala and lateral horns ofthe left ventricle won the Summa Cum Laude Award for scientific exhibits at the 31st

Annual Scientific Meeting of the American Society of Neuroradiology.Collaborations with Radiologists and Anthropologists at the Museum of NaturalHistory resulted in presentations at the 1994 International Conference onPaleontology in Denver. A presentation was made at the 1995 Society ofOtolaryngologists meeting on the structure and function of the muscles at the origin ofthe Eustachian tube. In collaboration with the Department of Anesthesiology "A NewInteractive Computer System to Learn the Celiac Plexus Block" won first prize forscientific exhibits at the American Society of Anesthesiologists Annual Meeting inAtlanta (October, 1995). It also took the first prize at another meeting of the samegroup in New York (November, 1995). This system based on the Visible Human,enables the trainee to practice correct needle placement in virtual reality. The VisibleHuman Project video produced and edited with the National Center for AtmosphericResearch for SIGGRAPH 1995 won first place honors at the NICCOGRAPHcomputer graphics conference, Tokyo, Japan, 1995.

The Visible Human Male already has become an accepted world standard ofcomputer-based anatomy, and the Female, generated at higher resolution, representsan additional step forward. The Center for Human Simulation has experience intraditional polygon rendering of anatomical structures, and has developed uniqueways of handling the three-dimensional data of the Visible Human to generatephotorealistic surgical simulations never seen before. Figures M2.21 through M2.24shows some examples of visual models generated from the visible human dataset.

45

Figure M2.21: 3-D color models of the abdomen and foot.

Figure M2.22: Frontal and sagital view of the visible human, showingthe skin and muscles layers.

46

Figure M2.23: Two different representations of the visible human,highlighting some organs.

47

Figure M2.24: The visible human skeleton with the left upper armmuscles schematically represented.

5. References

[Anjyo 92] Anjyo, K., Usami, Y. and Kurihara, T. “A Simple Method forExtracting the Natural Beauty of Hair” , Proceedings ofSIGGRAPH’92, Computer Graphics, v. 26, n. 2, pp. 111-120,1992.

[Aono 90] Aono, M. “A Wrinkle Propagation Model for Cloth” , Proceedingsof Computer Graphics International, Springer-Verlag, pp. 96-115,1990.

48

[Armstrong 86] Armstrong, W., Green, M. and Lake, M. “Near real time control ofhuman figure models, Graphics Interface’86, 1986.

[Badler 78] Badler, N., O’Rourke, J. and Toltzis, H. “A human body modelli ngsystem for motion studies” , Movement Project Rep., no 13,Department of Computer and Information Science, University ofPennsylvania, July 1978.

[Badler 79] Badler, N., O’Rourke, J. and Toltzis, H. “A sphericalrepresentation of a human body for visualizing movement” ,Proceedings of IEEE, v. 67, n. 10, October 1979.

[Badouel 90] Badouel, D. “An Eff icient Ray-polygon Intersection” . In: A. S.Glassner (Ed.). Graphics Gems. New York: Academic Press, 1990.p. 390-396.

[Baraff 92] Baraff , D. and Witkin, A. “Dynamic Simulation of Non-Penetrating Flexible Bodies” , Computer Graphics, v. 26, n. 2, pp.303-308, 1992.

[Bechmann 94] Bechmann, D. “Space Deformation Models Survey” , Computer &Graphics, v. 18, n. 4, p. 571-586, 1994.

[Boulic 95] Boulic, R., Çapin, T., Huang, Z., Kalra, P., Lintermann, B.,Magnenat-Thalmann, N., Moccozet, L., Molet, T., Pandzic, I.,Saar, K., Schmitt, A., Shen, J. and Thalmann, D. “TheHUMANOID Environment for Interactive Animation of MultipleDeformable Human Characters” , EUROGRAPHICS’95, 1995.

[Breen 94] Breen, D. E., House, D. H. and Wozny, M. J. “Predicting theDrape of Woven cloth using Interacting Particles” , Proceedings ofSIGGRAPH’94, Computer Graphics, v. 28, n. 4, pp. 365-372,1994.

[Cachola 86] Cachola, D. and Shrack, G. “Modeling and animating 3-Darticulated figures” , Graphics Interface’86, 1986.

[Carignan 92] Carignan, M., Yang, Y., Magnenat-Thalmann, N. and Thalmann,D. “Dressing Animated Synthetic Actors with ComplexDeformable Clothes” , Proceedings of SIGGRAPH´92, ComputerGraphics, v. 26, n. 2, pp. 99-104, 1992.

[Chadwick 89] Chadwick, J. E., Haumann, D. R. and Parent, R. E. “LayeredConstruction for Deformable Animated Characters” , ComputerGraphics, v. 23, n. 3, p. 243-252, July 1989.

[CHS 99] The Center for Human Simulation (CHS) at the University ofColorado Health Sciences Center, WEB site athttp://www.uchsc.edu/sm/chs/.

[Delingette 93] Delingette, H., Watanabe, Y., Suenaga, Y. “Simplex BasedAnimation” , Proceedings of Computer Animation’93, Eds. N.Magnenat-Thalmann and D. Thalmann, Springer Verlag, pp. 13-28, 1993.

[Denby 76] Denby, E. F. “The Deformation of Fabrics during Wrinkling – ATheoretical Approach” , Textile Research Journal” , Landcaster PA,v. 46, pp. 667-670, 1976.

49

[Dooley 82] Dooley, M. “Anthropometric Modeling Programs - A Survey” ,IEEE Computer Graphics and Applications, v. 2, n. 9, p. 17-25,November, 1982.

[Ekman 78] Ekman, P. and Friesen, W. Facial action coding system,Consulting Psychologist Press, Palo Alto, CA, 1978.

[Evans 76] Evans, S. M. “User’s guide for the programs of combiman”,Report AMRLTR-76-117, University of Dayton, Ohio, 1976.

[Farin 90] Farin, G. “Surfaces Over Dirichlet Tesselations” , Computer AidedDesign, v. 7, pp. 281-292, 1990.

[Fetter 82] Fetter, W. A. “A Progression of Human Figures Simulated byComputer Graphics” , IEEE Computer Graphics and Applications,v. 2, n. 9, p. 9-13, November 1982.

[Fuchs 77] Fuchs, H.; Kedem, Z. M.; Uselton, S. P. “Optimal SurfaceReconstruction from Planar Contours” . Communications of theACM, New York, v. 20, n. 10, p. 693-702, Oct. 1977.

[Gascuel 91] Gascuel, M.-P., Verroust, A. and Puech, C. “A Modelli ng Systemfor Complex Deformable Bodies Suited to Animation andColli sion Processing” , The Journal of Visualization and ComputerAnimation, v. 2, n. 3, p. 82-91, 1991.

[Glassner 84] Glassner, A. S. “Space Subdivision for Fast Ray Tracing” . IEEEComputer Graphics & Applications, Los Alamitos, CA, v. 4, n. 10,p. 15-22, 1984.

[Gourret 89] Gourret, J.-P., Magnenat-Thalmann, N. and Thalmann, D.“Simulation of Object Grasping and Human Skin Deformations ina Grasping Task” , Proceedings of SIGGRAPH’89, pp. 21-30,1989.

[Greenleaf 70] Greenleaf, J. F., Tu, T. S., Wood, E. H. “Computer Generated 3DOscill oscopic Images and Associated Techniques for Display andStudy of the Spatial Distribution of Pulmonary Blood Flow”. IEEETrans. Nucl. Sci., New York, v. NS-17, p. 353-359, 1970.

[Haley 96] Haley, Michael B.; BLAKE, Edwin H. “ Incremental VolumeRendering Using Hierarchical Compression” . Computer GraphicsForum, Cambridge, v. 15, n. 3, Aug. 1996. Trabalho apresentadono EUROGRAPHICS, 1996, Futuroscope-Poitiers, France.

[Herbison-Evans 74] Herbison-Evans, D., “Animated cartoons by computers usingelli psoids” , In: Proceedings of 6th Australian ComputerConference, p. 811-823, 1974.

[Herbison-Evans 78] Herbison-Evans, D. “NUDES 2 – a numeric utilit y displayingelli psoid solids” , Version 2, Proceedings of SIGGRAPH’78, 1978.

[Hsu 92] Hsu W., Hugues J. F., Kaufman H., Direct Manipulation of Free-Form Deformations, Proceedings Siggraph'92, pp. 177-184, 1992.

[Jones 97] Jones, Mark W. “Acceleration Techniques for VolumeRendering” . In: EARNSHAW, Rae; VINCE, John; JONES, Huw(Eds.). Visualization & Modeling. London, UK: Academic Press,1997. p. 253-286.

50

[Kajiya 89] Kajiya, J. T. and Kay, T. L. “Rendering Fur with ThreeDimensional Textures” , Proceedings of SIGGRAPH’89, ComputerGraphics, v. 23, n. 3, pp. 271-280, 1989.

[Kalra 91] Kalra, P., Mangili , A., Magnenat-Thalmann, N. and Thalmann, D.“SMILE: A Multil ayered Facial Animation System”, Proceedingsof IFIP WG 5.10, Tokyo, Japan, pp. 189-198, 1991.

[Kalra 92] Kalra, P., Mangili , A., Magnenat-Thalmann, N. and Thalmann, D.“Simulation of Muscle Actions using Rational Free FormDeformations” , proceedings of Eurographics’92, ComputerGraphics Forum, v. 2, n. 3, pp. 59-69, 1992.

[Kalra 93] Kalra, P. An Interactive Multimodal Facial Animation System,Ph.D Thesis, Swiss Federal Institute of Technology, Lausanne,1993.

[Kaufman 91] Kaufman, A. “Volume Visualization” . Los Alamitos, CA, IEEEComputer Society Press, 1991, 479 p.

[Kaufman 90] Kaufman, A.; Yagel, R.; Cohen, D. “ Intermixing Surface andVolume Rendering” . In: HOEHNE, K. H.; FUCHS, H.; PIZER, S.M. (Eds.). 3D Imaging in Medicine, Algorithms, Systems,Applications. Springer-Verlag, 1990.

[Keppel 75] Keppel, E. “Approximating Complex Surfaces by Triangulation ofContour Lines” . IBM Journal of Research and Development, NewYork, v. 19, n. 1, p. 2-11, Jan. 1975.

[Knittel 95] Knittel, Günter. “High-Speed Volume Rendering Using RedundantBlock Compression” . In: VISUALIZATION, 1995, Atlanta, GA.Proceedings... Los Alamitos, CA: IEEE Computer Society Press,1995. p. 176-183.

[Komatsu 88] Komatsu, K. “Human Skin Model Capable of Natural ShapeVariation” , The Visual Computer, v. 3, n. 5, p. 265-271, 1988.

[Kunii 90] Kunii , T. L. and Gotoda, H. “Modeling and Animation of GarmentWrinkle Formation Processes” , Proceedings of ComputerAnimation’90, Springer-Verlag, pp. 131-146, 1990.

[Kurihara 93] Kurihara, T., Anjyo, K. and Thalmann, D. “Hair Animation withColli sion Detection” , Proceedings of Computer Animation’93, pp.128-138, 1993.

[Lafleur 91] Lafleur, B., Magnenat-Thalmann, N. and Thalmann, D. “ClothAnimation with Self-Colli sion Detection” , Proceedings of the IFIPconference o nModeling in Computer Graphics, pp. 179-187,1991.

[Lamousin 94] Lamousin, H. J., Waggenspack, W. N. “Nurbs-based Free-FormDeformations” , IEEE Computer Graphics and Applications, v. 14,n. 16, pp. 59-65, November 1994.

[Levoy 88] Levoy, Mark. “Volume Rendering - Display of Surfaces fromVolume Data”. IEEE Computer Graphics & Applications, LosAlamitos, CA, v. 8, n. 3, p. 29-37, May 1988.

51

[Levoy 90] Levoy, Mark. “Eff icient Ray-Tracing of Volume Data”. ACMTransactions on Graphics, New York, v. 9, n. 3, p. 245-261, July1990.

[Levoy 90a] Levoy, Mark. “A Hybrid Ray Tracer for Rendering Polygons andVolume Data”. IEEE Computer Graphics & Applications, LosAlamitos, CA, v. 10, n. 3, p. 33-40, 1990.

[Lichtenbelt] Lichtenbelt, B.; Crane, R.; Naqvi, S. “ Introduction to VolumeRendering” . Hewlett-Packard Professional Books, Prentice Hall ,1998.

[Lorensen 87] Lorensen, Willi am E.; Cline, Harvey E. “Marching Cubes: A HighResolution 3D Surface Construction Algorithm”. ComputerGraphics, New York, v. 21, n. 4, July 1987.

[Magnenat 85] Magnenat-Thalmann, N. and Thalmann, D. “Computer animation:Theory and practice, Computer Science Workbench, Springer-Verlag, Berlin Heidelberg New York Tokyo, 1985.

[Magnenat 87] Magnenat-Thalmann, N. and Thalmann, D. “The Direction ofSynthetic Actors in the Film Rendez-vous à Montréal” , IEEEComputer Graphics and Applications, v. 7, n. 12, p. 9-19, 1987.

[Magnenat 88] Magnenat-Thalmann, N., Laperrière, R. and Thalmann, D. “Joint-Dependent Local Deformations for Hand Animation and ObjectGrasping” , Proceedings of Graphics Interace’88, Canadian Man-Computer Communication Society, pp. 26-33, 1988.

[Magnenat 90] Magnenat-Thalmann, N. and Thalmann, D. Computer Animation:Theory and Practice - Second Revised Edition, Computer ScienceWorkbench, Springer-Verlag, Berlin, 1990.

[Magnenat 93] Magnenat-Thalmann, N. and Thalmann, D. “The Artificial Life ofSynthetic Actors” , 1993.

[Magnenat 96] Magnenat-Thalmann, N., Carion, S., Courchesne, M., Volino, P.and Wu, Y. “Virtual Clothes, Hair and Skin for Beautiful TopModels” , Proceedings of Computer Graphics International’96, pp.132-141, 1996.

[May 96] May, S. F., Carlson, W., Philli ps, F. and Scheepers, F. “AL: Alanguage for procedural modeling and animation” , TechnicalReport OSU-ACCAD-12/96-TR5, ACCAD, The Ohio StateUniversity, December 1996.

[Mill er 88] Mill er, G. S. P. “From Wire-Frame to Furry Animals” ,Proceedings of Graphics Interface’88, pp. 138-146, 1988.

[Moccozet 97] Moccozet, L. and Magnenat-Thalmann, N. “Dirichlet Free-FormDeformations and their Application to Hand Simulation” ,Proceedings of Computer Animation’97, pp. 93-102, 1997.

[Montgomery 99] Montgomery, K. “3D Reconstruction” . Available byhttp://biocomp.stanford.edu/3dreconstruction/index.html (Sep.1999).

[Olabarriaga 89] Olabarriaga, Silvia D.; BRASIL, A. “RETRISE - Um Sistemapara Reconstrução Tridimensional a partir de Cortes Seriais” . In:

52

SIMPÓSIO BRASILEIRO DE COMPUTAÇÃO GRÁFICA EPROCESSAMENTO DE IMAGENS, 2., 1989, Águas de Lindóia,SP. Anais... São Paulo: SBC, 1989. p. 93-102.

[Parke 82] Parke, F. “Parametric models for facial animation” , IEEEcomputer Graphics & Applications, November 1982.

[Perlin 89] Perlin, K. H. “Hypertexture” , Proceedings of SIGGRAPH’89,Computer Graphics, v. 23, n. 3, pp. 253-262, 1989.

[Platt 81] Platt, S. and Badler, N. “Animating facial expressions” , ComputerGraphics, v. 15, n. 3, August 1981.

[Potter 75] Potter, T. E., Willmert, K. D., “Three-dimensional human displaymodel” , Computer Graphics, v. 9, n. 1, p. 102-110, 1975.

[Rosenblum 91] Rosenblum, R. E., Carlson, W. E. and Tripp III, E. “Simulating theStructure and Dynamics of Human Hair: Modeling, rendering andAnimation” , Visualization and Computer Animation, v. 2, n. 4, pp.141-148, 1991.

[Rowlan 94] Rowlan, J. S.; Lent, E.; Gokhale, N. et al. “A Distributed, Parallel,Interactive Volume Rendering Package”. In: VISUALIZATION,1994, Washington, DC. Proceedings... Los Alamitos, CA: IEEEComputer Society Press, 1994. p. 21-30.

[Scheepers 97] Scheepers, F., Parent, R.E., Carlson, W.E. and May, S.F.“Anatomy-Based Modeling of the Human Musculature” , ” , In:Computer Graphics Proceedings, p. 163-172, 1997.

[Sederberg 86] Sederberg, T. W. and Parry, S. R. “Free-Form Deformation ofSolid Geometric Models” , Computer Graphics, v. 20, n. 4, p. 151-160, August 1986.

[Shen 95] Shen, J. and Thalmann, D. “ Interactive Shape Design UsingMetaballs and Splines” , In: Proceedings of EurographicsWorkshop on Implicit Surfaces’95, Grenoble, France, p. 187-196,1995.

[Shen 96] Shen, J. “Human Body Modeling and Deformations” , PhD thesis,École Polytechnique Fédérale de Lausanne, 1996. Advisor: Prof.Daniel Thalmann.

[Silva 96] Silva, Marcelo R. M. “Visualização Direta de Dados VolumétricosUsando a Técnica de Ray-Casting” . Porto Alegre: Instituto deInformática/UFRGS, 1996. Trabalho de Diplomação.

[Silva 97] Silva, M. R. M. “Ferramenta para Exploração de DadosVolumétricos” . Porto Alegre: Instituto de Informática/UFRGS,Dez. 1997. (TI 721).

[Steketee 85] Steketee, S. and Badler, N. “Parametric keyframe interpolationincorporating kinetic adjustment and phrasing control” ,Proceedings of SIGGRAPH’85, v. 19, n. 3, 1985.

[Terzopoulos 87] Terzopoulos, D., Platt, J., Barr, A. and Fleischer, K. “ElasticallyDeformable Models” , Computer Graphics, v. 21, n. 4, p. 205-214,July 1987.

53

[Terzopoulos 91] Terzopoulos, D. and Waters, K. "Techniques for Realistic FacialModeling and Animation", In: Computer Animation'91, edited byMagnenat-Thalmann, N. and Thalmann, D., Springer-Verlag,Tokyo, p. 59-74, 1991.

[Tost 88] Tost, D. and Pueyo, X. “Human body animation: a survey” , theVisual Computer, v. 3, pp. 254-264, 1988.

[Turner 93] Turner, R. “ Interactive Construction and Animation of LayeredElastic Characters” , PhD Thesis, École Polytechnique Fédérale deLausanne, Switzerland, 1993. Advisor: Daniel Thalmann

[Turner 95] Turner, R. “Constructing and Animating Layered ElasticCharacters” , In: Proceedings of Computer GraphicsInternational’95, p. 185-203, 1995.

[Uda 93] Uda, N., Kimura, F., Tsuruoka, S., Miyake, Y., Shunlin, S., Tsuda,M. “Expansion of the Solid Modeler’s Functions for Use inDesigning a Model of the Human Body” , New Advances inComputer Aided Design & Computer Graphics, InternationalAcademic Publishers, pp. 357-363, 1993.

[Upson 88] Upson, C.; Keeler, M. “V-Buffer: Visible Volume Rendering” .Computer Graphics, New York, v. 22, n. 4, p. 59-64, Aug. 1988.

[Watanabe 89] Watanabe, Y. and Suenaga, Y. “Drawing Human Hair Using WispModel” , Proceedings of Computer Graphics International’89, pp.691-700, 1989.

[Weil 86] Weil , J. “The synthesis of Cloth Objects” , Proceedings ofSIGGRAPH’86, Computer Graphics, v. 4, pp. 49-54, 1986.

[Westover 89] Westover, L. “ Interactive Volume Rendering” . In: WORKSHOPON VOLUME VISUALIZATION, 1989, Chapel Hill , N.C.Proceedings... University of North Carolina Press, 1989. p. 9-16.

[Whitted 80] Whitted, T. “An Improved Illumination Model for ShadedDisplays” . Communications of the ACM, New York, v. 23, n. 6, p.342-349, 1980.

[Wilhelms 97] Wilhelms, J. and Van Gelder, A. “Anatomically Based Modeling” ,In: Computer Graphics Proceedings, p. 173-180, 1997.

[Witkin 90] Witkin, A. and Welch, W. “Fast Animation and Control of Non-Rigid Structures” , Proceedings of SIGGRAPH’90, ComputerGraphics, v. 24, pp. 243-252, 1990.

[Yang 92] Yang, Y., Magnenat-Thalmann, N. and Thalmann, D. “3DGarment Design and Animation – A New Design Tool for theGarment Industry” , computer in Industry, v. 19, pp. 185-191,1992.

[Yoshimoto 92] Yoshimoto, S. “Ballerinas Generated by a Personal Computer” ,The Journal of Visualization and Computer Animation, v. 3, p. 85-90, April 1992.

[Zeltzer 82] Zeltzer, D. “Motor control techniques for figure animation” , IEEEComputer Graphics and Applications, November, 1982.

54

[Zuiderveld 96] Zuiderveld, K. J.; Koning, A. H. J.; Stokking, Rik et al.“Multimodali ty Visualization of Medical Volume Data” .Computers&Graphics, Oxford, v. 20, n. 6, p. 775-791, 1996.

55

MODULE 3:Simulating Vir tual Human Bodies

1. Introduction

Considering the complexity of the human body, we can say that the realism ofmotion needs to be improved not only from the joint point-of-view, but also inrelation to the body deformation. This module proposes a method to simulate humanbeings based on anatomy concepts, because we believe that the closer a model is toreality, the better will be the results. Using this approach we describe a humanrepresentation method divided into three layers: the rigid body conception from a realskeleton, the muscle design and deformation and the skin generation. Someintegration aspects are also presented and discussed. To conclude, we briefly describea method to model and animate faces, a hands deformation method and a model torepresent hair. The Section 2 of this module is presented in detailed in [Nedel 98].

2. Modeling Anatomically-Based Human Bodies

2.1. Skeleton Model

The human body can be briefly defined as a composition of skeleton, muscles,fat and skin. The skeleton is formed by bones (about 206 in total) attached with eachother by joints, and constitutes the foundation of all human surface form. The base ofa skeleton is the spinal column where the two pairs of limbs (arms and legs) and thehead are connected. We also use the term skeleton in computer animation, todesignate a stick figure representing the positions and orientations of the joints, whichmake up the articulated figure.

To represent our stick made human model, we have used the basic skeletonstructure proposed by Boulic et al. [Boulic 91, 94] and briefly described in thissection. In order to generate a new model based on an anatomic approach, jointpositions and orientations were defined. Finally, we demonstrate the usefulness of thebones in our skeleton model.

2.1.1. Overview of the BODY Structure

Figure M3.1 shows the different components of the BODY data structure andindicates the dependencies between them with the arrows. The high level parametersinduce the size of the SKELETON which in turn induce some constraints over theVOLUME and ENVELOPE information.

56

Figure M3.1: BODY data structure showing the propagation ofdependencies [Boulic 94c].

A general mechanism allows the customization of the skeleton structure at twolevels, either at a high level with a small set of scaling parameters or at the low levelwith the position and orientation of the various articulations defined in theSKELETON data structure. In both cases the modifications are propagated to thelower level structure of the volume and envelope.

In some working context some optional part of the SKELETON may beswitched on or off to improve the real-time display performances. The hands are atypical example because they hold nearly as many degrees of freedom as the otherpart of the skeleton.

2.1.2. General Functionality

The BODY data structure allows the characterization of a human model froma template body and it provides entry points to the components of the corresponding3D hierarchy. The major purpose of a BODY data structure is to retain the basicinformation of the geometric characterization, the mass distribution and thedeformation entities for a general topology of a trunk connected to four limbs and ahead.

The information requested for this design is partitioned in two groups. First,the information necessary to define the topology, mobilit y, solids and deformationentities is frozen into the BODY framework. The user is still given a great flexibilit ywith the second group of parameters, which can produce a wide range of BODYindividuals at various levels of details.

57

2.1.2.1. BODY template and BODY instance

Two uses of a BODY structure have been identified for that goal (FigureM3.2): At the lower level a BODY template can be designed very precisely either

for the position and orientation of the joints and the SOLID volumes, andthe size of the solid volumes. A general mechanism automatically derivesthe length parameters of the SOLID from the skeleton structure. A BODY template can in turn serves as a reference for one (or more)BODY instance(s). The BODY instance can define a few high-levelnormalization parameters to scale the BODY template. The solids entitiesare also scaled accordingly.

Figure M3.2: A BODY instance needs a BODY template for high levelscaling [Boulic 94c].

In the general case of multiple BODY sharing a common environment, therewould typically be one BODY template serving as a family model to the BODYinstances, each of them holding a different set of normalization parameters.

Other differences between a BODY template and instance are the following: A BODY template maintains no 3D hierarchy, whenever one wishes tovisualize the low level design of a BODY template, (s)he can create aBODY instance making reference to it and displaying it. A BODY template maintains the number of its instances to postpone itsdestruction after the destruction of its last instance. A BODY template is always defined in the reference posture (all the jointshaving a zero value) while a BODY instance can be animated with amotion generator.

Changes in a BODY template are not automatically propagated to the BODYinstances for two reasons: it costs a lot and it may not be necessary. As aconsequence, the entry points of the BODY instances are not maintained by theBODY template. However, these changes are implicitl y reflected in the high-levelscaling operations allowed for BODY instances. A BODY instance just needs to re-

58

scale itself with its current set of normalization parameters to be consistent with itstemplate. This operation is called the fitting of the instance.

2.1.2.2. Normalization Parameters

A standardized topology and distribution of mobiliti es have been set with theaim of representing all the decisive human mobiliti es without overloading theresulting mechanical structure. Without the hands, the internal mobiliti es of the bodyhave a dimension of 62. Each hand adds 25 more mobiliti es. The detailed descriptionof the complete 3D hierarchy is given in Section 2.1.2.4.

The geometrical characterization of the standard mechanical structure is storedin a SKELETON data structure. The normalization parameters add a higher level forthe characterization of a human data structure. This aspect is provided in order toderive specific SKELETON instances from a standard one belonging to a templateBODY (Figure M3.3 and M3.4).

Figure M3.3: Scaling process from the template BODY [Boulic 94c]

Our first template was derived from the "marylin" skeleton model comingfrom initial developments made at the University of Montreal. However, a templatecan be constructed from anthropometric data to match a given population. Fiveparameters are used in a two-step process to scale the SKELETON of a BODY withrespect to its template BODY:

a) a uniform scaling is made along x, y, and z with the total_height.

b) specific scaling are made along the following independent directions: frontal: along y axis with frontal_scaling; lateral: high_lateral_scaling and low_lateral_scaling are proposed todifferentiate the higher body from the lower body ( useful for gendercharacterization); vertical: while keeping the same total height, the skeleton can be characterizedwith the parameter spine_origin_ratio which expresses the ratio of spine originheight over the total height.

59

The second type of normalization parameter is the total_mass, which isdistributed over a set of volumetric primitives approximating the body distribution (CfSOLID). This allows for the computation of the center of gravity and inertial matrices.

Figure M3.4: Var ious scaled BODY instances from a single BODYtemplate [Boulic 94c].

2.1.2.3. Options

A BODY H3D hierarchy contains at least the motion data structure includingthe skeleton. These node_3Ds are internal node_3D. The hands have to be explicitl ymentioned as part of the 3D hierarchy due to their inherent complexity (32 node_3Dfor each).

Two other families of terminal node_3D carrying more specific informationcan be derived from the template and defined in a BODY:� The volumetric primitives approximating human volume and weight

distribution: this is useful for approximation of the body volume forinterference checking and for the computation of inertial properties ofdynamics;� The polygonal surface approximating the human skin surface (withoutdeformation): this surface consists of separate surfaces approximatingsegments of the body (16 pieces are used considering the hands as awhole). The template can store different approximations of the samesurface piece.

All the node_3D can be directly accessed. Direct access to typed data asJOINT and FREE can be constructed on request.

2.1.2.4. Geometric Characterization

The skeleton topological structure appears on Figure M3.5 and M3.6. Eachnode_3D is identified by its unique name within its UNIT. Each JOINT node_3Dpossesses one degree of freedom, further denoted a dof, either translational orrotational. Some functional location node_3Ds are included by default in the basicskeleton or the hands. Only one default SOLID, the virtual_floor, is provided with theskeleton set. The location of the node_3d is shown on Figures M3.7 for the skeletonand M3.8 for the hand.

60

Figure M3.5: General topology of the skeleton without the hands

Figure M3.6: Hierarchy of the right hand [Boulic 94c]

61

Figure M3.7: 3D front view of the skeleton indicating the location ofthe node_3D [Boulic 94c].

Figure M3.8: 3D view of the skeleton of the right hand indicating thelocation of the node_3D in the rest position. (the integer values referto Figure M3.6) [Boulic 94c].

Let us now define our initial position also called the rest position. In thisposition, the body is standing up with dangling arms oriented such that the palmarsurface of hand lies in the sagital plane with medial orientation (Figure M3.9). In therest position, all the degrees of freedom have a zero value except the shoulder flexion

62

and elbow flexion (respectively -5 degrees and 5 degrees) in order to provide a morenatural posture.

Figure M3.9: The skeleton in the initial position (also called rest) [Boulic 94c]

The skeleton_root is the entry node_3D to the internal mobilit y of theskeleton. As such it is the default root of motion within the skeleton. As it is situatedat the base of the spine, it is quite close to the center of mass for large classes ofpostures. That property makes it a good root of motion for numerous movementsinvolving the whole displacement of the body (walking, jumping, etc.).

However, some movements are better designed with a different root of motion,e.g. hanging by the hand is easier designed when the root of motion is the hand center,or picking an object on the floor is better controlled when the root of motion is thesupporting foot. Changing the root of motion is possible without changing thetopology. However, one has to be aware that changing the root of motion during amovement, says for a transition between two types of movements, may introduce aperceptible motion discontinuity.

The naming convention of the mobiliti es dofs is:� tilt: forward-backward motion in the sagital plane along a lateral axis (xBODY);� roll: sideward balance motion in the coronal plane along a frontal axis (yBODY );� torsion: twisting along the vertical axis (z BODY).

2.1.3. Anatomic Template

As mentioned before, the main goal of this work is to produce a human modelbased on anatomic principles. At the skeleton level we suggest the creation of asimpli fied body with the joint positions defined anatomically. To accomplish it, wepropose the design of a new human template defining new position and orientation forthe joints. But finding the joints center location from the knowledge of externalmarkers is a very diff icult task. Joints li ke the shoulder or the hip for example, arespecially hidden within the flesh and complex bone structure (see Figure M3.10).

63

(a) (b) (c) (d)

Figure M3.10: Body silhouettes with skeleton in front view ((a)female, (b) male), back view ((c) male), and side view ((d) male)[Thomson 64].

We propose the template definition by analyzing a three-dimensional realskeleton. From a realistic reconstructed human skeleton, we have carefullydetermined the best position for each joint between the bones, based on anatomicconcepts and motion observation. In Figure M3.11 we can see the skeleton with bonylandmarks highlighted, representing the positions of the joints. Figure M3.12compares the new anatomic template with our old template, also designed at LIG.Essential differences can be observed on the spine and especially on the shoulderlevel. The foot complex was also completely redefined in our proposed skeleton.

Figure M3.11: Several views of the skeleton with the joint positionshighlighted.

64

(a) (b)

Figure M3.12: Compar ison between the two templates - the old one ison the left and the new one on the right side: (a) front view and (b)intermediate view.

Concerning the shoulder, the fundamental difference is seen in the claviclepositions. In the old template these are almost in the same plane than the shoulder andthe spine, while in the new one it is placed much more on the forward. The shoulderand the scapula were also changed. The new template has these two joints moredistant than before. On the arms, the concrete modification relates to the posture. Inthe new template the arms are a bit more open, to avoid the colli sion with the torso.On the spine, two greater changes can be observed: the new position of the joint vl1,nearest the hip joints; and the curvature of the spine, in general. On the foot, thebiggest change relates the ankle and the subtalar joints. Before, all the two joints hadthe same position while in the new template they had been separated, correspondingto two different articulations, as in reality.

A standard posture for which all the joint angles are null , was also defined andcan be seen in Figure M3.11. In this posture, the body is standing up with danglingarms oriented such that the palmar surface of hand lies in the sagital plane with medialorientation. Also concerning the joint angles, we have defined minimal and maximallimits for the movements, trying to avoid forbidden postures.

2.1.4. Why Representing Bones?

Using the joints based structure described before, we have created a realskeleton. Bones are linked to the joints reference systems and modeled by trianglemeshes defined locally in relation to their parent joint. Our system is modeled with 73bones, ignoring the hands and feet that are represented by complete sets of bones.Bones do not change shape during animation, but can change position in relation toeach other.

Our main goal in representing bones is to permit the muscles attachmentdefinition. As explained in the following section, skeleton muscles are fixed on thebones by their extremity points, in general. But also esthetic purposes were

65

considered. In fact, in some parts of the body, the skeleton contributes directly on theexternal appearance and should be considered during the skin generation process. Forexample, on the lower part of the legs, the elbow and some ribs, if we consider a slimperson. Furthermore, parts of bones that appear not to create surface form in somepostures do so in others.

Moreover, another reason pushed us to append the bone shapes in our model.The visualization of the skeleton during animation allows the avoidance of forbiddenpostures, a tedious and abstract task when only based on a stick figure. The definitionof the limit angles for each joint became also more precise when using bonesrepresentation between articulations. Figure M3.13 shows the skeleton while tryingsome interesting postures.

Figure M3.13: Skeleton performing different postures

66

2.2. Muscles Simulation

Among the anatomical systems that determine the skin shape, the musculatureis the most complex. Three kinds of muscles can be identified in the human body:skeletal, cardiac and smooth muscles. As our purposes involve the representation ofthe human body, more precisely the external appearance, we have modeled onlyskeletal muscles, referred in this work simply by muscles. Skeletal muscles arearranged side by side and in layers on top of bones and other muscles. Locatedthroughout the entire body, the skeletal muscles makes up from 40 to 45% of the totalbody weight [Tolo NY].

2.2.1. Action Lines

To mechanically quantify the force that a muscle produces over a bone, werepresent muscles by lines, regularly named action lines. The attachment of musclesto bones closer to the spine of the body is generally considered the origin while theother one represents the insertion. Normally, the origin is the fixed point, and theinsertion is where the muscle performs its action, but there is no formal rule about it.Most muscles are simply attached to two bones across a joint, and each bone mayfunction as either the fixed point or the moveable one [Cody 90].

In our approach to model the muscle forces, muscles can be represented byone or more action lines, basically defined by an origin and an insertion point. Thesepoints represent the links between the muscles and the bones, sometimes alsocorresponding to the extremities of the muscle shape. However, depending of theshape, position and complexity of the muscle, simple action lines are not enough torepresent the force produced over the skeleton. For this reason, we have decided torepresent the actions of the muscles not simply by a straight line, but by usingpolylines. To accomplish it we have developed the concept of control points, whoseobjective is to guide the line, avoiding the intersection with the bones. An example ofthis kind of action line is shown in Figure M3.14. The Biceps Brachii is representedby two action lines (one for each of the two muscle heads), but one of them needs touse a control point. If we try to represent this action line by a straight line, the bonewill be intercepted.

Other examples of intersection between bones and lines of action can beperceived only while animating the skeleton. It is exactly the case of the TricepsBrachii , for example, that are represented by three action lines (one for each musclebelly) attaching the upper arm to the upper part of the lower arm. Figure M3.15 showsthe definition of the action line that represents the Triceps middle head. In FigureM3.15.a, we can see the skeleton of the arm with the defined action line when themuscle is in its rest position. Figure M3.15.b shown the arm contraction and theresultant action line, if it was defined by a simple straight line. We can see in theexample, that if the action line is not correct, the muscle can contract during anextension action. Finally, Figure M3.15.c shows the correct simulation of the Tricepsaction line, by using some control points.

Moreover, Figure M3.15 shows also an example of the muscle shapesimulation and reaction to the arm motion. In these examples we can verify that themuscle belly is not attached to the bones by the origin and insertion of the action lines.It has its own origin and insertion points, simulating in this way the presence oftendons. In order to simulate the tendons we assume that the action line is, in fact, athread linking two bones and crossing some pulley, here represented by the controlpoints. Tendons stretch just about 8% of their original length. For the sake of

67

simplicity, we define the tendons by giving distances between the muscle boundariesand the action lines extremities. During the animation, these distances are keptconstant.

Figure M3.14: The Biceps Brachii muscle and corresponding action lines

(a) (b) (c)

Figure M3.15: The Triceps Brachii middle head action representation:(a) bones of the arm and Triceps action line with the muscle in its restposition; (b) action line and muscle reaction dur ing the arm

68

contraction and with a straight action line; (c) cor rect simulation ofthe action line by using additional control points.

2.2.2. Muscle Shape Design

To simulate muscle deformations we use a set of mass points linked by springsand organized in a way to ensure a correspondence between the action line and themuscle shape. We consider, for all the cases, that each muscle belly involvescompletely an action line or a part of an action line. Each point that composes thesurface of a muscle is arranged between two neighbors in the horizontal and twoothers in the vertical direction, considering the action line of a muscle as the upreference. In this sense, we can say also, that the extremities of the muscle shape havethe same position of the action lines extremities or, if we are using tendons, that theyare placed on the action line.

Taking into account the limitations imposed by our deformation model, whichwas designed specifically to simulate fusiform muscles, we developed a resamplingmethod whose goal is the generation of simple and regular surface of the muscles.From a muscle form composed by triangles, this method is able to generate anothermuscle shape designed in a way to achieve our objectives.

To resample given dense irregular triangle meshes into regular grids, the userneeds to define the number of slices perpendicular to the action line on the newmuscle shape, as well as the number of points in each slice. Excepting the extremities,every slice has the same number of points. The algorithm goes through all the actionline and, at each position of a new muscle slice performs the design of an imaginarycircle. Lines are drawn in a star-shaped manner with the origin on the intersectionbetween the action line and the slice. For each line we compute the outermostintersection point with the initial muscle. Each result point of these intersections willbe a point over the new muscle surface.

In Figure M3.16 we can see a modeled muscle with four examples ofresampled forms obtained by our editor.

Figure M3.16: Model of the internal par t of the Triceps (with 2160points) and the resampled models with respectively: 197, 182, 82 and32 points.

69

Note that there is a loss of quality involved in the resampling process. Here wehave the same usual dilemma: presentation quality versus speed. As we are lookingfor real-time deformations, our choice is speed. Muscles have a very complicatedform which is hard to reproduce. Furthermore, a lot of muscles are covered by othersand all of them are covered by the skin. Then, to represent a global animation, wereally do not need a high degree of accuracy in muscle shape representation.

Continuous level of detail and multi -resolution techniques [Certain 96][Lindstrom 96] have recently been studied to solve the conflicting requirements ofrepresentation quality and animation speed. In future work we intend to study theintegration of some multi -resolution methods in muscles presentation. For this, we areconsidering two kinds of multi -resolution: the spatial one, that aims to changeautomatically the degree of discretization of a muscle in function of its position inrelation to the others; and the temporal one, that changes the resolution according tothe position of the camera, during the visualization process.

2.2.3. Deformation Model

The problem of simulating deformable objects can be seen as a continuoussystem in space and time. The first step to solve this system is to discretize thecontinuous equations in material coordinates, resulting in a large system ofsimultaneous ordinary differential equations. The shape of a body is determined bythe Euclidean distances between nearby points while the evolution of the object in thescene depends on the forces acting on its points. The second step is the timediscretization that generates the evolution of the objects on time.

To physically simulate the deformation of a muscle, we use mechanical lawsof particles.

The motion of a particle is defined by its nature and by the position of otherobjects and particles in its neighborhood. In our specific case, we have decided toconsider only the representation of muscle surfaces, in order to reduce calculations. Infact, we believe we can simulate muscle deformation without considering directlytheir volume characteristics. The surface is composed by a set of particles with massdensity m. Their behavior are determined by their interaction with the other particlesthat define the muscle surface. In a correspondence with the geometric structurepresented before, each point of the mesh corresponds to a particle in the physicalmodel.

The physical model presented here is based on the application of forces overall mass points that compose the mesh, generating new positions for them. Adding allthe applied forces, we obtain a resultant force for each particle on the deformablemesh. For the sake of simplicity, we have considered three different forces: elasticityforce, curvature force and constraint force. Then, the resultant force in each particle ican be calculated as: �

),,,,(),,,,( 3210sconstraint

3210iiiiicurvatureelasticityiiiiiresult xxxxxfffxxxxxf ��� ,

where ix is the vector position of the particle i and 3210 ,,, iiii xxxx the positions of theparticles that compose its neighborhood.

Further, we present in detail the components involved on the resultant forceapplication over a particle, as well as the aspects involved in its definition andcalculus. We would like to emphasize that we have used a vectorial representation ofthe forces described here.

70

2.2.3.1. Elasticity

To simulate elastic effects between the particles on the muscle mesh, we haveused some concepts from the Theory of Elasticity, specifically concerning linearsprings. We suppose the connection between each particle and its four neighbors, withthe use of springs, as you can see in Figure M3.17.

Figure M3.17: Elastic model of the muscle sur face

Considering the force produced by a linear spring (also knowing as Hooke’sspring) over a mass point as � �

jiijijsjiijspring uxxxxkxxfij

)()(),(00 ����� ,

where ijsk is the coefficient that indicates the degree of elasticity of a spring, ix is

position of the spring’s oscill ating extremity and jx is the position of the fixed

extremity. 0i

x and 0j

x are the positions of the extremities i and j while the spring is in

the rest state and jiu the oriented unit vector from j to i. In extension, we can define

the elasticity force as the sum of all forces exerted over the point ix by the springsconnecting itself to its four neighbors. Now we can represent the elasticity force overone particle, as follows: �

�� 3

0

3210 ),(),,,,(j

jispringiiiiielasticity xxfxxxxxfiji

We have designed two classes of springs: vertical and horizontal ones (bothdefined in relation to the muscle action line), where the horizontal springs areperpendicular to the action line. The difference between the two relates thespecification of their elasticity coeff icients. Furthermore, we have two differentdegrees of elasticity: in width and height (always relating the muscle action line). Weconsider that the muscle height is defined in the same direction of its action line.Figure M3.18 shows the minimal elasticity component of a muscle, composed by aspring and a damping module.

71

Figure M3.18: Minimal elasticity component

2.2.3.2. Curvature and Torsion

The force that determines the degree of bending and twisting of a musclesurface was named curvature and torsion force. As the elasticity force, this force isalso calculated for each mass point over the surface as a function of its four neighbors.

The physical simulation of these effects was designed by employing anotherkind of linear spring, we have created and named angular springs. The differencebetween these springs and the other ones used to simulate elasticity is the way theyare attached. Consider a mass point 0x with a neighborhood formed by 4

030

20

10 ,,, xxxx ,

as shown in Figure M3.19. Each mass point 0x has two corresponding angular

springs, one in the angle defined by the vectors 100xx and 3

00xx , and the other one in

the angle between 200xx and 4

00xx .

Figure M3.19: Angular spr ing between the line segments 100xx and 3

00xx

The implementation of this kind of springs is done (consider the spring in

Figure M3.19, placed in the angle between 100xx and 3

00xx ) by attaching a spring

linking the point 0x with the mid-point of the line defined by the vector 30

10xx , when

all the points are in initial position, that is, with all springs in rest position.

As for the springs presented in Section 2.2.3.1, we can define the degree ofcurvature of a muscle in two dimensions. In order to implement this, we allow thespecification of curvature coeff icients in the two orientations (horizontal and vertical),considering the muscle action line as the up reference. We can say that the horizontalsprings are the ones perpendicular to the action line (defined between a mass pointand its neighbors over the same muscle slice - see Figure M3.20.a) and the verticalsprings, the springs defined between each mass point and its neighbors on the upperand lower slices (see Figure M3.20.b).

As we have simulated muscles only by their surfaces, we do not have therepresentation of a volume, but a surface designing a muscle shape. Like any surface

72

assembled by points linked to their neighbors by springs, it can twist and bend in anydirection, changing completely the expected final shape. With the use of angularsprings, we succeeded in avoiding this kind of behavior, as shown in Figure M3.21,where we compare a muscle compression with and without the use of angular springs.

(a) (b)

Figure M3.20: Example of the definition of a hor izontal angularspr ing (a) and a vertical one (b).

(a) (b) (c)

Figure M3.21: (a) Muscle in the rest position; (b) compressed musclewithout angular spr ings and (c) compressed muscle with angularspr ings.

Finally, we have realized that increasing the angular spring coeff icients, weare able to control the muscle volume during deformation, that is, we do not need apost-processing step for this purpose. Obviously, this is not a method thatmathematically guarantee the volume preservation. But it allows an efficient andquick way to provide different degrees of deformation to the same initial muscleshape, preserving the homogeneity and guaranteeing some kind of volume control inprocessing time.

2.2.3.3. Geometric Constraints

The geometric constraints were developed to improve the response to thedifferent conditions not formally explicit into our internal force model and that can beapplied to some specific points (local constraints), regions of muscles (zonalconstraints) or to all the muscle (global constraints). An example of local constraint isthe movement of the points that attach the muscle to the bones. The movement of

73

those points is function of the bones motion, that is, if you move your arm, themuscles attached at the corresponding bones will also move. Different conditions canalso result from a colli sion between a muscle and another internal structure, as organs,bones, fat, etc. One way to implement it can be by defining a zonal constraint. Withall the body in the rest position, the user define the regions of the muscle that are incontact with another organ, bone or so on and apply the same constraint over all theparticles comprise in the region. This constraint can be, for example, the conservationof a kind of link between a muscle particle and the surface of the other organ.

The implementation of the great part of the constraints force can be done byusing inverse dynamics. We specify the constraints actuating on a particle andcalculate the force necessary to compel the particle to obey these constraints. It canalso be called induced force. Knowing the position where a particle i should belocated, we apply a force that minimize the distance by creating a point to pointdisplacement. If ix is the current particle position in Euclidean space and

goalix its

ideal position, a force f should be applied to the particle i. This force f can gives amean to handle animation control as the path following or key-frame guiding. Toaccomplish it, at each time step, we need to define a new

goalix on the path.

The methodology used to satisfy constraints, allows easily the inclusion ofnew ones, without the need to modify the physical model. At this time, we are usingthe geometric constraints only to move the attachment points of muscles.

2.2.4. Motion Simulation

As already mentioned, the model used to deform muscles is based on a particlesystem where the interactions between particles are modeled by springs. Themovement simulation is done by applying systems of motion equations over eachparticle of the model. These systems rely on second-order differential equationsderived from a definition of elasticity as being the resistance to extension of amaterial, and viscosity as being resistance to a change in extension for a material[Holton 95].

We have used the Lagrange’s equations of motion as in [Terzopoulos 87]:

iexterniiiiiresultiiii fxxxxxfxxm ��� ),,,,( 3210��� �where iii xxx

���,, are respectively, the positions, velocities and accelerations of its mass

elements as function of material coordinates and time, iim�

, are the nodal mass andthe damping factor that dissipates the kinetic energy of the body’s mass elements,

),,,,( 3210iiiiiresult xxxxxf is the result force over the node i and

iexternf is an external

force. The external force is balanced against the force terms on the left hand side ofthe equation due to the deformable model. The first term represents the inertial forcedue to the model’s distributed mass. The second term is the damping force due todissipation and the third term is the elastic force due to the deformation of the modelaway from its natural shape.

To create animations simulating the dynamics of elastic models, the ordinarydifferential equations of motion are integrated through time. We use a fourth orderRunge-Kutta method [Press 92] for this. We have chosen this method because it ismore stable than Euler’s method while retaining an acceptable time of execution.

74

2.2.5. Animation Examples

As an example of our deformation method, we have simulated the motion ofthe Brachialis muscle, reconstructed from the images of the Visible Human Dataset(VHD). The Brachialis has the origin at the front of the lower half of the shaft of theHumerus and the insertion at the front of the coronoid process of the Ulna. It isresponsible for the flexion of the forearm and can be modeled by only one action line,between the ulna and the Humerus. We assume all muscles designed are in a state ofrest before the simulation.

In Figure M3.22 we can see a compression process of the Brachialis muscle.We have used a muscle composed by 50 mass-points and a compression rate of 30%.

Figure M3.22: Muscle compression sequence

In Figure M3.23 we have used the same parameters of the example shown inFigure M3.22, but with a different movement. In this case, we have simulated anextension rate of 30%.

Figure M3.23: Muscle extension sequence

75

In Figure M3.24, we have another example of compression. We have usedexactly the same parameters of the other two examples, but this time we havesimulated the deformation of the Brachialis with bones motion. Note that thecompression rate is much less significant than in the other examples. On the FigureM3.22 and M3.23 we have used more compression and extension rates than in reality.Exaggerating the motion, we can show better the model behavior.

Figure M3.24: Bones motion with muscle contraction

2.3. Skin Surface Generation

To generate the skin covering all the bones and muscles, we use a surfacesampled using ray-casting on semi-regular cylindrical grids. These sample points areused directly as cubic B-spline control points to smooth out the skin surface.Individual B-spline patches are triangulated, and these triangular meshes are stitchedtogether to connect different parts of the human body for final rendering and output.

This approach takes advantage from the fixed topology of the human skeleton.Human limbs exhibit a cylindrical topology and the underlying skeleton provides anatural centric axis upon which a number of cross-sections can be defined. Each limblink is associated with a number of contours (Figure M3.25.a). The cross-sectionalskin contours can automatically be extracted using the ray casting method. We castrays in a star-shaped manner for one contour, with ray origins sitting on the skeletonlink. For each ray, we compute the outermost intersection point with the muscles andbones surrounding the link. The intersection is a sample point on the cross-sectioncontour (see Figure M3.25.b).

2.3.1. Body Division

To facilit ate the manipulation, the body envelope was divided into seven parts,the so called skin pieces, each defined around a junction between two or more linkswhich contains a group of joints. These are: front torso, back torso, hip, left leg, rightleg, left arm, right arm. Each skin piece is represented by a bicubic B-spline patch,built from a collection of cross-sectional contours of the corresponding body part.

76

(a) (b)

Figure M3.25: (a) Cross-sectional plane or ientation and (b) raydistr ibution [Shen 96].

2.3.2. Cross-Sectional Configuration

Two rules are used to configure cross-sectional contours: joint angle drivenconfiguration and boundary driven configuration. The first rule is applied betweentwo links of a skin piece; the second is applied only to a few boundary contours of askin piece to ensure a natural transition across common boundaries of adjacent skinpieces. The joint angle driven configuration rule uses the angle between twoconnected links to drive the position and orientation of the cross-section planes in-between (Figure M3.25.a).

The joint angle driven configuration method regularly orientates the cross-sections of a skin piece in space according to the current joint angles of the relatedlinks, but it does not take into account the shape of adjacent skin piece's boundary.The idea behind the boundary driven configuration is to mimic a "force field" that, bycreating a few intermediate contours inbetween a start contour and a goal contour,makes the start contour tend to the goal contour. This ensures a natural and smoothtransition of shape across common boundaries. Here, the start contour and goalcontour belong to the boundary of two adjacent skin pieces respectively.

2.3.3. Grouping Primitives

However, some postures could produce errors on the skin generation whileusing the method described above. An example is when a leg or an arm of a characteris bended. Considering the arm example, our algorithm will generate the skincovering at the same time (as an unique piece) the upper and lower part of the arm.Using the same principle, we could also have a single piece of skin covering the twolegs of the character together. To solve this problem, we have decided to assign eachprimitive (a bone or a muscle) to a group. Each group will blend with primitives in itsown group or in the root group. Fifteen groups were defined covering whole body:upper_torso, lower_torso, hip, left_shoulder, right_shoulder, left_upper_arm,left_lower_arm, right_upper_arm, right_lower_arm, left_upper_leg, left_lower_leg,right_upper_leg, right_lower_leg, left_ankle, and right_ankle. Figure M3.26 locatesall of these groups over the body.

77

Figure M3.26: Groups position over the body

Each primitive is assigned to a group according to its attachment to theskeleton. Some primitives, located on special positions (for example, a musclecovering a joint), can fall i nto several groups simultaneously, as they may havecontributions to multiple groups. To sample a contour, we only need to consider theprimitives in its associated group. For example, skin piece left leg consists of threegroups: left_upper_leg, left_lower_leg, left_ankle.

2.3.4. Rapid Tesselation

We tesselate the B-spline patches into triangular meshes for three reasons:� Our facial animation and hand deformation systems use surface-basedtechniques, i. e., they use triangle meshes to represent the final shape offaces and hands. Thus, we need to produce the deformed body shape astriangle meshes also;� Interactive modeling demands fast display and manipulation. Althoughtoday's high-end workstations can display huge polygon meshes in realtime, they may not display several complex B-spline patches at interactiverates;� Seamlessly joining B-spline patches together isn't easy, whereas stitchingtogether triangular meshes is trivial.

The skin generation method presented above is based on techniques developedby Shen and described in [Shen 95, 96].

2.4. Integration of the Parts

To integrate our joints-based skeleton, the bones, the muscles and the skin, wehave extended the Body Builder system [Shen 96], originally designed for deformablehuman bodies with a stick skeleton, muscles and fat tissues represented by elli psoids,and the skin. Our extended system, Body Builder “ Plus” allows the construction ofan anatomically-based model of humans, created entirely with bones andreconstructed muscles. A second objective is to mix the modeled deformable muscles

78

described in Section 2.2, with elli psoids to represent some muscles and other fattissues. The goal, in this case, is to use physically-based deformable muscles tosimulate only the muscles that influence quite a lot on the human external appearance.

The Body Builder Plus is an interactive human body modeling anddeformation system developed on SGI workstations and based on a multi -layeredmodel. The system allows the design of three-dimensional animated human figures inan interactive environment [Paill et 97]. Models may be animated by input motionsequences to the system or by changing a joint angle individually and interactively.

Concerning the model, it is divided into three layers: the skeleton, composedby a topological tree structure and the bones, as explained in Section 2.1; thevolumetric primitives, composed by elli psoids and the muscles stressed in Section2.2; and the skin, represented by a B-spline surface covering all the body (Section2.3). The head, feet and hands are modeled separately and attached to the bodyafterwards. More information about hands and face modeling can be found in Sections3 and 4.

2.4.1. Volume Primitives

In Body Builder Plus, the volume primitives are the pieces used to representmuscles, fat, organs and so on. We have used two kind of primitives, the physically-based muscles and the elli psoids. Here we will explain shortly the use of theelli psoids, integrated in the system by Shen [Shen 96]. They are divided into twocategories: blendable volume which blends with other blendable primitives in thesame group, and unblendable volume which do not blends with other primitives. Forthe sake of simplicity, we use only elli psoids for unblendable primitives andisosurfaces with elli psoidal density distribution for blendable primitives.

Each primitive can also be classified as deformable or not. Each deformableprimitive is associated with a reference joint, whose value dynamically determines thecenter, orientation and shape of that primitive. When the skeleton moves, allprimitives attached to their relevant joints undergo the joint hierarchy transformationsas rigid body motions. Deformable primitives change also their state. In fact, eachDegree of Freedom of a joint is defined by three angles: the minimum, the maximumand the current angle (initially corresponding to the default posture). We defineparameters for the primitives when the joint angle is on this three states and after,during the animation, these parameters are interpolated. Figure M3.27 shows anexample of using this kind of volume primitives, presenting two bodies representedwith and without the skin. In fact, the difference between blendable and unblendableprimitives can be perceived only after the skin generation.

2.4.2. The Extremities

The body extremities (the head, the feet, the hands and the penis) are nottreated in detail i n Body Builder Plus. In fact, they are modeled separately and savedin individual files (<filename>.sm) containing triangular-mesh surfaces. The systemreads the sm files [Kalra 97] and allows the user to position them interactively. Thelink with the body is made at the skin level.

79

Figure M3.27: Example of a body composed by unblendable (upperbodies) and blendable pr imitives (lower bodies). Pr imitives aredifferentiated by their colors: green (unblendable) and yellow(deformable unblendable) in the upper body, pink (blendable) andviolet (deformable blendable) in the lower body.

2.4.3. System Organization

2.4.3.1. Architecture

The Body Builder Plus (bb+) system is an application program based onseveral li braries. In general we can say that it furnishes the interface functions and themotion motor, while all the functions concerning the body modeling and motion, themuscles and elli psoids deformation, and the skin generation are provided by thelibraries. Figure M3.28 shows the system architecture and the libraries dependencies.

The most important libraries called by bb+ are skinlib [Shen 98] andmusclelib. Skinlib supports the deformation of the elli psoids, the skin generation andall other functions concerning the realistic deformation of actors. It provides also thereading of the model file (<filename>.M) and the generation of the output files.Musclelib provides all the functions about physically-based deformation of musclesand is called directly by bb+ and skinlib. Bodylib [Boulic 94b] is the library thatmaintains the topological tree structure for a vertebrate body with predefinedmobilit y. It is responsible of the animation of the body, at the joints level. Scenelib[Boulic 94c] is a lower level li brary that maintains the general purpose hierarchy andallows multiple positioning of complex functional units. Smlib [Kalra 97] is thelowest level li brary we have and is responsible by the triangle mesh surface

80

manipulation. Facelib and handlib [Boulic 95] are responsible respectively of facialanimation and hand deformation. The user interface was designed by using OpenInventor [Wernecke 94] library for bodies visualization and Motif for the windowsand two-dimensional interface design.

Figure M3.28: System architecture

Concerning the system input, we need to consider two important data. Themodel (<filename>.M) and the motion files (<filename>.TRK). The motion filecontains all the information about body motion on the joints level, but the mostimportant data set is on the model file. This file contains all the data required to modela human body and can be edited manually or by using bb+ interactively. It is writtenin a standard ASCII f ormat and include the skeleton scaling parameters, skeletontemplate file, head/hands/feet/penis sm file names, parameters of all primitives,definition of physically-based muscles and action lines and bones specification.

The system output is made visually or by using a set of output files, allowingthe integration of the human figures with other applications. Body Builder Plus is ableto export human models in different formats: SM (<filename>.sm), Wavefront(<filename>.obj), Rayshade (<filename>.ray), Open Inventor (<filename>.iv) andcontours (<filename>.dat) for real-time animation [Thalmann 96].

2.4.3.2. The Motion Motor Algorithm

Currently, there are two ways to perform human motion. The first wayconsists in picking a joint and changing interactively the value of this joint, while thesecond one is to read and execute the motion from an animation file. In both cases, thesystem detects the joints motion and enables the deformation of the volumetricprimitives, the motion of bones and the update of the action lines.

Each object in bb+ is represented globally and locally. For simpli ficationpurposes during the motion process, we consider the same global reference system forall the human figure and several local reference systems, one for each Degree ofFreedom of the body. Every object that composes a human body is attached to a jointreference and represented in its local reference system. Then, when this joint moves,the object moves in the same time, automatically. This is valid for bones, for the

81

extremities (sm surfaces) and for non-deformable primitives. In the case ofdeformable primitives and physically-based muscles, another procedure is yetnecessary. Deformable primitives have their parameters interpolated in the localreference system, while the physically-based muscles motion are guaranteed byupdating the action lines.

The action line is the lowest level in our muscle simulation system. We firstsimulate the action line motion and then, the muscle deformation. In this way, themuscle compression or extension depends directly on the action line. However, thefinal deformation depends also of other parameters, as shown in Section 2.4.4.3. Infact, action lines are attached to bones that move accordingly with the joints. When ajoint angle changes, bones move and consequently, muscles attached to these bonesalso move on the same way.

The action line motion is produced in a module that, after each movement ofthe skeleton, detects which action lines are concerned and update these ends andcontrol points positions, calculating the new tendons location over the action line. Toaccomplish it, we maintain a list with pointers to all the segments of action lines and aflag for each one indicating if it is in motion or not. If the action line changes during ajoint motion, then the flag is set.

To enable the physically-based muscles deformation, the system maintains acallback function called 20 times per second. This function verifies if there are someaction line or muscle in motion. If one of the two conditions is satisfied, the muscledeformation procedure described in Section 2.2, starts. A muscle will be “static”again, only when the motion of all the particles can be considered as non-representative (very small ). Figure M3.29 shows the flow chart of the motion motordescribed here.

2.4.4. How to Build an Anatomically-Based Body?

Due the complexity of the system, the design of a new body in Body BuilderPlus is produced on, at least, four steps.

2.4.4.1. Skeleton Definition

The first step is the definition of the skeleton characteristics. From a basichuman topology, it is possible to edit globally the limbs length in order to create anamount of different humans, with distinct sizes and proportions. Five normalizedparameters are used to scale the standard skeleton template to accommodate variationsin age, sex and race [Boulic 94b]. Global scaling consists of an uniform scaling madealong x, y, z to attain a given height). Frontal scaling is a specific scaling made alongone independent direction. High lateral scaling and low lateral scaling are proposedto differentiate the higher body from the lower body (useful for generatecharacterization). Finally, spine origin ratio express the ratio of spine origin heightover the total height, while keeping the same total height. Figure M3.30 demonstratesall the skeleton scaling general options, while Figure M3.31 furnishes an example of askeleton instance definition, in a model file.

82

Figure M3.29: Motion motor flow char t

Figure M3.30: Skeleton scaling [Boulic 94c]

83

inst skeleton 1 {high 1749.790039 ,spine_ratio 1.000 ,high_lateral_scale 0.950 ,low_lateral_scale 1.000 ,frontal_ratio 1.000 ,gender male ,template “newton.tpl” ,

} ;

Figure M3.31: Extract of a model file with the definition of theskeleton instance.

2.4.4.2. Adding Bones

The second step comprises the addition of the bones. Bones are represented bytriangle meshes and are previously stored in individual files. Their insertion in BodyBuilder Plus is made by editing the model file and by adding the related data. Asshown in the example of Figure M3.32, each instance of a bone have a name, the jointname where the bone is attached, the topology file name and the associated group(group concept and use was explained in Section 2.3.3).

inst bone 5 {name “lhumerus” ,parent joint l_shoulder_twisting ,file “lhumerus.iv” ,group l_arm_upper l_shoulder ,

};

inst bone 40 {name “lfemur” ,parent joint l_ hip_twisting ,file “lfemur.iv” ,group l_leg_upper left_ankle ,

};

Figure M3.32: Model file extract with the definition of two bones

We provide a database with the great part of the human body bones,normalized. Before to integrate it in bb+ , the user needs to scale all the bones, but thistask can be accomplished with the help of a littl e tool developed by us. Furnishing thepositions of the two articulations that comprise a bone, it is scaled and saved inanother topology file (using Open Inventor file format). We maintain also the scaledmuscles to be used with the skeleton structure described in the Section 2.1, but theuser is free to change the base skeleton or yet the bones form, by editing the old onesor by modeling new shapes.

2.4.4.3. Creating Muscles

The third step is the muscle creation. In our system, the user can defineinteractively the muscles action lines, saving the data in the model file. Muscle shapeswith deformation parameters are defined in an instance called mms (Muscle ModelSimulation data) and attached to the action line segments also interactively. Thedeformation parameters can be added by editing the model file or by reading it from amms file format.

In Figure M3.33 we can see an example of an instance model of the muscletype. Each muscle has a name, is associated to one or more groups and is defined byone or more action lines. Each extremity of the action line is attached to a joint andhave a position defined in the reference joint local coordinate system. Each action line

84

can be related or not to a muscle shape by indicating the mms index (-1 if no muscleattached). Each mms instance contains the muscle shape file name and the parametersrequired to perform the deformation. The tendons length are furnished for each actionline, as well as a rotation angle used to better adapt the muscle shape to the desiredorientation over the skeleton. For more information on action lines or tendons, refersto Section 2.2.1.

inst model 297 {name “right_triceps” ,type muscle ,group r_arm_upper r_arm_lower r_shoulder ,origin joint r_clav_abduct position -52.0 117.0 -34.0 keyblob 10,insertion joint r_shoulder_abduct position -5.0 13.0 -13.0 keyblob 5,tendon_origin 0.0 ,tendon_insertion 36.0 ,mms 1 ,rotation 36.0 ,

origin joint r_clav_abduct position -52.0 117.0 -34.0 keyblob 10,insertion joint r_shoulder_abduct position -5.0 13.0 -13.0 keyblob 5,tendon_origin 42.0 ,tendon_insertion 158.0 ,mms 2 ,rotation 180.0 ,

origin joint r_clav_abduct position -52 .0 117.0 -34.0 keyblob 10,insertion joint r_shoulder_abduct position -5.0 13.0 -13.0 keyblob 5,tendon_origin 15.0 ,tendon_insertion 120.0 ,mms 3 ,rotation 57.0 ,

} ;

inst mms 1 {file “triceps_Lportion.iv” ,degree_of_subdivision_on_x 10 ,degree_of_subdivision_on_y 5 ,elasticity_horizontal 0.0 ,elasticity_vertical 5.0 ,curvature_horizontal 0.0 ,curvature_vertical 5.0 ,total_mass 20.0 ,damping 0.5 ,

} ;

Figure M3.33: Extract of a model file with the definition of a complex muscle

2.4.4.4. Skin Generation

The fourth step of the process is the skin generation and it is madeautomatically by Body Builder Plus. We can also consider another intermediate step,if the user desires to model a part of the body by using elli psoids. This procedure is inthe same level of the muscles definition step and could be made interactively.

2.4.5. Results

On Figure M3.34, we can see an example of a body composed by metaballsand two physically-based deformable muscles to simulate the Biceps Brachii , coveredby the skin.

85

Figure M3.34: Body design with metaballs and physically-baseddeformable muscles.

Figure M3.35 shows the current state of our work in designing real muscles forthe representation of a complete human body. The upper line shows our skeleton withthe muscles represented by their action lines while the lower line presents theresampled bones we have designed for the moment. Our current model is composed of31 joints with 62 degrees of freedom, 73 bones, 33 muscles (represented by 105fusiform muscles parts) and 186 action lines. In Figure M3.36 the muscles areidentified.

86

Figure M3.35: Muscled body

87

Figure M3.36: Muscles identification

Figure M3.37 shows an example of elbow flexion performed by a bodycomposed of bones and some muscles, specifically the Biceps Brachii and the TricepsBrachii .

Figure M3.37: Elbow flexion with muscles deformation

3. Facial Modeling

The face modeling involves all the problems concerned in the body modeling,but as its shape is specially irregular and the movements associated to it (mainly ofexpression and speaking) are rather complex, its diff iculties are greater. Because thehuman face plays the most important role for identification and communication,realistic construction and animation of the face is of immense interest in the researchof human animation.

The ultimate goal of the research results presented here would be to modelexactly the human facial anatomy and movements to satisfy both structural and

88

functional aspects. However, this involves many problems to be solved concurrently.The human face is a very irregular structure, which varies from person to person. Theproblem is further compounded with its interior details such as muscles, bones andtissues, and the motion, which involves complex interactions and deformations ofdifferent facial features.

The properties of facial expressions have been studied for 25 years by apsychologist, Ekman, who proposed a parameterization of muscles with theirrelationships to emotion: the Facial Action Coding System (FACS) [Ekman 78].FACS describes the set of all possible basic actions performable on a human face, andis the base for defining the principal parameters in the modeling of faces.

One of the firsts works published on facial modeling is perhaps the paper ofParke [Parke 82]. In his system, Parke models the face as a set of surface patchestaking into account some characteristics parameters of modeling. The number and theshape of the patches are pre-fixed and, particularly, the number of the patches is biggerin the problematic regions (eyes, mouth, etc.). Two types of parameters areconsidered: the expression parameters and the conformation parameters. Theconformation parameters are chosen when modeling the face and are concerned withmeasurements such as nose length, eyes separation, forehead width, etc. They definethe face at a “neutral” state. The expressive parameters, variable, are not concernedwith the shape of the face but with its possible movements: mouth opening, pupildilation, etc.

Platt and Badler [Platt 81] consider the modeling of the three anatomic levelsof the face: bones, muscles and skin. The skeletal level is modeled as a rigid surfaceunder the deformable surface of the skin. Muscles are represented as the intermediatelevel between those two. Links are established between the points of the surface of theskeleton, related one to another by arcs passing through a set of intermediate pointsthat simulate muscles. This complex modeling allows a real simulation of the facialmovements.

3.1. Motion Simulation

Facial deformations can be understood as externalization or manifestation ofverbal or non-verbal communication agents on a face. These agents activate certainchannels of the face associatively. Each activated channel in turn triggers the relevantmuscles. Activation of muscles eventually deforms the face. The module for facialanimation presented by Boulic et al. [Boulic 95] resolves the diff iculty of manuallymanipulating the facial model by offering a multi -level structure to the system, whereeach level is independently controllable [Kalra 93]. The different levels encompassinformation from the various levels of abstraction from sentences and emotions to thefacial geometry and image visualization. For simulation of muscle actions they useRational Free Form Deformations (RFFD) as they are: simple and easy to perform,intuitive and rapid to use [Kalra 92]. RFFD includes an extra term for weights for thecontrol lattice of FFD. This provides additional control for deformations. The muscledesign uses a region-based approach which means that the region of interest is definedwhere a muscle comprising RFFDs can be associated. The deformations which areobtained by actuating muscles to stretch, squash, expand and compress the insidefacial geometry, are simulated by displacing the control points of the lattice and/or bychanging the weights of the control points.

Human face may render complex movements of its different parts during afacial expression. There are more than thirty muscles responsible for the facial

89

movements. These muscles are associated with the source of movements such asspeech or emotion. The use of muscles for specifying an animation is diff icult for theuser. The multi l evel structure enables users to specify animation in terms of high levelglobal actions like emotion and sentences for speech. These actions are decomposedinto discrete actions like expressions and phonemes for words. The discrete actionscontain the basic motion parameter referred to as Minimum Perceptible Action (MPA)[Kalra 91]. These MPAs are, in fact, aggregation of muscle actions. In order to controlthe duration and instance of each global action, a synchronization mechanism isprovided. The high level specification of animation is independent of low level facialmodel.

3.2. Texture Mapping

To improve the “Barbie-like” aspect of virtual human beings, they propose atechnique based on texture mapping of photos of real faces. A separate tool formatching the 3D facial topology on a given picture/photo of a face was developed.Only a few feature points are selected from the 3D model to exactly match thecorresponding points on the picture. Delaunay triangulation was used to connect thesepoints, which can be moved and displaced on the picture interactively. Aninterpolation scheme in a triangular domain was used to get the desired texturecoordinates. As a result, the picture was deformed and mapped on the 3D model. Inorder to map the entire head, multiple views are needed. These pictures are projectedon a cylinder. Then the corresponding matching is performed between the cylindricalprojected 3D model points and cylindrical projected pictures. By using texturemapping, the quality of rendering was improved considerably. In addition, it allows usto put a picture of a specific person on a given 3D model. Figure M3.38 and M3.39show expressions with texture mapping.

Figure M3.38: Facial expressions examples [Miralab 99]

4. Hands

Hand animation is generally treated separately from the rest of the body. Themotivation for a different deformation technique for the hands comes from their veryspecific behavior. The inner part of the hand is crisscrossed by lines and wrinkles thatcreate discontinuities on the surface during deformation. The palm covers fiveskeleton segments and is very flexible. Finger deformations are very complexcompared to other parts of the body, as they involve a wide range of angles’ variationsand configurations applied to very short skeleton segments.

Magnenat-Thalmann et al. [Magnenat 88] propose a model for handdeformation based on local geometric operators called Joint Local Dependentdeformations, which are used for hand animation when there is no contact with the

90

environment. Gourret et al. [Gourret 89] deal with animation and interaction usingFinite Elements. Their method is used when contact response is needed in graspingtasks. Delingette et al. [Delingette 93] describe a simplex-based model for objectrepresentation which is particularly suited for physical simulation and allows 3Dmorphing, data extraction and reconstruction, and hand animation. Uda et al. [Uda 93]extend the capabiliti es of their solid modeler to simulate hands. Hands areautomatically skinned with rough volume polyhedron approximations. Skin curvatureat joints is generated by an ad-hoc polyhedral subdivision method.

Figure M3.39: Facial expressions example [Miralab 99]

An alternate approach consists in enhancing the free-form deformations(FFD’s) techniques [Sederberg 86] by using results of data interpolation [Farin 90] toremove existing limitations of the current FFD’ models (especially for the animationof articulated characters). In [Farin 90], Farin extents the natural neighbors interpolantbased on the natural neighbors coordinates, by using the natural neighbors as thesupport for a multivariate Bézier simplex, in which any point can be expressed with asimilar relation as in FFD’s. Farin defines a new type of surfaces defined with thisextended interpolant called Dirichlet surfaces. Combining FFD’s and Dirichletsurfaces leads to an enhanced model of FFD’s: Dirichlet FFD’s or DFFD’s [Moccozet

91

97]. One major advantage of this technique is that it removes any constraint on theposition and topology of control points. Three types of control points are defined:� normal: they are defined around or inside the surface at a variable distance

from it;� constraint: they lie on the surface and allow its direct manipulation. Theyare generally assigned to a point of the surface, so that any displacement ofthe control point is completely passed to the surface’s point;� inflating: a special type of normal control points but with varying weight,so that they can attract the surface in their neighborhood and simulateinflation of the surface.

From that general free-form deformation model, we derive a specialized datastructure for multi -layered deformations of articulated object, where the set of controlpoints is used to simulate the muscle layer [Chadwick 89, Lamousin 94].

4.1. Hand Topography

A simpli fied description of hand topography is presented and ill ustrated inFigure M3.40.

(1) palm(2) upper transversal li ne(3) lower transversal li ne(4) thenar eminence(5) thenar line(6) thumb first line(7) thumb second line(8) hypothenar eminence(9) finger first line(10) finger second line

(11) finger third line

Figure M3.40: Topography of the hand [Moccozet 97]

The palm of the hand is divided into three parts :� the palm itself (1), crossed by two parallel li nes: the upper (2) and lower(3) transversal li nes, which will be further simpli fied by one line;� the thenar eminence (4), delimited by the base of the thumb and the thenarline (5);� the hypothenar eminence (8), delimited by the wrist, the thenar line andthe lower transversal li ne.

The four fingers are placed at the top of the palm, each being approximatelydivided by three lines (9), (10), (11) associated with the joints of the underlyingskeleton. The thumb is placed at the base of the external side of the hand, separatedfrom the thenar eminence by one line (6). It is divided in two phalanxes, separated byone line (7).

From the observation of a hand, especially its topography, we can derive thefollowing simpli fied observations:� each joint of the skeleton is associated with a hand line on the surface of

the hand;

92

� each wrinkle that can be seen often has a closed line, and has almost nochange in its shape;� rotation at the joint creates an inflation of the part delimited by the handline associated with the rotating joint and the hand line just above;� hand lines stop the propagation of the skin inflation.

4.2. Hand Simulation Model

From the previous observations, we define an adequate data structure torepresent a wrinkle. One wrinkle will be associated with each joint or degree offreedom of a joint.

A wrinkle is composed of the following information:� the wrinkle itself, which is a set of constrained control points that aregenerally selected around the joint and formed a closed 3D line. We callsuch points wrinkle control points;� two points among the wrinkle control points that are used to define theaxis on which the associated skeleton joint should lie. That way, a skeletonmodel can be easily adapted to the hand's skin. This data allows an easyand realistic hand-skeleton mapping by defining an implicit skeleton towhich the skeleton can be fitted;� a mixed set of control points and constrained control points that surroundthe upper part of the hand surface that will be affected by rotation of thejoint associated with the current wrinkle. We call these points influencedwrinkle control points, as they are influenced by the rotation of the wrinkleitself;� one control point, called an inflation control point, which will be used tosimulate inflation at the upper limb associated with the joint.

The set of wrinkles is designed interactively from the 3D hand's skin bybuilding points from the geometric elements of the mesh. For example, we can createa control point by selecting a triangle: a control point will be built from it by firstcalculating the barycenter of the three extremities, and then moving the point outsideof the figure along the triangle normal.

For each wrinkle, the muscle layer gets the joint angle variation from theskeleton layer. If the rotation angle is a, the wrinkle itself is rotated of an angle of a/2,and the set of influenced control points is rotated by a. At rest position, all controlpoints have a weight of 1. When the joint angles vary, the weights of the inflationcontrol points vary accordingly, such that weight(P) = f(a), where P is an inflationcontrol point and a an angle of rotation at the joint. This point is placed on the mesh sothat when its weight increases, it attracts the mesh.

One unclosed wrinkle exists at the base of each of the four fingers. The unionof these four wrinkles defines a closed composite wrinkle which simulates thetransversal li ne going through the palm, and each distinct wrinkle will simulate theline at the first joint of the finger. The topography is then simpli fied without alteringthe visual aspect of the deformed hand. We also associate a wrinkle data structure witheach metacarpus [Hsu 92], but we use fewer settings. As there is no real associatedhand line, the wrinkle itself is empty, i.e., contains no wrinkle control points, and therotation axis is defined by the wrinkle control points defining the wrist wrinklerotation axis. Only influenced control points are defined for metacarpi wrinkles. Inorder to simulate the influence from neighboring metacarp, neighboring metacarpi

93

wrinkles contain some common influencing control points. Their rotation is notdirectly defined from the rotation angle of one joint, but is an average of the rotationangles assigned to the two wrinkles to which they belong. For the thumb, the wrinkleassociated to the thenar line is adapted to its particular situation, dividing the palm intotwo regions. The wrinkle itself is not a closed line, but starts from the wrinkle's wriston the inside part of the hand, and ends at the wrinkle's wrist on the back of the hand.Wrinkle control points on the back are not constrained control points, but standardcontrol points, in order to maintain the smoothness of the skin on the back of the hand.

Figure M3.41 shows how control points, constraint control point and attractivecontrol points are designed around the surface of the hand to build the control pointsset and the different wrinkles.

Figure M3.41: Control points set and wr inkle design [Moccozet 97]

Figure M3.42 ill ustrates how the control lattice and the wrinkles are designedover the hand's geometrical skin. The thumb is zoomed to show the three associatedhand lines drawn with white control points.

Figure M3.42: Hand sur face and control latt ice [Moccozet 97]

Figures M3.43 and M3.44 show some examples of hands modeling in differentpostures.

Another problem connected to hands animation is the problem of grasping anobject. To grasp an object, the hand has to be used, and the joints of the hand mustmove correctly. And, if an object is grasped, it has to move with the hand. When thehand is turned, the object must turn along with the hand. When an object is graspedand moved to a new position, it is the arm that essentially guides the object. However,to grasp the object, flexion angles have to be determined. This problem is known inrobotics as an inverse kinematics problem. Magnenat-Thalmann et al. [Magnenat 88]describe algorithms and methods used to animate a hand for a synthetic actor. Thealgorithms allow not only the movement of the hand and grasp objects, but also theycompute the deformations of the hands: rounding at joints and muscle inflations.Figure M3.45 shows some examples of grasping.

94

Figure M3.43: Some examples of hands modeling [Moccozet 97]

Figure M3.44: More examples [Moccozet 97]

Figure M3.45: Some examples of grasping [Moccozet 97]

5. Hair

In the field of human animation, hair presents perhaps the most challengingrendering problem an therefore has been one of the least satisfactory aspects of human

95

images rendered to date. The diff iculties of rendering hair result from the large numberand detailed geometry of the individual hairs, the complex interaction of light andshadow among the hairs, and the small scale of the hair width in comparison with therendered image. The rendering of hair therefore constitutes a considerable anti-aliasingproblem in which many individual hairs, reflecting light and casting shadows on eachother, contribute to the shading of each pixel.

Several researchers have published methods for rendering fur and human hair.Gavin Mill er [Mill er 88] modeled hair with triangles to form a pyramid, usingoversampling to avoid aliasing. Watanabe and Suenaga [Watanabe 89] modeledhuman hairs as connected segments of triangular prisms and were able to render a fullhead of straight human hair in a reasonably short time using a hardware Z-bufferrenderer with Gouraud shading. Perlin and Hoffert [Perlin 89] employed volumedensities, controlled with pseudo-random functions, to generate soft fur-li ke objects.Perhaps the most impressive rendering of fur to date was achieved by Kajiya and Kay[Kajiya 89] for a teddy bear using a generalization of 3D texturing known as texels.Rosenblum et al. [Rosenblum 91] presented hair animation method using a massspring model. Anjyo et al. [Anjyo 92] proposed methods using one-dimensionalprojective differential equations and pseudo-force fields. Both methods neglect theeffect of colli sion between hairs for simplicity. Kurihara et al. [Kurihara 93] proposeda simpli fied colli sion detection method using cylindrical representation.

5.1. Hair Animation

To generate natural hair animation, physical simulation must be applied.However, precise simulation including colli sion response is impractical because of thenumber of individual hairs.

The simulation model presented by Magnenat-Thalmann et al. [Magnenat 96]takes benefit from two properties of the hairs, they are, on one side, low deformationof the initial shape, and on the other side, weak variation of the shape from one hair toanother. The resulting model consists of reducing hair’s deformations to those of atube. Each hair stand is parametrically defined inside the tube, so that it follows itsdeformations (Figure M3.46). Physical simulation is performed on a set of such tubes,defined as the hair’s core. Tubes are the interpolated over the support according to thehair density.

Figure M3.46: Hair deformation reduced to the deformation of a tube[Magnenat 96].

The hair’s core is described using a spline that is interpolated from a fewanimated vertices linked to each other by edges. The final hair’s shape is computedfrom that spline (Figure M3.47). A spline is extracted from the position of the verticesof the hair’s core. The representation is achieved by computing a number ofcoordinates, which depend on the complexity of the shape. The coordinates are thenconsidered as control points and the hair is represented as a NURB.

Hairs do not exist by themselves. Each hair strand extremity must be attachedto a vertex of the support object: the hair’s root. An orientation matrix is associated tothe hair’s root. The matrix is unique for the whole hair if the support is rigid, or

96

different for each root otherwise. The orientation matrix allows us to perform hairanimation on any type of support, and simulate for example a fur coat.

Figure M3.47: Progression from the hair 's core formed with animatededges and vertices, to the spline defined with interpolated controlpoints, to the hair itself [Magnenat 96].

5.2. Colli sion Detection

In order to properly achieve hair animation, the model must include a colli siondetection and treatment process. This model takes into account two types of colli sion:

a) Colli sion between hair strands: the result of these colli sions is to givevolume to the hair. Instead of detecting colli sion, a repulsion force isintroduced when two strands are closer than a proximity factor. That factordepends on hair density and shape. Colli sion treatment is restricted toneighbor hair’s cores.

b) Colli sion between hairs and support: the goal is to avoid hair strandpenetration inside the hair support object. Colli sion treatment is alsorestricted to the hair’s core.

5.3. Hair Rendering

Rendering an image of hair involves several steps:� creating a database of hair segments;� creating shadow buffers from all li ghts;� rendering the hairless objects using all shadow buffers;� composing the hair on the hairless image.

The simulation module produces a database of hair segments for each frame,that is used as input for the rendering module. In this system, hair rendering is done byray-tracing using a modified version of the public domain Rayshade program. Theprocess is step by step. First, the shadow of the scene is calculated for each lightsource i, as well as for the light sources for the hair shadows. The hair shadows arecalculated for the object surface and individually for each hair. Finally, the hair style isblended into the scene, using all shadow buffers. The result is an image with a three-dimensional realistic hair style rendering where complex shadow interaction andhighlight effects can be seen and appreciated.

The raytracer basically has two roles in the hair rendering pipeline. The first isto construct shadow buffers from light sources, and the second is to render hairlessobjects with full shadowing (coming from the hair and the hairless objects).

Figure M3.48 shows an example of hair animation and Figure M3.49 anexample of hair rendering.

97

Figure M3.48: Animating hairs [Magnenat 96]

Figure M3.49: Example of hair rendering [Miralab 99]

98

6. References

[Anjyo 92] Anjyo, K., Usami, Y. and Kurihara, T. “A Simple Method forExtracting the Natural Beauty of Hair” , Proceedings ofSIGGRAPH’92, Computer Graphics, v. 26, n. 2, pp. 111-120, 1992.

[Boulic 91] Boulic, R. and Renault, O. “3D Hierarchies for Animation” , In: NewTrends in Animation and Visualization, Edited by Nadia Magnenat-Thalmann and Daniel Thalmann, John Wiley & Sons ltd., England,1991.

[Boulic 94] Boulic, R., Huang, Z., Magnenat-Thalmann, N. and Thalmann, D.“Goal-Oriented Design and Correction of Articulated Figure Motionwith the Track System”, Computer and Graphics, v. 18, n. 4, p. 443-452, 1994.

[Boulic 94b] Boulic, R., Çapin, T., Huan, Z., Molet, T., Shen, J., Magnenat-Thalmann, N. and Thalmann, D. “Human Data Structure & ParallelIntegrated Motion - Specialized Hierarchy for a General VertebrateBody - User Reference Manual” , Report of the ESPRIT Project 6709- HUMANOID, EPFL, Lausanne, Switzerland, 1994.

[Boulic 94c] Boulic, R., Çapin, T., Huang, Z., Molet, T., Shen, J., Kalra, P.,Moccozet, L., Werner, H. M., Magnenat-Thalmann, N. andThalmann, D. “General Purpose Hierarchy - User ReferenceManual” , report of the ESPRIT Project 6709 - HUMANOID, EPFL,Lausanne, Switzerland, 1994.

[Boulic 95] Boulic, R., Capin, T., Kalra, P., Lintermann, B., Moccozet, L.,Molet, T., Huang, Z., Magnenat-Thalmann, N., Saar, K., Schmitt, A.,Shen, J. and Thalmann, D. “A system for the parallel integratedmotion of multiple deformable human characters with colli siondetection, In: EUROGRAPHICS’95, Computer Graphics Forum, n.14, v. 3, p. 337-348, Maastricht, Holland, 1995.

[Certain 96] Certain, A., Popovic, J., DeRose, T., Duchamp, T., Salesin, D. andStuetzle, W. “ Interactive Multi resolution Surface Viewing” ,Computer Graphics, New Orleans, ACM SIGGRAPH, 1996.

[Chadwick 89] Chadwick, J. E., Haumann, D. R. and Parent, R. E. “LayeredConstruction for Deformable Animated Characters” , ComputerGraphics, v. 23, n. 3, p. 243-252, July 1989.

[Cody 90] Cody, M. D. J. “Visualizing Muscles - A New Ecorché Approach toSurface Anatomy”, University Press of Kansas, 1990. 231p.

[Delingette 93] Delingette, H., Watanabe, Y., Suenaga, Y. “Simplex BasedAnimation” , Proceedings of Computer Animation’93, Eds. N.Magnenat-Thalmann and D. Thalmann, Springer Verlag, pp. 13-28,1993.

[Ekman 78] Ekman, P. and Friesen, W. Facial action coding system, ConsultingPsychologist Press, Palo Alto, CA, 1978.

[Farin 90] Farin, G. “Surfaces Over Dirichlet Tesselations” , Computer AidedDesign, v. 7, pp. 281-292, 1990.

99

[Gourret 89] Gourret, J.-P., Magnenat-Thalmann, N. and Thalmann, D.“Simulation of Object Grasping and Human Skin Deformations in aGrasping Task” , Proceedings of SIGGRAPH’89, pp. 21-30, 1989.

[Holton 95] Holton, M. and Alexander, S. “Soft Cellular Modeling: A Techniquefor the Simulation of Non-rigid Materials” , Computer Graphics:Developments in Virtual Environments, R. A. Earnshaw and J. A.Vince, England, Academic Press Ltd., p. 449-460, 1995.

[Hsu 92] Hsu W., Hugues J. F., Kaufman H., Direct Manipulation of Free-Form Deformations, Proceedings Siggraph'92, pp. 177-184, 1992.

[Kajiya 89] Kajiya, J. T. and Kay, T. L. “Rendering Fur with Three DimensionalTextures” , Proceedings of SIGGRAPH’89, Computer Graphics, v.23, n. 3, pp. 271-280, 1989.

[Kalra 91] Kalra, P., Mangili , A., Magnenat-Thalmann, N. and Thalmann, D.“SMILE: A Multil ayered Facial Animation System”, Proceedings ofIFIP WG 5.10, Tokyo, Japan, pp. 189-198, 1991.

[Kalra 92] Kalra, P., Mangili , A., Magnenat-Thalmann, N. and Thalmann, D.“Simulation of Muscle Actions using Rational Free FormDeformations” , proceedings of Eurographics’92, Computer GraphicsForum, v. 2, n. 3, pp. 59-69, 1992.

[Kalra 93] Kalra, P. An Interactive Multimodal Facial Animation System, Ph.DThesis, Swiss Federal Institute of Technology, Lausanne, 1993.

[Kalra 97] Kalra, P., Moccozet, L., Werner, H. M., Boulic, R. Huang, Z., Shen,J., Magnenat-Thalmann, N. and Thalmann, D. “ InteractiveConstruction of 3D Human Shapes - Surface Manager - UserReference Manual” , report for the ESPRIT project 6709 -HUMANOID, University of Geneva, Geneva, Switzerland, 1997.

[Kurihara 93] Kurihara, T., Anjyo, K. and Thalmann, D. “Hair Animation withColli sion Detection” , Proceedings of Computer Animation’93, pp.128-138, 1993.

[Lamousin 94] Lamousin, H. J., Waggenspack, W. N. “Nurbs-based Free-FormDeformations” , IEEE Computer Graphics and Applications, v. 14, n.16, pp. 59-65, November 1994.

[Lindstrom 96] Lindstrom, P., Koller, D., Ribarsky, W., Hodges, L. F., Faust, N. andTurner, G. A. "Real-Time, Continuous Level of Detail Rendering ofHeight Fields", Computer Graphics, SIGGRAPH'96, New Orleans,ACM SIGGRAPH, 1996.

[Magnenat 88] Magnenat-Thalmann, N., Laperrière, R. and Thalmann, D. “Joint-Dependent Local Deformations for Hand Animation and ObjectGrasping” , Proceedings of Graphics Interace’88, Canadian Man-Computer Communication Society, pp. 26-33, 1988.

[Magnenat 96] Magnenat-Thalmann, N., Carion, S., Courchesne, M., Volino, P. andWu, Y. “Virtual Clothes, Hair and Skin for Beautiful Top Models” ,Proceedings of Computer Graphics International’96, pp. 132-141,1996.

[Mill er 88] Mill er, G. S. P. “From Wire-Frame to Furry Animals” , Proceedingsof Graphics Interface’88, pp. 138-146, 1988.

100

[Miralab 99] Miralab at the University of Geneva, Switzerland, WEB site athttp://miralabwww.unige.ch.

[Moccozet 97] Moccozet, L. and Magnenat-Thalmann, N. "Dirichlet Free-FormDeformations and their Application to Hand Simulation", In:Proceedings of Computer Animation'97, p. 93-102, Geneva,Switzerland, June, 1997.

[Nedel 98] Nedel, L.P. “Anatomic Modeling of Human Bodies UsingPhysically-Based Muscle Simulation” , Ph.D. Thesis, LIG –Computer Graphics Lab, EPFL – École Polytechnique Fédérale deLausanne, Lausanne, Switzerland, June 1998.

[Paill et 97] Paill et, O. and Shen, J. Body Builder User Manual, ÉcolePolytechnique Fédérale de Lausanne, Computer Graphics Lab.,internal report, 1997.

[Parke 82] Parke, F. “Parametric models for facial animation” , IEEE computerGraphics & Applications, November 1982.

[Platt 81] Platt, S. and Badler, N. “Animating facial expressions” , ComputerGraphics, v. 15, n. 3, August 1981.

[Perlin 89] Perlin, K. H. “Hypertexture” , Proceedings of SIGGRAPH’89,Computer Graphics, v. 23, n. 3, pp. 253-262, 1989.

[Press 92] Press, W. H., Teukolsky, S. A., Vetterling, W. T. and Flannery, B. P.Numerical recipes in C - The Art of Scientific Computing - SecondEdition, Cambridge University Press, 1992. 994 p.

[Rosenblum 91] Rosenblum, R. E., Carlson, W. E. and Tripp III, E. “Simulating theStructure and Dynamics of Human Hair: Modeling, rendering andAnimation” , Visualization and Computer Animation, v. 2, n. 4, pp.141-148, 1991.

[Sederberg 86] Sederberg, T. W. and Parry, S. R. “Free-Form Deformation of SolidGeometric Models” , Computer Graphics, v. 20, n. 4, p. 151-160,August 1986.

[Shen 95] Shen, J. and Thalmann, D. “ Interactive Shape Design UsingMetaballs and Splines” , In: Proceedings of Eurographics Workshopon Implicit Surfaces’95, Grenoble, France, p. 187-196, 1995.

[Shen 96] Shen, J. “Human Body Modeling and Deformations” , PhD thesis,École Polytechnique Fédérale de Lausanne, 1996. Advisor: Prof.Daniel Thalmann

[Shen 98] Shen, J., Nedel, L. P. and Thalmann, D. Body Builder Programmer’sGuide, internal report, EPFL, Lausanne, Switzerland, 1998.

[Terzopoulos 87] Terzopoulos, D., Platt, J., Barr, A. and Fleischer, K. “ElasticallyDeformable Models” , Computer Graphics, v. 21, n. 4, p. 205-214,July 1987.

[Thalmann 96] Thalmann, D., Shen, J. and Chauvineau, E. “Fast Realistic HumanBody Deformations for Animation and VR Applications” , In:Computer Graphics International Proceedings, p. 166-174, 1996.

[Thomson 64] Thomson, A. A Handbook of anatomy for art students, Doverpublications, Inc., New York, 1964.

101

[Tolo NY] Tolo, V. “Skeletal muscles, joints, and fascial structures” , In: TheJohns Hopkins Atlas of Human Functional Anatomy, edited by G. D.Zuidema, 2nd edition, The Johns Hopkins University School ofMedicine, p. 9-17, No Year.

[Uda 93] Uda, N., Kimura, F., Tsuruoka, S., Miyake, Y., Shunlin, S., Tsuda,M. “Expansion of the Solid Modeler’s Functions for Use inDesigning a Model of the Human Body” , New Advances inComputer Aided Design & Computer Graphics, InternationalAcademic Publishers, pp. 357-363, 1993.

[Watanabe 89] Watanabe, Y. and Suenaga, Y. “Drawing Human Hair Using WispModel” , Proceedings of Computer Graphics International’89, pp.691-700, 1989.

[Wernecke 94] Wernecke, J. The Inventor Mentor - Programming Object-Oriented3D Graphics with Open Inventor - Release 2, Addison-WesleyPublishing Company, 1994.

102

103

MODULE 4:Surgery Simulation

1. Introduction

Generally, the goal of computer-based surgery simulation is to createenvironments that support medical education and training by allowing the user tovisualize and rehearse clinical, surgical procedures. In this context, surgery simulationsystems can ideally provide an efficient, safe, realistic, and relatively economicalmethod for training physicians in a variety of surgical tasks. The emphasis in surgerysimulation is usually placed on a user’s real-time interaction with medical instruments,surgery techniques, and models that represent various anatomical structures andphysiological processes. This module presents a survey on surgery simulation, whichincludes a classification of surgery simulation systems, the technology needed to buildand use these systems, and two special sections focusing on minimally invasivesurgeries and the use of surgery simulation systems for education and training.

2. OverviewSurgery simulations can give insight into the potential consequences of a

surgical technique before performing an actual operation. For instance, a tendontransfer can be simulated by graphically moving a tendon attachment from onelocation on the limb to another. The computer model then determines how thissimulated surgery affects the muscle forces and other biomechanical variables.

The mere rendering of a 3D database is of no use in a surgical simulator. Thecore task is the manipulation of a database, which contains the tissues, organs andorgan-systems of the human body. This translates into the abilit y to simulate thebehavior of f lexible as well as rigid tissues and to maneuver them surgically in real-time. Computer modeling can aid in navigation by augmenting the limited endoscopeview with a more global view of the patient’s anatomy and can provide guidance bypreventing the surgical instrument from moving into pre-defined sensitive regions,which is very useful in surgery planning [Gibson 97]. Also, the possibilit y ofreproducing bleeding and other processes introduces other advantages in thesimulation of surgeries.

Besides the importance of surgery simulators systems for surgery planning andevaluation, another important application is medical education and training. In thiskind of application, despite the lack of f idelity with the real organs, the virtual onesbehave as real in response to surgeon's actions. So students can see the effect of organsinteraction and, if haptic displays are being used, they can feel the forces in theirfingers [Dionisio 97, Salisbury 97, Srinivasan 97]. Some of these systems try toprovide for a great variety of cases and possible emergency situations, giving asystematic solution for training and evaluating student performance. In this manner,

104

the surgeon can practice a specific procedure many times, without patient pain or lostand with a reduced cost, considering the low availabilit y and high cost of cadaver andanimal specimens [El-Khalili 98, Gibson 97]. Nowadays, the test and evaluation ofprotocols is being carried on to determine how the simulators could contribute forsurgical education.

Although surgical simulators systems have several applications, to develop acomplete system is a huge task and a multidisciplinary task. Research in supportingareas can be, for example, the study of deformable models based on finite elementmethods [Cotin 96, Koch 96, Bro-Nielsen 97], the study of structure of tissuealteration, by cuts or dissection. Also interaction with haptic devices at one or multiplepoints [Basdogan 98] is an important issue as well as the study of physiologicalmodels [Basdogan 99].

The conceptual components of a surgery simulation system are differentdepending on the goals and architecture of each application. The main components arebased on techniques from three big sub-areas [Bro-Nielsen 98]: computer graphics,haptic interfaces and physically based modeling. The first one is needed to renderrealistic views of the surgery scene, while the second deals with the representation ofinstruments and tools needed to provide a physical sensation of touching and sensingobjects in the virtual scene. The last one ensures that the virtual scene reflects thebehavior of the physical reality, by modeling the visco-elastic deformation of humanskin, the fluid flow of blood from a wound etc.

Gross [Gross 98] states that the conceptual components of an advanced surgerysimulation environment range from the acquisition of highly accurate images throughmedical scanners, to the generation of photorealistic images of the patient. Amongthese components he identifies: pre-processing steps to extract anatomic substructures and create

geometric models of the patient; modeler, that allows the surgeon to modify the geometry and topology ofindividual parts of the model; force feedback, that has to be computed and interpolated to meet highupdate rates; real-time processing of tissue volume forces, deformationfields and colli sion detection.

Lin [Lin 99], in turn, defined four major components: geometric modeling,object deformation, objects cutting and texture mapping. All of them wereimplemented as different modules in one simulation system.

3. Classification

Since in general all surgery simulator systems are tailored to a specificapplication, with well -defined objectives, some authors classify, or group, them intosubclasses.

Cotin et al. [Cotin 96a] distinguish three main classes of surgical simulators,according to the modeling techniques employed and level of simulation achieved withthe system.. The first class (C1) corresponds to simulators that do not deal withdeformable models. They restrict the surgeon to navigate in a virtual environmentreconstructed from medical images. Usually, this type of simulator is used forendoscopic applications, as the interactive virtual colonoscopy method presented byHong et al. [Hong 97]. These simulators are very limited, since the surgeon can not

105

interact with the model as in real situations. A higher level simulation is reached bythe use of deformable models to represent the shape of the target organ in the secondclass (C2). In this case the model may be physically based, but remains surface-based.In that sense it only represents the external boundary of the organ. Such simulatorsallow surgery training, but the limitations are due to the surface-based nature of themodel. Finally, in the third class (C3), a more sophisticated simulator would includeall the previous possibiliti es but with volumetric modeling of the organs. It would thenbe possible to navigate within the virtual environment, to interact with the model, andeven to cut parts of the virtual body.

Satava [Delingette 98] uses a second classification to identify differentgenerations of simulators. The first generation of simulators applied the concept ofnavigation and immersion to three-dimensional anatomical datasets, considering onlythe geometric nature of the human body. Their functions are limited, but haveapplications in the field of education or training. The second generation aims atmodeling the physical interaction between each anatomical structure. For osseousstructures, those simulators model the coupling between kinematics constraints andmuscle deformation. For soft tissue, it is necessary to model their deformabilit y underthe influence of neighboring structures or surgical instruments. And the thirdgeneration of simulators takes into account the functional nature of human organs.Figure M4.1 shows how the different levels of simulation (anatomy, physical orphysiology) interact with each other.

Pathology

Brain

ShapeMorphology

SurfaceVolume

Blood Cells

Forces

Kinematics

DeformationTemperature

Liver Cells

Liquid/Solid

Physiology

Anatom

yP

hysicsFi

rst G

ener

atio

nT

hird

Gen

erat

ion

Seco

nd G

ener

atio

n

Figure M4.1: The three generations of surgery simulation systems asproposed by Satava [Delingette 98].

According to these classifications, it’s possible to separate the surgicalsimulator systems basically in three categories:! systems that only permits the navigation in the volume;! systems that work with the deformation of the organs;! systems that allow the treatment and implementation of physiological

models.

106

The first category embodies those systems, which simulates endoscopicprocedures, and are described in Section 3.1. The second one involves theimplementation of models of deformation to simulate the physic interaction with theorgans. The most used techniques are described in Section 3.2. Finally, the third onesconsider the functional nature of the organs that is the physiologic models needed toassess the functional consequences of the injury and the simulated surgicalintervention. Some of these models are circulatory models and tissue bleeding models[Basdogan 98a]. The bleeding occurs principally when a tissue cutting is done. Thesetwo phenomena, physiological and physical, are described in Sections 3.4 and 3.3,respectively.

3.1. Navigation Systems

Generally, the navigation systems are used to simulate endoscopy applications,which are the base of minimally invasive surgeries. Virtual endoscopy is a newmethod of diagnosis using computer processing of three-dimensional image datasets(such as CT or MRI scans) to provide simulated visualizations of patient specificorgans similar or equivalent to those produced by standard endoscopic procedures.These procedures are invasive and often uncomfortable for patients. Furthermore,endoscopes display only the inner surface of hollow organs and yield no informationabout the anatomy within or beyond the wall . Virtual endoscopic visualization avoidsthe risks associated with real endoscopy, and when used prior to performing an actualendoscopic exam can minimize procedural diff iculties and decrease the rate ofmorbidity, especially for endoscopy specialist in training. Additionally, there are manybody regions not accessible to or compatible with real endoscopy that can be exploredwith virtual endoscopy. Eventually, when optimized virtual endoscopy may replacemany forms of real endoscopy [Robb 97, Jolesz 99].

The first step in an endoscopy simulation is the acquisition of three-dimensional images from a scanner (e.g., spiral CT, MRI). After that, invariably, somepreliminary processing on this data, as interpolation and registration, is required toproperly prepare it for modeling. Segmentation is also necessary to reduce the datasetto the desired specific anatomic structure(s). Many approaches to model creation havebeen proposed, but usually the surface of single anatomic objects are extracted aftersegmentation. The isolated surface may then be converted to a geometricrepresentation, by transforming surface coordinates to a meshwork of polygons thatmay have appearance modifying information added to it, such as color, lighting,texture, etc. This processed dataset comprises a model, which then may be renderedfor visualization using a surface-rendering algorithm. The endoscopic displayprocedure is then simulated. In this final step an algorithm to avoid colli sions isusually implemented [Robb 97].

The type of instruments used for interaction with the patient and the influencethese instruments have on the patient also modifies the MIS simulation system.Minimally invasive instruments come in increasingly different shapes and materialsbased on the requirements of the medical procedure and the anatomy involved in it.Some instruments are just rigid scopes such as those used in bronchoscopeexaminations, whereas others are flexible, and multi -layered such as catheters forcoronary angioplasty [Bro-Nielsen 97a].

An interactive navigation in the human colon, which uses a physically basedcamera control model and a hardware-assisted visibilit y algorithm, was developed tosimulate virtual colonoscopy [Hong 97]. In general, this procedure consists of three

107

steps: first the patient’s colon is cleansed and inflated with air in a way similar to thatof optical colonoscopy. Second, while the patient is holding his or her breath, a helicalCT scan of the patient’s abdomen is taken, capturing a sequence of two-dimensionalslices which covers the entire range of the colon, producing several hundred slices,which are subsequently reconstructed into a three-dimensional volume. Finally, thecolonic surface is extracted, and the physician virtually navigates inside the colon toexamine the surface for possible polyps. The camera control supplies a convenient andintuitive mechanism for examining the colonic surface while avoiding colli sions. Z-buffer-assisted visibilit y algorithm culls invisible regions based on their visibilit ythrough a chain of portals thus providing interactive rendering speed.

To the benefit of the patients, that have less recovery time and a cost reduction,MIS and endoscopic procedures are getting more and more common in surgicalpractice in the last few years. Specifically the area of endoscopy simulators fortraining surgeons is also receiving enormous attention, mainly because the mechanicalinteraction problems are relatively simple, since the endoscopic devices themselvesenormously reduce the richness of reality that need to be simulated. But evenendoscopy simulators suffer from poor realism in their visualizations, and in thesimulation of behavior when interacting with the visualized structures. Although thereare many endoscopy simulators with realistic feel, since real endoscopes give onlycrude tactile feedback, none of them looks realistic, especially when the surgeoninteracts with the visualized organs and other tissues [Waterworth 99].

Even though virtual endoscopy is in the initial evaluation stages in clinicalpractice, descriptions of methods and preliminary results are still i ncreasing. However,the visual fidelity of current generation virtual endoscopy images is not yet at the levelof diagnostic accuracy suitable for regular clinical use. There are a number oftechnical problems that have to be solved to make virtual endoscopy a clinical toolwith suff icient scientific validity and ease of use to be employed on a routine basis,including: three-dimensional image resolution, accurate surface rendering, automaticsegmentation, robust registration and appropriate preparation [Robb 97].

3.2. Modeling Techniques

In a surgery simulation system the surgeon usually needs to visualize themodification of the geometry and the topology of individual parts of the derived modelwhile simulating cuts, bone reposition, etc. So, based on the need for shape changes inmedical simulation, Bro-Nielsen [Bro-Nielsen 97a] affirm that the scope of simulationtechniques for anatomic models is very broad, ranging over the following ones:" Static computer graphics models e.g. used in endoscopic examinations." Animated physically based deformations." Deformable mass-spring models." Advanced fast Finite Element Models (FEM).

These techniques are the most frequently implemented in the state-of-artsystems, although they are very different both in terms of mathematical theory, andwith respect to the computational complexity.

Static computer graphics models used in endoscopic examinations (section 3.1)are the simplest one, and usually don’ t allow, and even don’ t need, a tissuedeformation. In this case, it’s only necessary to model the desired organs from thescanned set of images, and then navigate into the reconstructed structure.

108

In computer animation, physically based modeling is a common approach tosimulate natural phenomenon. It combines physics and engineering disciplines withgeometric modeling in order to produce realistic-looking behaviors. Physically basedsimulation necessarily involves geometric and temporal discretizations of thetheoretical relations describing the physical phenomena. Among physically basedmodeling approaches that may be applied for soft tissue simulation it’s possible topoint out: mass-spring models; finite element models; biomechanical modeling, andparticle systems [Nedel 98].

A great interest has been given to mass-spring models due to their simplicity ofimplementation and their relatively low computational complexity. A mass-springmodel consists of a number of nodes, connected by springs. The mass of the modeledobject is concentrated on the nodes, thus the name. But, despite these approachesallow interactive rates, they exhibit a lack of realism since they represent a solid as adiscrete set of masses. Another problem appears when the tissue model is submitted tocutting or tearing operations and the topology changes. By construction, mass-springmodels can be easily modified to allow topology changes; however, they exhibit non-realistic behavior during deformation or cutting operation. Even so, many authors havealready implemented these models in surgery simulation systems [Bro-Nielsen 98a,Cotin 98, Lin 99].

FEM are widely regarded as the most appropriate alternative for thedeformation of solid volumetric objects in surgery simulation systems, despite it isdiff icult to implement and has a larger computing time. It’s important to consider thatlinear elasticity is often used as a way to obtain a good approximation of the behaviorof a deformable body. The stress-strain relation provided by the theory of elasticityallows the modeling of various physical behaviors [Cotin 96]. Due to the hardprocessing, and since there is a tendency to use FEM to solve the equations governingthe deformable models [Cotin 96a], several optimizations on FEM were alreadyproposed to reach real-time results. Some of them will be briefly presented now.

Bro-Nielsen and Cotin [Bro-Nielsen 96] achieve a computing time reductionand reach a real-time surgery simulation using fast FEM of linear elasticity, using anew method called condensation. Through it, the computation time required for thedeformation of a volumetric model can be reduced to the computation time of a modelonly involving the surface nodes of the mesh. Berkley et al [Berkley 99] developed the“banded matrix” technique as a means for achieving real-time deformation and forcefeedback. This method is applicable to a wide range of surgery simulations, and isaimed at re-arranging the finite element equations so that the majority of calculationscan be performed before the simulation begins. It is important to point out that thebanded matrix technique is appropriate when there are only a limited number ofcontact scenarios.

Considering that a simulation’s emphasis may be on deformation visualization,haptic interaction at one or multiple points, stress development within the tissue,alteration of tissue structures (i.e. cutting or dissection), or all of the above, Berkley etal [Berkley 99] identifies several modeling scenarios, described in Table M4.1. Thesescenarios encompass the majority of surgical tasks. The banded matrix technique, forexample, encompasses the scenario 1.

109

Table M4.1 – Modeling scenar ios [Berkley 99]

S

cena

rio Mesh Type Contact Type Surgical Tool

ModelingExample

Static mesh (meshnot changing, i. e.no cutting)

Single point or limitedcontact

Rigid Dermatologicalexcision closure

Static mesh Unlimited contact Rigid Orthoscopic palpationDynamicallychanging mesh(allows for cutting)

Single point or limitedcontact

Rigid Endoscopic dissection

Dynamicallychanging mesh

Unlimited contact Rigid Open procedures

-8“All of the abovemeshes”

“All of the aboveTypes”

Deformable Catheterizationprocedures

A prototype system for surgical planning and prediction of human facial shapeafter craniofacial and maxill ofacial surgery for patients with facial deformities wasalso developed using non-linear FEM [Koch 96]. The system combines, unifies, andextends various methods from geometric modeling, finite element analysis, and imageprocessing to render highly realistic three-dimensional images of the post surgicalsituation. In this research, they aimed at integrate both the current surface and volumespring model and allow incorporation of more anatomic features.

Another technique of deformable model, inserted in the geometric modelscategory instead of in the physically-based, that was used as referential in one surgicalsimulation system is Free-Form Deformation (FFD) [Sederberg 86, Nedel 98]. Coveret al. [Cover 93] developed a methodology for creating deformable, graphical three-dimensional models that improves on earlier work in deformable models and activecontours. Its main characteristic is the combination of elements of both free-formdeformations and energy-minimizing surfaces. By combining these elements in onemodel, they have produced a prototype system that lets the user interactivelymanipulate models of the gall bladder and surrounding organs. A different approachwas also used by Yagel et al. [Yagel 96] in their Endoscopic Sinus Surgerysimulation. They use a real-time volume rendering technique, and for spacedeformation they applies the required shape transformation not to the objects in thescene but rather to the rendering agents.

Due to the very complex visco-elastic behavior of human tissue, Bro-Nielsen[Bro-Nielsen 97a] concludes that none of these mathematical models can becharacterized as being really accurate. Consequently, it’s not important to performvery precise physical measurements of human tissue parameters, since currentmathematical models cannot use or benefit from these parameters. However,parameters are typically determined by comparing videotapes of real surgery with thesimulation results, and making adjustments to obtain visually close results. In somecases this is combined with simple tissue experiments to obtain rough initial estimatesof the model parameters. Naturally, as computers become more powerful, moreadvanced and accurate mathematical models will be used. At that point, precisemeasurements of tissue parameters become an important issue.

110

3.3. Tissue Cutt ing

The simulation of tissue cutting is a very important component of a surgicalsimulator, since surgeons use a number of instruments to perform incision anddissection of tissues during surgeries. The impact of this operation in terms of tissuemodeling is considerable, because it implies that the geometric representation of tissuemust change its topology over time. The cost of such a topological change dependslargely on the chosen representation [Basdogan 99, Delingette 98].

Tissue cutting includes tracking of the tip of the virtual cutting device, with forexample a haptic feedback device (see section 4), and accurate re-generation and re-connection of the new polygons and vertices. In fact, the behavioral model of thetissue must be adapted at parts where cutting occurs. Little is know about thestress/strain relationship occurring during and after cutting, so, usually, the assumptionis that the physical properties of tissue are only modified locally. However, in practice,cutting can greatly modify the boundary conditions between tissue and thesurroundings organs, which entails considerable change in terms of deformabilit y[Delingette 98, Lin 99].

Basdogan [Basdogan 99] aff irms that the cutting through soft tissue generatesan infinitesimally thin slit until the sides of the surface are separated from each other.So, the simulation of an incision through tissue surface is modeled in three steps: first,the colli sions between the instrument and the tissue surface are detected as thesimulated cutting tool passes through. Then, the vertices along the cutting path areduplicated. Finally, a simple elastic tissue model is used to separate the vertices fromeach other to reveal the cut.

In such case, the two major challenges in simulating three-dimensional tissuecutting are:# developing physically-based tissue models that can estimate the interaction

forces and the new positions of the nodes that are separated in real-time,and# accessing and updating the haptic and graphics database in real-time tohandle topological changes in geometry as a result of the cut [Basdogan99].

Some authors try to solve these problems using alternative approaches. Oneidea proposed is an interactive two-dimensional finite-element template for cuttingelastic virtual surfaces. A template of nodes moves as the cutting tool moves over aflat two-dimensional tissue surface. Another alternative is the development of a tissuecutting technique implemented on three-dimensional surfaces that require incrementalupdate of vertices, connectivity arrays, and texture maps. This approach of mappingtextures onto the regions where virtual tissue surface spreads apart to reveal the cutprovides a visually realistic display [Basdogan 99].

An approach of pre-generate polygons by modeling objects as tetrahedronswas implemented by Lin [Lin 99]. In this case, a polygon is not rendered until itbecomes visible, and the connections between vertices are also defined. The cuttingresults in exposure of polygons and removal of connections. In this simulation, thefirst step is to detect the intersection between the cutting tool, modeled as a two-dimensional polygon, and the object. Then, the intersection between the cutting deviceand the triangle faces of each tetrahedron is evaluated. Each intersection generates twonew visible triangles and the connections between them and any neighbor tetrahedron

111

that shares the same triangle are adjusted (removed). The new visible triangles areadded to the list for display and the new visible vertices are added to the list fordeformation calculation. The object volume is preserved but the cutting will notcorrectly address the interior tetrahedrons after the object is deformed. The solution isto keep a copy of the original geometry of the tetrahedral model and transform thecutting device coordinates to match this copy, as shown in Figure M4.2. The cutting isthen performed on the original version of the object.

Figure M4.2: Mechanism to address cor rect volumetr ic elements afterthe object is cut and deformed [L in 99].

Delp et al. [Delp 97] implemented boolean subtraction and booleanintersection algorithms to perform tissue cutting. These algorithms operate on two-closed polyhedral of arbitrary shape, and begin by calculating the intersections of eachpolygon in the first polyhedron (P1) with the polygons in the second polyhedron (P2).In the general case, a polygon either lies completely inside or outside the otherpolyhedron or it intersects it and is divided into two or more pieces. The next step inthe process is to build the output polyhedron by collecting the polygons and polygonpieces computed in the first step. The specific set of polygons collected depends onwhich boolean operation is being performed. The subtraction algorithm, whichcomputes P1-P2, collects all of the polygons and polygons pieces in P1 that areoutside P2, and all of the polygons in P2 that are inside P1. This second set ofpolygons defines the new “cut” surface of P1. For boolean intersection, the algorithmcollects all of the polygons and pieces from P1 that are inside P2, and vice versa. Thespecific method of creating the cutting object, to implement tissue cutting, depends onthe surgical instrument being used, but the basic process is to use the path of thecutting blade(s) to build the polyhedron. As the instrument is moved through thetissue, the coordinates of the blades are used as vertices of the cutting object.

Auxili ary surfaces were also proposed to simulate tissue cutting. These kind ofsurfaces, ill ustrated in Figure M4.3, provide convenient means for mapping three-dimensional surface geometry onto a local two-dimensional rectangular mesh toreduce the dimensionality of the physics-based problem in consideration (cutting inthis case) by one, so that it can be solved eff iciently in real-time. The solution then canbe mapped back to the original three-dimensional surface to display the results[Basdogan 99].

3.4. Bleeding

During the execution of tissue cutting, discussed above, bleeding may occurand blood flows over the tissue surfaces. So, to obtain a realistic simulation, someresearchers have already studied and developed algorithms to display bleeding invirtual environments. However, accurate simulation of bleeding is a challenging

112

problem, because of the complexities of the circulatory system and the physics ofviscous fluid flow [Basdogan 99, Basdogan 98b].

Figure M4.3: Auxili ary sur face to simulate tissue cutting [Basdogan 99]

Basdogan [Basdogan 98b] presented one blood flow model visually realisticwith real-time computational performance, based on a circulatory model. To bleedingdisplay he developed two separate flow algorithms. To represent blood that flows overa surface and pools at the bottom of a depression, a surface flow algorithm wascreated. This method is based on a simpli fied form of the Navier-Stokes equationsgoverning viscous fluid flow. To represent pulsatile bleeding (blood moving throughthe air), a particle flow model was designed and implemented. This algorithmcalculates how many blood droplets (modeled as small spheres) are needed to model aparticular instance of bleeding, and assigns a velocity to each droplet in a gravitationalfield so that droplets flow out of the injured region at the proper rate. This flow modelis used to graphically represent bleeding from severed blood vessels and highlyvasculated tissues. The display of the blood involves groups of particles that flow outof the vessel with a projectile motion based on: the speed of particles, direction offlow, number of particles, time the particle was created and li fe span of particle. Thesetwo flow models are connected by a set of subroutines that are called whenever atissue is cut by scalpel. These subroutines determine which bleeding model isappropriate for representing the blood flow in this bleeding instance, depending on thetype of tissue that is cut [Basdogan 98a].

The integration of the partial differential equations, that is the Navier-Stokesequations, over time is computationally very expensive. Meanwhile, a less expensivetechnique that is based on the solution of wave equations was already developed[Basdogan 99]. Using this technique, it’s possible to represent the fluid surface as aflat two-dimensional surface grid and solve the wave equations using a finitedifference scheme to update the fluid depth, in the X and Y directions separately, ateach node of the grid. The resulting depth values are averaged and are used in thegraphical display as well as in subsequent numerical iterations.

Bro-Nielsen et al. [Bro-Nielsen 98a] implemented a bleeding algorithm thatuses a diffusion-style algorithm to model the flow of blood on the surface of polygonalmodels. For performance and complexity reasons the blood is stored only at thevertices of the polygonal surface. This limits the precision of the blood movement tothe distance between polygon vertices, but allows the cutting algorithm to be used onthe bleeding surface as well . The algorithm also modifies the polygonal surface usinga bump-mapping approach to provide the ill usion of blood lying on top of thepolygonal model. At rendering time a vertex with blood on it is moved in the normaldirection of the surface. The distance is determined based on the amount of blood onthe vertex.

113

4. Technologies

With the popularization of the VR devices, these new technologies began to beused combined with medical data visualization, especially in surgery simulationsystems. VR has enormous educational potential and will li kely become thefoundation of next-generation simulations used by medical professionals at all l evels[Satava 93]. VR-based visualization and training environments could providebeginning students with engaging and immersive 3-D views of anatomic structuresand their spatial relationships. Surgical residents could learn new techniques orrehearse procedures before attempting the actual operations. An unlimited pool ofvirtual patients, combined with existing curricular opportunities, would increase thedepth and breadth of clinical exposure, ensure the uniformity of training experiences,enhance skill acquisition, and ultimately improve the quality of health care delivery.

VR could also be used to develop a unifying interface for accessing supportingmultimedia and telecommunications resources. In education this is important becauseno single technology can provide a complete curriculum for our students. The wealthof extant medical teaching materials including books, atlases, image banks, databases,animations, and documentary videos should not be discarded as we move towardsadopting more advanced systems. Rather, these multimedia resources should be madeavailable within VR to add depth and clarity to the learning process as well as supportand enrich the virtual experience.

Considering the specific case of the application of VR technology in the fieldof surgery simulation, there are a growing number of different devices commerciallyavailable. In addition to some multi -purpose devices, li ke head-mounted displays,shutter glasses, spaceballs, datagloves, and so on, also useful for medical applications,we can consider the use of specific 3-D manipulators with digital sensors dedicated,for example, to endoscopy (Figure M4.4a) and laparoscopy (Figure M4.4b). Morerealistic devices actuate like a kind of mock-up, as shown in Figure M4.5 and M4.6,giving yet an extra-sense of reality to the user.

(a) (b)

Figure M4.4: Three-dimensional manipulators

114

Figure M4.5: Mock-up for endoscopy simulation [Sabbatini 99]

Figure M4.6: Knee ar throscopy mock-up [Sabbatini 97]

Jointly with the VR devices, the use of haptic and force feedback devices meritour attention. In short, a haptic device provides a touch feedback to the user. Thegrowing of the use of haptic devices is so big, that a new area, analogous to ComputerGraphics and called Computer Haptics, has appeared. It is possible to say now, thatthis area could be defined as a discipline concerned with the techniques and processesassociated to the generation and display of haptic stimuli to the human user. Withthese new technologies, the user could feel in his or her fingers the physics propertiesof an object that is being explored or touched. The haptic devices are very useful inmany medical applications, but we can point out its use in surgery simulation systems[Srinivasan 97, Rosenblum 97].

The sensing of forces (haptic feedback) and the sensation of textures are alsoimportant, and the feeling of tissue squeezing could greatly enhance the quality ofsurgical simulations. Interacting with virtual environments by a haptic interface, thehuman user conveys desired motor actions by physically manipulating the interface,which, in turn, displays tactile sensory information to the user by appropriatelystimulating his or her tactile and kinesthetic sensory systems [Srinivasan 97]. So,haptic device serves at least two purposes in a surgical simulator: kinesthetic andcognitive. First, it provides the movement sensation to the user and therefore it greatly

115

enhances the surgical performance. Second, it is used to distinguish between tissues bytesting their mechanical properties [Delingette 98, Salisbury 97].

In the specific case of surgery simulation, the sensation of forces is coupledwith the geometric and deformable nature of the virtual model. The flow ofinformation will t hen form a closed loop: the model deforms according to themovement of the force feedback device, this deformation allows the computation ofthe contact force and finally, the loop is closed by generating the force of contactthrough a mechanical transmission [Cotin 96a]. Besides, the haptic interface is alsoused to represent the instruments and tools that the surgeon uses to work. By trackingthe position of these tools and feeling their state, the computer is able to determine theactions of the surgeon and to provide them as input to the simulation system. Inreaction to these inputs, a haptic interface can provide the surgeon with a physicalsensation of touching and sensing objects in the virtual scene using force-feedbacktechniques. The haptic interface thus closes the loop between action and reaction byproviding the tactile ill usion of reality. However, despite these advantages, whenadding force feedback in a simulation the diff iculty related to the real-time constraintimposed by such a system appears. To remain realistic, the forces must be computed ata very high frequency [Cotin 96, Bro-Nielsen 98].

A device that is used to give force feedback in VR applications and iscommercial available is the PHANToM, from SensAble Technologies, Incorporated[Salisbury 97, SensAble 99] (see Figure M4.7). Their haptic interface models providedifferent degrees of freedom in a limited and variable three-dimensional space ofwork. The basic PHANToM design was influenced by three observations: the first oneestablishes the type of haptic stimulation that the device would provide, the seconddetermines the number of actuators that the device would require and the thirdestablishes the volume or workspace that the device would possess [Massie 98]. Somesurgery simulation systems have already adopted this device to provide forcereflectance and feedback for haptic interaction [Gibson 97, Berkley 99].

Figure M4.7: Phantom force-feedback device [Sabbatini 97]

However, some authors need a specific hardware to provide more realisticprocedures. For example, the arthroscopic surgery in addition to manipulation of thecamera and instruments has a key part that is the manipulation of the limb itself. Thismanipulation is necessary to open up the joint for observation and so the eye/handcoordination required for minimally invasive surgery is compounded by additional

116

proprioceptive and visual cues. So, for a better virtual reality simulation Dr.Trowbridge and Dr. Hollands used an artificial knee, similar to the kinds already beingused for arthroscopic training, to provide realistic forces when manipulated[Trowbridge 98], as the one previously shown in Figure M4.6.

Bro-Nielsen et al [Bro-Nielsen 98a] configured a simulation test-bed to looklike a real patient lying on a surgery table. The abdomen of the patient is exposed,while a blue surgical drap covers the rest of the body. To accomplish this, thecomputer monitor is mounted horizontally into a special stand that has the head andlegs of a mannequin attached to it. The computer monitor serves as the exposedabdomen and it is camouflaged using the blue surgical drap. Yagel et al [Yagel 96],for his turn, use a physical interface in their endoscopic sinus surgery simulation: amock-up patient head that will house the electromechanical mechanisms providing theforce reflection to the user. While the user moves the instruments, sensors monitor theposition and orientation of the tool within a given three-dimensional workspace.

As exposed before, many authors are worried in relation to the force feedback,and some of them think that it is indispensable in surgery simulation systems. Gross[Gross 98] aff irms that the design of a sophisticated natural human computer interfaceis certainly one of the keys for the success of simulations, cause the ill usion of a fullmedical working environment can only be created with the help of advancedhardware. Much more important and fundamental for any surgeon than the displayitself is the provision of highly accurate tactile and force feedback information.Therefore, it is necessary highly accurate haptic interfaces, which capture theresponses of the human tissue to mechanical stimuli . Nowadays there were manyendoscopic simulators with some force feedback, and although it was not convincing,as real tissue (from doctors point of view), it apparently helps in training (fromengineers opinion).

So, this is a big challenge. Despite several haptic interfaces available inresearch labs or on the market and enable first experimental steps into the rightdirection, we are still far from convenient solutions. In other words, everybody istrying to figure out how to do tactile tissue simulation, but a convincingimplementation was not seen yet. Recently, an interesting alternative that started toappear is the use of sound feedback [Gross 98, Waterworth 98].

5. Minimally Invasive Surgery Simulation (MIS)

The principle of minimal access to the area of intervention leads to a numberof disadvantages for the surgeon. The direct manual operation at free visible organs isno longer possible. The surgeon’s sense of touch and dexterity are highly reduced dueto the use of long and narrow endoscopic instruments. As a matter of fact, endoscopicsurgery requires the surgeons to be familiar with a new form of hand-eye coordinationand skill ed in the manipulation of instruments by looking at the endoscopic imageprojected on a video monitor. These skill s are not intuitive, and the optimization of thesurgical procedures requires a considerable amount of training and experimentationbefore they can actually be applied in real operations. The realistic graphical modelingof the area of intervention allows defining a virtual environment in which surgicalprocedures can be learned and practiced in an eff icient, safe and realistic way. Asurgeon can repeat and practice the operation as many times as necessary. Mistakescan be eliminated until the gesture becomes eff icient and accurate, thereby assuringthe high safety and quality requirements.

117

In neurosurgical operations, endoscopy is routinely performed either as theonly optical instrument during the operation or as an adjunct to the operatingmicroscope (endoscopy assisted microsurgery). In this case, an anatomical model,consisting of a plastinated cadaver head with injection of dye into the blood vesselswas used. Surgical approaches were dissected prior to the plastinating process.Different surgical approaches were performed with rigid endoscopes and documentedon videotape. The model was scanned in a spiral CT scanner and the data is send to asoftware package that provides fast volume visualization and modeling of the opticaland mechanical properties of different endoscopes used in neurosurgery. After,animation sequences according to the videotaped endoscopic approaches wereinteractively determined and rendered off-line. For each approach and endoscope, thetrue and simulated video sequences were compared using parallel and overlay displaytechniques. Through this study is possible to compare real and simulated views[Darabi 97].

As another example of minimally invasive surgery simulation, Yagel et al[Yagel 96] presented an Endoscopic Sinus Surgery (ESS) simulation. ESS is currentlythe procedure of choice for the treatment of medically resistant, recurrent acute andchronic sinusitis. The technique consist of visualizing landmark structures within thenasal cavity and sinuses, excising and “biting” out diseased tissue, probing andsuctioning under direct visualization through the endoscope or via a video monitorwith attached endoscopic camera. The ESS simulator provides direct interaction with avolumetric model of the anatomical region while delivering haptic feedback (forcereflection) to the user. So, the ESS system requires the accurate representation ofanatomical information as well as force data.

Approaches to minimally invasive surgery simulation are quite specificbecause of model characteristics and modeling constraints. Some other examples areshown in the following images (Figure M4.8, M4.9, M4.10, M4.11 and M4.12).

Figure M4.8: Development of a Vir tual Reali ty abdominal surgerytraining system. Simulation techniques allow the modeling of " vir tualtissue" based on a data-model which reflects the physicalcharacteristics li ke mass, stiffness and damping of real tissue. Acolli sion test algor ithm detects contact between surgical instrumentsand the vir tual organs. As a by-product, contact forces between thetissue and the instrument end-effector is calculated which can be usedto dr ive a force-reflecting surgeon inter face.

118

Figure M4.9: Model of Human Anatomy. Such models are used as abasis for a surgery trainer . The organs are modeled as B-Splinesur faces.

Figure M4.10: Endoscopic Surgery Training, Overview. The KISMETimage shows an " outside" view of the surgery trainer scenar io withsimulated camera view (upper left viewport). KISMET allows as wellfor model based camera tracking. The camera viewing area isdisplayed as a pyramidal shape.

Figure M4.11: Colecystectomy Training, Endo-View

119

Figure M4.12: Tumor/Cyst Detection. The current focus of theresearch is to develop a vir tual reali ty simulation that will give theuser the sensation of feeling an object beneath the sur face of another.In order to do this, studies have been performed to evaluate theimpact of model complexity on simulation speed. Experimental studieswere done on a hard-within-a-soft rubber ball model to determine theshape of a realistic force/deflection curve for this scenar io. Thesimulation under development will display a muscle for the user totouch and evaluate. We are working to allow the user to feel a ``hardspot'' under the sur face, representing a tumor or cyst not visible to theeye.

6. Using Vir tual Humans and Surgery Simulation Systems inMedical Education

Recently, traditional teaching method complementation became necessary, dueto the world “ information explosion” alli ed, among other factors, to the lack of time ofteachers and students and the few resources and specialists available. So, the use ofcomputers in medical education started to grow up, especially with the cost reductionof hardware and software and with the widespread use of the Internet. The use ofcomputers helps the direct and individualized study, giving a bigger motivation, andproviding interaction with immediate feedback and multimedia use.

One of the first systems that allowed clinical simulation by computer was theCASE (Computer-Aided Simulation of the clinical Encounter). Developed in 1970(US), in CASE a computer simulated a patient, and a student acted as a physician,interacting with the program through commands and questions posed in naturallanguage. In the early 80’s several clinical simulation systems for microcomputersstarted to appear, and with software and hardware advance of the 90’s, surgerysimulators systems started to be developed. Multimedia and virtual reality devices andthe arising of haptic displays [Dionisio 97, Salisbury 97, Srinivasan 97] have givenimpulse to the development of simulation systems, since they allow datarepresentation by images and sounds, and immersive data interaction, increasing thesimulation “ realism” level [Jesus 99]. Moreover, the digital image data set of twohumans available through the Visible Human Project [National 99] provides thepossibilit y of developing a complete virtual human.

A usual analogy is made with flight simulators, where trainee pilots gain manyhours of experience before moving on to practice in a real cockpit. The success offlight simulators provides a convincing example of the importance of simulation

120

technology when applied to the teaching of abiliti es, which have an impact on humanlives. It also provides important examples of both the value of early simplisticsimulators and the long-term potential of simulation techniques when applied to afield. Approaches to learning to navigate within a human body have benefited fromtechniques developed to train pilots to fly advanced commercial and military aircraft.There are quite strong similarities between the two application fields since bothcombine the need for great manual dexterity in a three-dimensional environment withli fe-criti cal information access and decision making.

However, just as early flight simulators were met with considerableskepticism, early surgery simulators have often been criti cized for being too simplisticand not realistic. But this is changing. The technology for simulation of surgicalprocedures has improved dramatically in recent years. By using new advances inComputer Graphics and physical modeling techniques, as well as increased realismcombined with enhanced feedback, and at the same time concentrating on specificmedical procedures, very realistic simulation systems have been created. Especially inthe field of minimally invasive procedures, both academic institutions and commercialcompanies have presented a range of different simulation systems. This occursbecause minimally invasive surgery and endoscopic procedures brings many benefitsfor the patients, but, on the other hand, the indirect access to the operation area causesa lot of disadvantages for the surgeon: restricted vision, diff icult hand-eyecoordination and handling of instruments with limited mobilit y. Therefore the surgeonrequires a lot of training and experience to execute an operation successfully andsafely. Then, recently, simulation systems have demonstrated the feasibilit y of thetechnology, and commercial systems are currently under development [Bro-Nielsen97, Kühnapfel 99, Waterworth 99, Waterworth 99a].

Among several advantages of the use of simulation systems in medical area,it’s possible to point out:$ Students could see and feel the effects of a group of basic diseases, and

could evaluate the importance and cost of the exams.$ The possibilit y to teach diagnostic abiliti es, since every student work withthe same case.$ Interactivity, due the immediate feedback received by the students,allowing them to evaluate their selves and to correct their mistakes,specially when the system also simulate the patient state according to thetreatment that he/she received from the student.$ The faciliti es introduced by the RV use to teach surgical and diagnosticprocedures (e.g. laparascopy, arthroscopy, etc.), allowing the students torepeatedly explore the structures of interest, take them apart, put themtogether, and view them from almost any point of view.$ Training could be done anytime and anywhere the equipment is available,providing a new way to practice several times and to simulate seldom or“hard” cases, until the physicians or the students get used with newtechnologies.$ Since the mistakes are done in the computer, and not in real patients, theuse of simulation systems reduce the operative risks associated with thelearning use of new techniques, consequently avoiding longer operationsand reducing surgical morbidity and mortality.

121

% Cost reduction, since the use of cadavers and animal experiments areavoided [Gross 98, Waterworth 99a, Jesus 99, Satava 98].

So, it’s possible to notice that an important aspect of recent practicalsimulation systems is the emphasis on teaching and training procedures, especiallywith the significant contribution of VR. The development of VR training simulatorsallows the surgeon to practice difficult procedures under computer control. Anexample is the training system for minimally invasive surgery developed under theKISMET Project, in Karlsruhe research center [Kühnapfel 99]. Its goal is to supportthe teaching and training of the operators with a computer-based simulation system,which imitates the operation area and provides a real-time synthetic endoscopic view,as ill ustrated in Figure M4.13. The coordination of instruments, the hand-eyecoordination, and the teamwork can be practiced as well as the operation proceduresand the use of new instruments. It is possible to implement several complications in atraining parcours as well as anatomic specialties. Structured training steps arerepeatable and reproducible. An evaluation of training success can be performed;expert system feedback could be another useful feature.

Figure M4.13: Kar lsruhe Endoscopic Surgery Trainer

The Georgia Institute of Technology and the Medical College of Georgiaalready developed an eye surgery simulation system, which provides both visual andtactile feedback while a surgeon operates on a computer model of the eye in a virtualenvironment. In practice, ophthalmic surgeons operate on an eye by looking through astereo microscope while steadying their hands (holding the surgical instruments) on awrist rest that surrounds the patient's head. The simulator also includes a stereooperating scope and a wrist rest. However, instead of looking directly at a real eye, thesurgeon interacts with a virtual eye using a virtual surgical instrument controlled by ahand held 3D position tracking stylus that continuously reports position andorientation to the computer (Figure M4.14). The stylus provides force feedback inresponse to the tool-tissue interaction. The simulation includes options to change

122

instruments, record and playback training sessions, reset the models, and peel awayouter layers of the eye to reveal interior anatomical components. Dials allow thesurgeon to rotate the model, change transparency, zoom, and adjust stereo viewingparameters. An instrument activation switch on the stylus controls actions such asopening and closing forceps and scissors [Peifer 99].

Figure M4.14: Kar lsruhe Endoscopic Surgery Trainer

Microsurgery simulation systems have also been developed in US. In oneinteresting possibilit y, the physician could train and improve his abilit y to do complexand delicate surgery of blood vessels microsuture, using RV equipments that allow theuse of real scissors, calipers, etc, over a virtual image of a vessel. Eventually, this kindof application could be used in a telesurgery procedure. Figure M4.15 shows a standwith micromanipulators and a series of images of a simulated surgery area [Sabbatini99].

Figure M4.15: Microsurgery simulation

The objective of the MIAS project is to develop technologies for obtainingminiaturization and enhancement of reality in arthroscopy in order to fulfil widely

123

perceived and real requirements to the surgeon. The main requirements of mostsurgeons using MIS (Minimally Invasive Surgery) techniques are:& To have access to more effective surgical tools, which replicate as closely

as possible the dexterity of the surgeons hand;& To achieve a restoration of the sensory (primarily visual and tactile)information that, previously available by open sky surgery, is missed withMIS.

The MIAS project aims to provide solutions to needs in a specific, butclinically very important (and with very important industrial background) field, suchas arthroscopy in orthopedic surgery. A first specific goal of MIAS project is todevelop an enhanced reality system to help surgeons to use multidimensional data(images, forces, planned procedures) in a fully integrated and effective manner. It alsoaims to develop new and more effective miniaturized tools for the execution ofaccurate and safe surgical procedures on human joints by arthroscopy. In particular theMIAS project aims to develop a set of miniature surgical devices exploitingMicrosystems and Micromechatronic technologies. The instrument will i ncorporatesuitable actuation systems and mechanisms for the active parts of a steerable andadaptive arthroscope, and for the several and interchangeable surgical tools mounted atits tip, a control system, a master-slave telemicromanipulation system, and aminiaturized camera system.

Concerning the software aspects, the MIAS project involves the developmentof a graphic tool to allow real-time surgery simulations, for the use in the education ofsurgery students. Nedel [Nedel 98] proposes the implementation of a hybriddeformation model to simulate cartilageous tissues. During an arthroscopy, cartilagescan suffer serious injuries if the surgical tools touch it. The simulation of the cartilageselastic reaction when touched by the tools can aid on these injuries avoidance. FigureM4.16 presents some examples of the cartilages reaction when touched by a surgicaltool.

Figure M4.16: Knee car tilages deformation. Detailed view of the bonesand car tilages dur ing a deformation.

124

Another example of one emergent area of medical education to which VR isbeing applied is that of dealing with catastrophic emergencies threatening or damagingthe health of large number of people: earthquakes, plane crashes, major fires, and soon. Here, rather general VR techniques are used to simulate a disaster scene, and thetrainee medical use the environment to learn how to allocate resources, prioriti ze casesfor treatment, etc. Recently, the scope of VR applications in medicine has alsobroadened to include physical and psychiatric rehabilit ation and diagnosis[Waterworth 99a, Waterworth 99].

Waterworth [Waterworth 99a] said that nowadays there is something of crisisin current surgical training, because as the techniques become more complicated, andmore surgeons require longer training, fewer opportunities for such training exists.Training in the operating theatre itself brings increased risk to the patient and longeroperations. New surgical procedures require training by other doctors, who are usuallybusy with their own clinical work. Moreover, it’s diff icult, for example, to trainphysicians in rural areas in new procedures, and animal experiments, which of coursehave different anatomy from human beings, are expensive.

So, the solution to these problems seems to be the development of trainingsimulators, mainly with the use of VR techniques. It’s possible to conclude that theeducational challenge is to use the potential of surgery simulation systems toincorporate into the curriculum the most effective way of teaching manual skill swithin the context of a structured educational process, then stimulating the student toexplore through practice. The greatest power of these systems is the abilit y to try andfail , without consequences to an animal or patient, since the surgeon is able to practicediff icult procedures under computer control. It is only through failure that the truepathway to success lies [Waterworth 99a, Satava 98].

7. References

[Basdogan 98] Basdogan, C; HO, C.; Srinivasan, M. A. et al. “Force Interactionsin Laparoscopic Simulations: Haptic Rendering of Soft Tissues” .In: MEDICINE MEETS VIRTUAL REALITY, 6., 1998, SanDiego, CA. Proceedings... Amsterdam: IOS Press. p. 385-391.Available at http://touchlab.mit.edu/people/cagatay/deform05.ps(Dez 1998).

[Basdogan 98a] Basdogan, C. “Simulating Surgical Procedures in VirtualEnvironments” . Available athttp://touchlab.mit.edu/people/cagatay/surgicalVR.html (Dez 98).

[Basdogan 98b] Basdogan, C. “Simulation of Bleeding in Virtual Environments” .Available athttp://touchlab.mit.edu/people/cagatay/surgicalVR.html (Dez.1998).

[Basdogan 99] Basdogan, C; HO, C.; Srinivasan, M. A. “Simulation of TissueCutting and Bleeding for Laparoscopic Surgery Using Auxili arySurfaces” . In: MEDICINE MEETS VIRTUAL REALITY, 7.,1999, San Francisco, CA. Proceedings... Amsterdam: IOS Press.Available at http://touchlab.mit.edu/people/cagatay/mmvr99.ps(Dez. 1998).

[Berkley 99] Berkley, J.; Weghorst, S. et al. “Fast Finite Element Modeling forSurgical Simulation” . In: MEDICINE MEETS VIRTUAL

125

REALITY, 7., 1999, San Francisco, CA. Proceedings...Amsterdam: IOS Press.

[Bro-Nielsen 96] Bro-Nielsen, M.; Cotin, S. “Real-Time Volumetric DeformableModels for Surgery Simulation Using Finite Elements andCondensation” . Computer Graphics Forum, Cambridge, v. 15, n. 3,Aug. 1996. Work presented on EUROGRAPHICS, 1996,Futuroscope-Poitiers, France.

[Bro-Nielsen 97] Bro-Nielsen, M. “Fast Finite Elements for Surgery Simulation” . In:MEDICINE MEETS VIRTUAL REALITY, 5., 1997.Proceedings... Amsterdam: IOS Press. Available athttp://serv1.imm.dtu.dk/documents/users/bro/mmvr5_paper.pdf(Nov. 1998).

[Bro-Nielsen 97a] Bro-Nielsen, M. “Simulation Techniques for Minimally InvasiveSurgery” . Journal of Minimally Invasive Therapy & Alli edTechnologies (MITAT), 6. p. 106-110, 1997. Available athttp://serv1.imm.dtu.dk/~bro/p_mitat2.pdf (Dez. 1998).

[Bro-Nielsen 98] Bro-Nielsen, M. “Finite Element Modeling in Medical VR”.Journal of the IEEE, 86(3). p. 490-503, 1998. Available athttp://serv1.imm.dtu.dk/~bro/p_ieee97_final.pdf (Dez. 1998).

[Bro-Nielsen 98a] Bro-Nielsen, M.; Helfrick, D.; Glass, B. et al. “VR Simulation ofAbdominal Trauma Surgery” . In: MEDICINE MEETS VIRTUALREALITY, 6., 1998, San Diego, CA. Proceedings... Amsterdam:IOS Press. p. 117-123. Available athttp://serv1.imm.dtu.dk/~bro/p_mmvr6_7.pdf (Dez. 98).

[Cotin 96] Cotin, S.; Delingette, H.; Ayache, N. “Real-Time VolumetricDeformable Models for Surgery Simulation” . In:INTERNATIONAL CONFERENCE ON VISUALIZATION INBIOMEDICAL COMPUTING, 4., 1996, Hamburg, DE.Proceedings... Berlin: Springer-Verlag, 1996. p. 535-540. (LectureNotes in Computer Science, v. 1131).

[Cotin 96a] Cotin, S.; Delingette, H.; Bro-Nielsen, M.; Ayache, N. “Geometricand Physical Representations for a Simulator of Hepatic Surgery” .In: MEDICINE MEETS VIRTUAL REALITY, 4., 1996, SanDiego, CA. Proceedings... Amsterdam: IOS Press. p. 139-151.Available at http://www-sop.inria.fr/epidaure/personnel/cotin/MMVR4/MMVR4.html (Dez.1998).

[Cotin 98] Cotin, S.; Delingette, H.; Ayache, N. “Eff icient Linear ElasticModels of Soft Tissues for Real-Time Surgery Simulation” .Research Reports, Oct. 1998. Available athttp://www.inria.fr/RRRT/publications-eng.html (Dez. 1998).

[Cover 93] Cover, S. A.; Ezquerra, N. F.; O’ Brian, J. F. “ InteractivelyDeformable Models for Surgery Simulation. IEEE ComputerGraphics & Applications, Los Alamitos, CA, v. 13, n. 6, p. 68-75,Nov. 1993.

[Darabi 97] Darabi, K; Resch, K. D. M.; Weinert, J. et al. “Real and SimulatedEndoscopy of Neurosurgical Approaches in an Anatomical

126

Model” . In: FIRST JOINT CONFERENCE COMPUTER VISION,VIRTUAL REALITY AND ROBOTICS IN MEDICINE ANDMEDICIAL ROBOTICS AND COMPUTER-ASSISTEDSURGERY, CVRMed-MRCAS, 1., 1997, Grenoble, France.Proceedings... Berlin: Springer-Verlag, 1997. p. 323-326. (LectureNotes in Computer Science, v. 1205).

[Delp 97] Delp, S. L.; Loan, P.; Basdogan, C. et al. “Surgical Simulation: Anemerging technology for emergency medical training. Presence:Teleoperators and Virtual Environments (Special issue on VirtualEnvironments in Medicine)” , MIT Press, v. 6, n. 4, p. 147-159,1997. Available athttp://touchlab.mit.edu/people/cagatay/cbpresence2.ps (Dez 98).

[Delingette 98] Delingette, H. “Towards Realistic Soft Tissue Modeling in MedicalSimulation” . Proceedings of the IEEE: Special Issue on SurgerySimulation, 1998. Available athttp://www.inria.fr/RRRT/publications-eng.html (Dez. 1998).

[Dionisio 97] Dionisio, J.; Henrich, V.; Jakob, U. et al. “The Virtual Touch:Haptic Interfaces in Virtual Environments” . Computers&Graphics,Oxford, v. 21, n. 4, p. 459-468, July/Aug. 1997.

[El-Khalili 98] El-Khalili , N. H.; Brodlie, K. W. “Architectural Design Issues forWeb-Based Virtual Reality Training Systems”. Available athttp://www.scs.leeds.ac.uk/vis/nuha/websim/el-khalili nh.html (Set.1998).

[Gibson 97] Gibson, S.; Samosky, J.; Mor, A. et al. “Simulating ArthroscopicKnee Surgery Using Volumetric Object Representations, Real-Time Volume Rendering and Haptic Feedback” . In: FIRST JOINTCONFERENCE COMPUTER VISION, VIRTUAL REALITYAND ROBOTICS IN MEDICINE AND MEDICAL ROBOTICSAND COMPUTER-ASSISTED SURGERY, CVRMed-MRCAS,1., 1997, Grenoble, France. Proceedings... Berlin: Springer-Verlag,1997. p. 369-378 (Lecture Notes in Computer Science, v. 1205).

[Gross 98] Gross, M. H. “Computer Graphics in Medicine: From Visualizationto Surgery Simulation” . Computer Graphics, New York, v. 32, n. 1,p. 53-56, Feb. 1998.

[Hong 97] Hong, L.; Muraki, S.; Kaufman, A. et al. “Virtual Voyage:Interactive Navigation in the Human Colon” . In: SIGGRAPH,1997, Los Angeles, CA. Proceedings... Addison Wesley, Aug.1997. p. 27-34.

[Jesus 99] Jesus, P. C. de. “Simulações Clínicas no Ensino Médico” .Informática Médica, v. 2, n. 2, Mar./Abr. 1999. Available athttp://www.epub.org.br/informaticamedica/n0202/simulacao.htm(Sep. 1999).

[Jolesz 99] Jolesz, F. A.; Lorensen, W. E.; Shinmoto, H. et al. “ InteractiveVirtual Endoscopy” . Available athttp://splweb.bwh.harvard.edu:8000/pages/papers/frank/endoscopy.html (Jan. 99).

127

[Koch 96] Koch, R. M.; Gross, M. H.; Carls, F. R. et al. “Simulating FacialSurgery Using Finite Element Models” . In: SIGGRAPH, 1996,New Orleans, Louisiana. Proceedings... ACM, Aug. 1996. p. 428-421.

[Kühnapfel 99] Kühnapfel, U. “The KISMET 3D-Simulation Software”. Availableat http://iregt1.iai.fzk.de/KISMET/UGK_home.html (Sep. 1999).

[Lin 99] Lin, W.; Robb, R. A. “Dynamic Volume Texture Mapping andModel Deformation for Visually Realistic Surgical Simulation” . In:MEDICINE MEETS VIRTUAL REALITY, 7., 1999, SanFrancisco, CA. Proceedings... Amsterdam: IOS Press.

[Massie 98] Massie, T. H.; Salisbury, J. K. “The PHANToM Haptic Interface:A Device for Probing Virtual Objects” . Available athttp://www.sensable.com/haptics/asme.html (Set. 1998).

[National 99] National Library of Medicine. “The Visible Human Project” .Available athttp://www.nlm.nih.gov/research/visible/visible_human.html (Set.1999).

[Nedel 98] Nedel, Luciana Porcher. “Anatomic Modeling of Human BodiesUsing Physically-Based Muscle Simulation” , Ph.D. Thesis, LIG –Computer Graphics Lab, EPFL – École Polytechnique Fédérale deLausanne, Lausanne, Switzerland, June 1998.

[Peifer 99] Peifer, J. “Eye Surgery Simulation” . Available athttp://www.bitc.gatech.edu/bitcprojects/eye_sim/eye_surg_sim.html (Sep. 1999).

[Poston 96] Poston, T.; Serra, L.; Solaiyappan, M. et al. “The GraphicsDemands of Virtual Medicine”. Computers&Graphics, Oxford, v.20, n. 1, p. 61-68, 1996.

[Rhodes 97] Rhodes, M. L. “Computer Graphics and Medicine: A ComplexPartnership” . IEEE Computer Graphics & Applications, LosAlamitos, CA, v. 17, n. 1, p. 22-28, Jan.-Feb. 1997.

[Robb 97] Robb, R. A. “Virtual Endoscopy: Evaluation Using the VisibleHuman Datasets and Comparison with Real Endoscopy inPatients” . In: MEDICINE MEETS VIRTUAL REALITY, 5., 1997.Proceedings... Amsterdam: IOS Press. p. 195-206.

[Rosenblum 97] Rosenblum, L. J.; Macedonia, M. R. “Projects in VR”. IEEEComputer Graphics & Applications, Los Alamitos, CA, v. 17, n. 5,p. 6-10, Sept./Oct. 1997.

[Sabbatini 97] Sabbatini, R. M. E. “Realidade Virtual no Ensino Médico (Show deSlides)” . Núcleo de Informática Biomédica da UniversidadeEstadual de Campinas, 1997. URL:http://www.nib.unicamp.br/slides/realidade-virtual.

[Sabbatini 99] Sabbatini, R. M. E. “Realidade Virtual no Ensino Médico” ,Revista Informática Médica, 2(2), 1999. URL:http://www.epub.org.br/informaticamedica/n0202/sabbatini.htm.

[Salisbury 97] Salisbury, J. K.; Srinivasan, M. A. “Phantom-Based HapticInteraction with Virtual Objects” . IEEE Computer Graphics &

128

Applications, Los Alamitos, CA, v. 17, n. 5, p. 6-10, Sept./Oct.1997.

[Satava 93] Satava, R M. “The Role of Virtual Reality in Medicine of the 21stCentury” , Virtual Reality Systems. 1(2), pp. 64-67, 1993.

[Satava 98] Satava, R. M. “The Virtual Surgeon”. The Science, v. 38, n. 6, p.34-39, Nov./Dec. 1998.

[Sederberg 86] Sederberg, T. W.; Parry, S. R. “Free-Form Deformation of SolidGeometric Models” . Computer Graphics, New York, v. 20, n. 4, p.151-160, Aug. 1986.

[SensAble 99] SensAble Technologies. “ Innovators in 3D Touch”. Available athttp://www.sensable.com (Jan. 1999).

[Srinivasan 97] Srinivasan, M. A.; Basdogan, C. “Haptics in Virtual Environments:Taxonomy, Research Status, and Challenges” .Computers&Graphics, Oxford, v. 21, n. 4, p. 393-404, July/Aug.1997.

[Trowbridge 98] Trowbridge, T.; Hollands, R. “Virtual Arthroscopic Knee SurgerySimulator” . Available athttp://www/shef.ac.uk/~vrmbg/arthrol.html (Nov. 1998).

[Waterworth 98] Waterworth, J. A. “Trip Report: Medicine Meets VR andNextMed”, San Diego, January 1998. Available athttp://www.informatik.umu.se/~jwworth/NM&mmvr98.html (Dez.1998).

[Waterworth 99] Waterworth, J. A. “The State of the State-of-the-Art: VR inSurgery is not yet Reality” . In: MEDICINE MEETS VIRTUALREALITY, 7., 1999, San Francisco, CA. Proceedings...Amsterdam: IOS Press.

[Waterworth 99a] Waterworth, J. A. “Virtual Reality in Medicine: A Survey of theState of the Art” . Available athttp://www.informatik.umu.se/~jwworth/medpage.html (Sep.1999).

[Yagel 96] Yagel, R.; Stredney, D.; Wiet, G. J. et al. “Building a VirtualEnvironment for Endoscopic Sinus Surgery Simulation” .Computers&Graphics, Oxford, v. 20, n. 6, p. 813-823, 1996.

129

MODULE 5:Final Considerations

1. Introduction

This module presents other applications for the use of computer graphics andvirtual human bodies in medicine, showing part of the large market that still could beexplored. In Section 2 a brief description of the virtual humans’ use in psychiatry isdone. Some augmented reality applications are presented in Section 3, and the newfield of collaborative visualization is described in Section 4. Finally, some commentstrying to enumerate the problems still t o be solved and the challenges of this area aredrawn in Section 5.

2. Vir tual Humans: Psychiatry and Behavior

Virtual humans are not only visual. They have a behavior, perception, memory,and some reasoning. Behavioral human animation is a new topic consisting ofdeveloping a more general concept of autonomous actors, reacting to environmentsand making decisions based on perception systems, memory, and reasoning. Behavioris often defined as the way animals and humans act, and is usually described in naturallanguage terms that have social, psychological, or physiological significance, butwhich are not necessarily easily reducible to the movement of one or two muscles,joints, or end effectors.

A typical human behavioral animation system is based on the three keycomponents: the locomotor system; the perceptual system; and the organic system.

A locomotor system is concerned with how to animate physical motions ofvirtual humans in their environment. A perceptual system is concerned withperceiving the environment. The modes of attention are: orienting, li stening, touching,smelli ng, tasting, and looking. The organism system is concerned with rules, skill s,motives, drives, and memory. It may be regarded as the “brain” of the actor.

With such a system, it will be possible in the future to change parameters in thethree key components for simulating some specific behavioral troubles in psychiatry.

Another aspect addressed by Whalley [Whalley 92] is the use of Virtual realityand virtual humans in psychotherapies. Using this new technique, it will be possible torecreate situations in a virtual world, immersing the real patient into virtual scenes. Forexample, it will be possible to reunite the patient with a deceased parent, or to simulatethe patient as a child allowing him or her to relive situations with familiarsurroundings and people.

130

3. Augmented Reality Applications

In the past few years, Virtual Reality equipment and techniques have attracteda great deal of media attention. The basic objective of these technologies is to immersea user inside an imaginary, computer-generated "virtual world". Although manydifferent technologies can be used to achieve this effect, they all share a commonresult: the user is cut off from any view of the real world outside. A variation of thiskind of technology is the so-called Augmented Reality (AR). In contrast, AR allowsthe user to see the real world; with virtual objects superimposed upon or compositedwith the real world. Ideally, it would appear to the user that the virtual and real objectscoexisted in the same space, enhancing the observer’s perception of the real world andgiving him a “superman” like vision [Azuma 99]. AR can be thought of as the“middle ground” between VR (completely synthetic) and telepresence (completelyreal). We can define as AR, any system that:

1. Combines real and virtual,

2. Are interactive in real time and

3. Have the real and virtual scenes registered in the space 3-D [Azuma 97].

Two basic choices are available to accomplish the combining of real andvirtual in the designing of an AR system: optical and video see-through HMDs. Eachhas particular advantages and disadvantages.

Optical see-through HMDs work by placing optical combiners in front of theuser’s eyes. These combiners are partially transmissive, so that the user can lookdirectly through them to see the real world. The combiners are also partially reflective,so that the user sees virtual images bounced off the combiners from head-mountedmonitors. The optical combiners usually reduce the amount of light that the user seesfrom the real world, acting like a pair of sunglasses when the power is cut off . FigureM5.1 shows a conceptual diagram of an optical see-through HMD, while Figure M5.2shows an optical see-through HMD made by Hughes Electronics.

Figure M5.1: Optical see-through HMD design

The video see-through HMDs work by combining a closed-view HMD withone or two head-mounted video cameras. The video cameras provide the user’s viewof the real world. Video from these cameras is combined with the graphic imagescreated by the scene generator, blending the real and virtual. The result is sent to themonitors in front of the user’s eyes in the closed-view HMD. Figure M5.3 shows aconceptual diagram of a video see-through HMD and Figure M5.4 shows an actual

131

video see-through HMD, with two video cameras mounted on top of a Flight Helmet[Azuma 97].

Figure M5.2: Optical see-through HMD

Figure M5.3: Video see-through HMD design

AR systems can also be built using monitor-based configurations; instead ofsee-through HMDs. Figure M5.5 shows how a monitor-based system might be built .In this case, one or two video cameras, static or mobile, view the environment. Thevideo of the real world and the graphic images generated by a scene generator arecombined and displayed in a monitor in front of the user. In this case, the user does notwear the display device.

In medicine, Augmented Reality could be used as a visualization and trainingaid for surgery. The 3-D datasets can be collected of a patient in real time; using non-invasive sensors like Magnetic Resonance Imaging (MRI), Computed Tomographyscans (CT), or ultrasound imaging. These datasets could then be rendered andcombined in real time with a view of the real patient. As mentioned by Azuma, thiswould give a doctor “X-ray vision” inside a patient. This method would be very usefulduring minimally invasive surgery, which reduces the trauma of an operation by using

132

small i ncisions or no incisions at all . AR technology could provide an internal viewwithout the need for larger incisions.

Figure M5.4: Video see-through HMD device

Figure M5.5: Monitor-based AR conceptual diagram

Another important application of AR is for general medical visualization tasksin the surgical room. Surgeons can detect some features with the naked eye that theycannot see in MRI or CT scans. AR would give surgeons access to both types of data,simultaneously. This might also guide precision tasks, such as displaying where todrill a hole into the skull for brain surgery or where to perform a needle biopsy of atiny tumor. The information from the non-invasive sensors would be directlydisplayed on the patient, showing exactly where to perform the operation. Figure M5.6and M5.7 show some applications of these AR techniques developed at the Universityof North Carolina and at Harvard Medical School.

133

Figure M5.6: Image-guided surgery, implemented as AugmentedReali ty, at the University of Nor th Carolina [Azuma 97].

Figure M5.7: Combined neurosurgery planning and augmentedreali ty from Harvard Medical School.

134

There are a potentially large number of important and exciting applications,where AR techniques can be useful. However, a number of technologies must maturebefore AR can attain this bright future: HMDs must offer higher resolution monitors,greater comfort and become less conspicuous. Time criti cal rendering algorithms, suchas just-in-time incorporation of tracker measurements, are needed, as are methods forpredictive and hybrid tracking. Research into eye tracking technologies has thepotential to yield registration improvements, and further research is needed intotechniques for system delay reduction. Finally, issues of portabilit y must be resolvedso those observers can wear these new improved systems at any location where thereis work or play to be done [Elvins 98].

4. Telemedicine: Collaborative Visualization

Despite some problems, as cost, quality and efficiency, telemedicineapplication area had a crescent growing in the last few years. Telemedicine consists intelecommunications technology use for interaction between health professionals andpatients, or between physicians, with the goal of distance medical action fulfillment’s[Sabbatini 98]. Then, telecommunication technologies are used to transfer medicalinformation between sites, trying to facilit ate medical procedures. Among several sub-areas that could be identified in telemedicine applications, as teleradiology, whereradiological images are sent from one place to the other, and teleconference, that allowremote clinical consultations, in this section only collaborative visualization isdescribed.

Nowadays, medical diagnosis in criti cal diseases, and scientific research ingeneral are seldom executed by only one person. Often a team of collaborators,sometimes of different areas and institutions, works in the same subject, sharing anddiscussing partial results. Moreover, it might happen that there are no specializedphysicians in the same city of a patient. With the growing of the WWW - World WideWeb, several high technology solutions have been developed to support cooperativework. Communication technologies are used to overcome the geographical separationof collaborators and to achieve the expected level of cooperation using teleconference,videoconference, electronic mail , and network documents management systems [Spurr94].

Since in areas where results are presented as images, visualization can beexecuted in different places, at different moments, and of course by more than oneperson, and considering the development of new technologies, researchers ofvisualization techniques started to work in “collaborative visualization” systems. Inthese kinds of systems, collaborators in remote sites can simultaneously analyze thesame set of scientific data. The support of the so-called collaborative visualizationsystems has to foresee simultaneous data visualization by several users and also theinteraction among users and between users and data. Recently, this compelli ng area,called CSCV – Computer Supported Collaborative Visualization, is becoming morepopular and important. However, the development of collaborative visualizationsystems still presents many challenges inherent of the multi -user nature of theseinteractive applications [Johnson 98].

Although several authors aff irm that visualization is a collaborative activity,actual visualization systems treat visualization as an individual activity. Certainly,scientists have a great number of powerful systems as IRIS Explorer, AVS(Application Visualization System), Khoros and IBM Data Explorer. However, whenusing these systems scientists need to be physically together to verify the results. Due

135

to this limitation, the developers of such systems and of others less generic butextensively described in the literature (e.g. VisAD [Hibbard 98]) started to work intheir improvement to allow collaborative visualization. One example is thecollaborative modules of AVS under development at San Diego SupercomputerCenter. These new modules will augment the AVS capabiliti es enabling multiplephysically remote users to cooperatively interrogate the same scientific data usingfamiliar visualization tools [Johnson 98, Johnson 98a]. Since the researchers knowhow to work with the system (AVS), they are encouraged to adopt collaborative workhabits.

The majority of visualization systems follow the dataflow model to implementthe visualization pipeline, which has been detailed by Haber and McNabb [Haber 90]in their reference model. In this model, visualization is obtained in a three-processpipeline: filtering, mapping and rendering (figure M5.8). The first process correspondsto the gathering of data from an entry process that could be an interpolation process toconvert an irregular grid to a regular one and the selection of the desired resolutionand region of interest. Mapping of the filtered data consists in transforming it to ageometrical representation, li ke an isosurface, and finally the rendering stage generatesan image, animation or other product from the geometrical description [Wood 97,Johnson 98].

Figure M5.8: Visualization pipeline of Haber and McNabb

Sharing of images obtained from a single data set can be achieved, forexample, by duplicating all user interfaces in different workstations or producingidentical images from the same data set in a synchronous way. In the latter case,besides the use of separated pipelines operating in different machines, thesynchronization guarantee that the model consists in a single pipeline with multiplescontrols.

So, to achieve collaborative visualization the Haber and McNabb model can beextended to have intermediate entry and output points to control information and data[Wood 97]. To support several independent participants, there should be avisualization pipeline for each collaborator (figure M5.9), and data and controlinformation are exchanged at arbitrary points. Each stage could accept data andcontrol information to permit their collaborative operation and data exportation forsharing purposes. The control information of a pipeline could be exported to anotherto synchronize the collaboration. To share control, parameter definition has to belocally controlled by the user, or externally by a collaborator. For example, if twousers are collaborating in some mapping process, they could act in two ways. First,user A can export filtered data to the mapping stage of B and then they couldexchange control information. However, if the filtered data should remain private toA, they exchange control information in the mapping stage, after pipeline A hadexported the geometrical representation to pipeline B. In this case, the mapping stageof B consists only in a "ghost" process that generates appropriate control parameters tothe mapping process of A.

Recently, according to the existent applications that provide visualizationcapabiliti es in a collaborative context, Johnson [Johnson 98a] has categorized these

Data Filtering(F) Mapping(M) Render(R)

Image

136

systems by the level of shared control they provide over the visualization process. Inthe first category, Local Control, a collaborative visualization application consists ofimage data rebroadcast to all participants. In this case, only the user creating theimage(s) has direct interaction with the visualization process, while the otherparticipants are limited to passive viewing of the results and exchanging ideas, forexample, via a telephone or whiteboard teleconferencing software.

The second category, Local Control with Shared Data, used in somecollaborative AVS modules, is a more complex variation represented by applicationsin which participants can share data from any step in the visualization process. Directinteraction and control over the visualization process occurs locally. Partially or fullyprocessed data can be shared. Limited Shared Control is the category represented byapplications in which participants can share the viewpoints of others, and insert itemsinto this shared view. Cooperate control of the visualization process is primarilylimited to annotation of the resulting visual elements, and control of the view position.Finally, the Fully Shared Control category includes those applications that provideshared control over the parameters associated with a given visualization. Theseparameters may affect how a dataset is filtered, mapped to graphical elements, viewedand aspects of the products output. Thus, the work of “steering” any aspect of thevisualization process can be a shared activity. Shastra environment (section 4) andsome AVS modules have this kind of cooperative interaction [Johnson 98a].

Figure M5.9: Reference model extension for collaborativevisualization [Wood 97].

Then, collaborative visualization systems for the medical area emerged as a setof techniques that deal with the problems of presenting images to remote collaboratorsand supporting interaction based on these images. Several collaborative systems havealready been developed, and to exempli fy this system category, some of them will bebriefly presented here. The first one, TeleInViVo [Coleman 96], was developed at theFraunhofer Center for Computer Graphics Research with the sponsorship of theDefense Advanced Research Projects Agency (DARPA) and the US Army MedicalAdvanced Technology Management Off ice (MATMO). It supports collaborativevisualization and exploration of volumetric data including computed tomography,magnetic resonance and PET - Positron-Emission Tomography. The main goal ofTeleInViVo is to facilit ate diagnosis, medical training, surgery, and therapy planningand treatment, using real-time visualization in a distributed environment.

TeleInViVo, which run in UNIX environment, was developed using an object-oriented approach unifying volume visualization, CSCW and telecommunicationsconcepts. It is based on point-to-point communication with TCP/IP protocol. A session

Shared dataShared Control

DataFiltering(F) Mapping(M) Render(R)

Image

USER A

DataFiltering(F) Mapping(M) Render(R)

ImageUSER B

137

begins when a user calls another specifying an IP address. The caller user has thecontrol over the other. The participants could return to the "stand-alone" mode at anymoment. To optimize data transfer it is possible to transmit the data with a lowerresolution and, after specifying the subset of interest, this is requested and transmittedwith a higher resolution. TeleInViVo, which interface is presented in Figure M5.10,also provides tools for filtering, segmentation, volume visualization, isosurfaceconstruction and arbitrary slice display.

Figure M5.10: TeleInViVo inter face [Fraunhofer 99]

Another collaborative visualization system, SDSC_NetV [Elvins 96], wasdeveloped as an experimental system at the San Diego Supercomputer Center. It is adistributed system that: run in UNIX environment; uses advanced renderingtechniques; has a friendly graphical interface based on Motif; exhibits stereo images;and was designed to overcome the problems arising from the heavy processing oflarge volume of data used by volumetric visualization systems in the center's sharedenvironment. Users of SDSC_NetV can access several resources without knowing ifthey are locally or remotely located, can wear liquid crystal glasses to observe stereoimages, and a cyberglove for interacting with a shared immersive environment basedon textured polygonal rendering. To visualize volume data in motion, a pre-renderingis available, and then, the frames are exhibited in a loop.

Los Alamos National Laboratory (LANL) in collaboration with the NationalJewish Center for Immunology and Respiratory Medicine have recently developed acollaborative health-care environment system called TeleMed [Kilman 97]. Thissystem is a prototype for VPR - Virtual Patient Records, which provides a commonformat for visualization. It stores patient data in graphical format mode in such a waythat physicians can remotely query the data set, independent of the platform that isbeing used, since it was developed using Java and CORBA (Common Object RequestBroker Architecture). CORBA provides software inter-communication independentfrom the internal design or location of application modules [Vogel 97].

138

TeleMed goal is to standardize the electronic management of patientinformation. TeleMed dynamically unites graphical patient records with the supportfor interactive collaboration in real time among multiple users. With this systemseveral physicians can simultaneously access, edit and annotate patient data. TeleMedis called from a browser and an interface is downloaded. A login window is shownand authorized users can select a data base site. A patient can be selected throughhis/her name and the corresponding record is shown (Figure M5.11). Icons representradiological exams, which can be selected for analysis. Collaboration is achievedwhen two users load the same slice and one of them, for example, changes the positionof an arrow to show a point of interest. At the same time, the other user will see themovement of the arrow. Each time one wants to visualize another slice, the other hasto change the slice too. If another physician enters the session, the others areimmediately noticed through a list of on-line users in the browser.

Figure M5.11: TeleMed graphical patient records [TeleMed 99]

Finally, Shastra [Anupam 94] is a collaborative multimedia scientificmanipulation environment in which experts in a cooperating group communicate andinteract across a network to solve problems. Developed by a research group at thePurdue University (West Lafayette, US), Shastra allows distribution and collaborativevisualization through the implementation of two distributed visualization algorithms.Briefly, it consists of a group of interoperating applications collectively called toolsthat basically provide managing, communication and rendering faciliti es. Platformindependence is achieved by building applications atop abstract libraries that hidehardware specifics. These abstract libraries can be easily extended to supportstandardized interfaces as they evolve.

Shastra environment provides faciliti es for media-rich interaction over thecontext of shared visualizations. It has a rendering and visualization tool called Poly

139

that encapsulates graphical object manipulation, rendering and visualizationfunctionality. Poly interoperates with other Shastra tools and provides a very highlevel abstraction for manipulating graphical data. The Shastra environment, whichruns on an Ethernet network, consists of a collection of Poly instances. A collaborativesession is initiated by one of the Poly users in the environment that becomes the groupleader and then could specify the list of Poly users who will be invited to participate inthe session. Users who accept are incorporated into the session, and any Poly instance,or user, not in the conference can request admittance or can leave an ongoing sessionat any time. Every participant Poly instance creates a shared window in which allcooperative interaction occurs, allowing a fully shared control. Collaborating users canadjust visualization modes and parameters and modify viewing modes and direction.

Then, it is possible to say that with the widespread use of WWW newtechnologies were introduced, and a growing number of users aim to change theirsystems platforms into this new environment. One of such tendencies is using the Webas a wide collaboration environment. At present, we can find several collaborativesystems that were developed for the Web as well as frameworks that were designed toallow the development of Web based collaborative environments. But, according toworks already reported in literature, many challenges face both users and developers,especially in systems for the medical area. Among the many features that need to beworked on in collaborative visualization systems, it’s possible to point out thecommunication technology to be used, an efficient identification handling, thecoherence of shared data and synchronization of users activities. The creation ofattractive and user friendly interfaces to awake the physicians interests through itsadvantages is another challenge. Moreover, the problem to achieve real timevisualization and interaction, mainly because of network traffic and interaction withlarge volume data using medical instruments still exists. Depending of the interactiveapplication, touch feedback is also very important, but the high cost and low accuracyof VR devices difficult its use. The poor realism of images where faithful skin andmuscle textures are still mi ssing, is another open issue. So, for sure the building of acollaborative system is an interdisciplinary effort that needs a user-centered approach,an eff icient data management and an object-oriented design and programming toprovide reusabilit y of code [Greenes 90, Freitas 97].

5. Conclusions

The use of several computer graphics techniques applied to medicine wasintroduced in this work, trying to emphasize the necessary steps involved in thecreation of virtual human bodies. This motivating area covers many procedures thatare extremely dependent of special hardware and computer graphics techniques, andintroduces challenge and fascinating research topics, where image accuracy, qualityand fast generation (considering acquisition modaliti es as well as the generatedresults) are fundamental.

Initially, some concepts about the great variety of imaging acquisitionmodaliti es that can provide inputs to medical visualization systems, and some imageprocessing techniques including registration and segmentation, were presented. Thesecond step in virtual human modeling, which includes reconstruction andvisualization techniques, is described in Module 2, emphasizing the human bodiesmodeling methods and the visible human project. In the third module, a method tosimulate human bodies based on anatomy concepts is proposed and a humanrepresentation method is described. The systems and hardware improvements alli edwith the use of new interactive and virtual reality techniques lead to the development

140

of surgery simulators, as related in Module 4. The last module presents some otherapplications and trends.

In the course of this work, many diff iculties still found for the real use ofmedical systems could be identified. For example, the problem of achieving real timeinteraction and the poor realism of the images are still open questions. Then, in thissection some of the open issues of the field will be briefly described since now.

Recently, we saw a great hardware evolution, leading to the development anduse of VR and AR techniques. These technologies have been used in medicineespecially in the education area, since it avoids the use of cadavers and patients duringtraining with new equipments. However, enormous challenges remain and surgerysimulation environments are not effectively used yet. These challenges include:' design of powerful editors allowing the user to modify the geometry and

topology of anatomic structures;' creation of an appropriate user-friendly interface;' implementation of fast rendering algorithms;' improvement of technical fidelity;' standardizing accurate metrics for performance evaluation;' stringent evaluation of simulators for effectiveness and value added tosurgical training; and' the cost that is still high.

Fortunately, this scenario is changing and with the extensive use of VR inentertainment market there is a tendency to decrease the price while increasing theaccuracy, and so medical education will be profit of it [Gross 98, Sabbatini 99].

One of the central problems in surgery simulation research is still t hedevelopment of real-time algorithms for physically based modeling of volumetricthree-dimensional objects. The real-time interaction and modeling (15-20 frames persecond) of complex solid volumetric objects deformation remains one of the biggestproblems to achieve eff icient results in surgery simulation. The surface visualizationalgorithm, which is faster than the volumetric one, cannot be used in simulators sincethe interior organ description is not available. Another important aspect is that theutili zation of volumetric models has to be done with care because it is very diff icult toachieve a good performance in real-time due to the complexity of the models.

The development of advanced interaction tools, indispensable in endoscopicprocedures, is another area of crescent interest. The requirements for a computersystem to simulate complex interaction are many and the simulator has to provide bothfine detail and realistic behavior during interaction. This is particularly diff icult withsoft tissue that moves, for example in the abdominal region, where most surgicalintervention is focused. The system usually must be able to generate realistic visualand haptic cues providing temporal and spatial coherence with the data manipulation.This correspondence of action/reaction provides the user with a sense of causalitysimilar to that experienced in interactions with the real world [Yagel 96, Waterworth99].

As mentioned before, generally, medical visualization systems, includingsurgery simulators, are not used by only one person, because it is necessary a group ofpeople working together to draw a diagnosis for a diff icult or rare disease, or toachieve a common research objective. Then, people started to develop medical

141

collaborative visualization systems and some of them were described here. Thesestudies provide a basis to identify some trends and diff iculties still found for the realutili zation of such systems. First of all , it became clear that the development ofplatform independent systems is fundamental, since several people in different placeswith different equipment will use it. So, technologies as Corba are getting moreimportance. Nor less important is the use of VR and AR techniques, but its use incollaborative systems is still i n the beginning, mainly due to the low accuracy and highcost problems.

Interaction is also a very important issue in the development of these systems.To provide efficient interaction in such applications we need to include, at least, thefollowing tools: annotation, authoring, pointing and selection in shared views of data;shared whiteboard; and synchronization. Visualization tools include user-specifiedvisual representations, multiple views of data and rendering at different levelresolution or using a partial dataset. Considering interaction and visualization, it isalso important to define how the communication, data management and data retrievalwill be. In the first case we should to consider the possibilit y of distributing data, codeor image. Data management includes the use of a centralized or distributed database,dealing with security and control of data access generated by different users. Dataretrieval involves a visual query language, associated visualization tools and recordingof queries [Freitas 97].

According to works already reported in literature and briefly described here,many challenges face both users and developers, still exists. Among the many featuresthat need to be worked on in collaborative visualization systems, it is possible to pointout the communication technology to be used, an efficient identification handling, thecoherence of shared data and synchronization of users activities. Moreover, theproblem to achieve real-time visualization and interaction, mainly because of networktraff ic and interaction with large volume data using medical instruments still exists.The poor realism of images where faithful skin and muscle textures are still mi ssing, isanother open issue. So, for sure the building of a collaborative system is aninterdisciplinary effort that needs a user-centered approach, an eff icient datamanagement and an object-oriented design and programming to provide reusabilit y ofcode [Greenes 90, Freitas 97].

Concerning the development of virtual human bodies for medicine, the socalled virtual patients, there yet a lot of open questions. To better address thesequestions, we can divide the problem in three different parts: geometric andtopological modeling of basic elements (bones, muscles, skin, tendons, organs andfat); mechanical modeling of soft tissues and complex joints; and motion control of thecomplex model of body [Magnenat 94].

The first step consists of designing models for all body materials: flesh, bones,and muscles. Data should be selected, prepared, and converted from medical imagingof human patients. Depending on the type of organ, various geometric methods shouldbe investigated, most notably parametric surfaces. For the muscles, the mesh should beadapted to the fibril structure in relation with the modeling of the contraction. Toensure continuity of shapes, topological problems should be also addressed. Becauseof the numerous approximations that are introduced into a computerized patient, it isnecessary to provide anatoic and physiologic expertise. Medical knowledge shouldserve at least the following two stages: conception of the models (the topology of theanatomic structures, sometimes based on some physiologic data); and matching of theanatomical models on the imaging data (whatever the quality of the segmentation,some simpli fications and decisions have to be made by an specialist).

142

Modeling of complex joints is essential. The use of physical laws should beimproved to control the motion of skeletons, but they also have an importantapplication in calculating deformations of bodies and faces, because thesedeformations are muscular in origin and the muscle action is most appropriatelycharacterized in mechanical terms. About the motion control of the complex body, theanimator is not concerned with forces but with the body positions required by thescript.The animator should impose prescribed displacements on bones using a"classical" motion control method, while the system should decide if chosen jointpositions are realistic, and determine the consequences on skin and eventual objects incontact.

Finally, it is important to notice that the participation of physicians in the rollprocess of medical systems development is crucial. Since they will be the users, theywill have to identify the important features of the interface, enumerate the appropriatehardware, visualization and interaction tools and analyze the results. Without amedical validation, a system will not be useful.

6. References

[Anupam 94] Anupam, V.; Bajaj, C.; Schikore, D. et al. “Distributed andCollaborative Visualization” . Computer, Los Alamitos, CA, v. 27,n. 7, p. 37-43, July 1994.

[Azuma 97] Azuma, R. T. “A Survey of Augmented Reali ty” . Presence:Teleoperators and Virtual Environments, v. 6, n. 4, p. 355 – 385,Aug. 1997.

[Azuma 99] Azuma, R. T. “Registration Errors in Augmented Reali ty” .Available at http://www.cs.unc.edu/~azuma/azuma_AR.html (Sep.1999).

[Bro-Nielsen 97] Bro-Nielsen, M. “Simulation Techniques for Minimally InvasiveSurgery” . Journal of Minimally Invasive Therapy & Alli edTechnologies (MITAT), 6. p. 106-110, 1997. Available athttp://serv1.imm.dtu.dk/~bro/p_mitat2.pdf (dez. 1998).

[Coleman 96] Coleman, J. et. al. “TeleInViVoTM: Towards Collaborative VolumeVisualization Environments” . Computers&Graphics, Oxford, v. 20,n. 6, p. 801-811, 1996.

[Dionisio 97] Dionisio, J.; Henrich, V.; Jakob, U. et al. “The Virtual Touch:Haptic Interfaces in Virtual Environments” . Computers&Graphics,Oxford, v. 21, n. 4, p. 459-468, July/Aug. 1997.

[Elvins 96] Elvins, T. T. “Volume Visualization in a Collaborative ComputingEnvironment” . Computers&Graphics, Oxford, v. 20, n. 2, p. 219-222, 1996.

[Elvins 98] Elvins, T. “Augmented Reality: "the Future's So Bright, I GottaWear (See-through) Shades"” , Computer Graphics, v. 32, n. 1,February 1998.

[Fraunhofer 99] Fraunhofer – Center for Research in Computer Graphics.TeleInViVoTM Version 2.5.1 – GUI Overview. Available athttp://www.crcg.edu/projects/medvis/TeleInViVo/refman/gui-overview.html (Sep. 1999).

143

[Freitas 97] C. M. D. S. Freitas. “Challenges in the Development of Interactionand Visualization Tools for Collaborative Applications” . Lecturepronounced at Brazili an Symposium on Computer Graphics andImage Processing, 10., 1997, Campos do Jordão, SP.

[Greenes 90] Greenes, R. A.; Brinkley, J. F. “Radiology Systems”. In:SHORTLIFFE, Edward H.; PERREAULT, Leslie E. (Eds.).Medical Informatics. USA: Addison-Wesley, 1990. p. 324-365.

[Gross 98] Gross, M. H. “Computer Graphics in Medicine: From Visualizationto Surgery Simulation” . Computer Graphics, New York, v. 32, n. 1,p. 53-56, Feb. 1998.

[Haber 90] Haber, R. B.; Mcnabb, D. A. “Visualization Idioms: A ConceptualModel for Scientific Visualization Systems”. Visualization inScientific Computing, IEEE, p. 74-93, 1990.

[Hibbard 98] Hibbard, B. “VisAD: Connecting People to Computations andPeople to People”. Computer Graphics, New York, v. 32, n. 3, p.10-12, Aug. 1998.

[Jesus 99] Jesus, P. C. de. “Simulações Clínicas no Ensino Médico” .Informática Médica, v. 2, n. 2, Mar./Abr. 1999. Available athttp://www.epub.org.br/informaticamedica/n0202/simulacao.htm(Sep. 1999).

[Johnson 98] Johnson, G. “Collaborative Visualization” . Computer Graphics,New York, v. 32, n. 2, p. 8-11, May 1998.

[Johnson 98a] Johnson, G. “Collaborative Remote Visualization Using theApplication Visualization System”. Available athttp://www.sdsc.edu/PET/projects-year2/collab-avs.html (Nov.1998).

[Kilman 97] Kilman, D. G.; Forslund, D. W. “An International CollaboratoryBased on Virtual Patient Records” . Communications of the ACM,New York, v. 40, n. 8, p. 111-117, Aug. 1997.

[Kühnapfel 99] Kühnapfel, U. “The KISMET 3D-Simulation Software”. Availableat http://iregt1.iai.fzk.de/KISMET/UGK_home.html (Sep. 1999).

[Magnenat 94] Magnenat-Thalmann, N. and Thalmann, D. "Towards VirtualHumans in Medicine: A Prospective View", Computerized MedicalImaging and Graphics, v. 18, n. 2, p. 97-106, 1994.

[National 99] National Library of Medicine. “The Visible Human Project” .Available athttp://www.nlm.nih.gov/research/visible/visible_human.html (Set.1999).

[Peifer 99] Peifer, J. “Eye Surgery Simulation” . Available athttp://www.bitc.gatech.edu/bitcprojects/eye_sim/eye_surg_sim.html (Sep. 1999).

[Sabbatini 98] Sabbatini, R.M.E. Telemedicina e Educação à Distância no Brasil(Show de Slides). Núcleo de Informática Biomédica daUniversidade Estadual de Campinas, 1998. URL:http://www.nib.unicamp.br/slides/telemed/index.htm.

144

[Sabbatini 99] Sabbatini, R. M. E. “Realidade Virtual no Ensino Médico” .Informática Médica, v. 2, n. 2, Mar./Abr. 1999. Available athttp://www.epub.org.br/informaticamedica/n0202/sabbatini.htm(Sep. 1999).

[Salisbury 97] Salisbury, J. K.; Srinivasan, M. A. “Phantom-Based HapticInteraction with Virtual Objects” . IEEE Computer Graphics &Applications, Los Alamitos, CA, v. 17, n. 5, p. 6-10, Sept./Oct.1997.

[Satava 98] Satava, R. M. “The Virtual Surgeon”. The Science, v. 38, n. 6, p.34-39, Nov./Dec. 1998.

[Spurr 94] Spurr, K.; Layzell , P.; Jennison, L.; Richards, N. “ComputerSupport for Cooperative Work” . Chichester, UK: John Wiley &Sons Ltd, 1994, 246 p.

[Srinivasan 97] Srinivasan, M. A.; Basdogan, C. “Haptics in Virtual Environments:Taxonomy, Research Status, and Challenges” .Computers&Graphics, Oxford, v. 21, n. 4, p. 393-404, July/Aug.1997.

[TeleMed 99] TeleMed User Manual. “Graphical Patient Record (GPR)Window”. Available at http://www.acl.lanl.gov/TeleMed (Sep.1999).

[Vogel 97] Vogel, A.; Duddy, K. “JavaTM Programming with CORBA”. NewYork, Wiley Computer Publishing, 1997, 425 p.

[Waterworth 99] Waterworth, J. A. “The State of the State-of-the-Art: VR inSurgery is not yet Reality” . In: MEDICINE MEETS VIRTUALREALITY, 7., 1999, San Francisco, CA. Proceedings...Amsterdam: IOS Press.

[Waterworth 99a] Waterworth, J. A. “Virtual Reality in Medicine: A Survey of theState of the Art” . Available athttp://www.informatik.umu.se/~jwworth/medpage.html (Sep.1999).

[Wood 97] Wood, J.; Wright, H.; Brodlie, K. “CSCV - Computer Support forCollaborative Visualization” . In: EARNSHAW, Rae; VINCE,John; JONES, Huw (Eds.). Visualization & Modeling. London,UK: Academic Press, 1997. p. 13-25.

[Yagel 96] Yagel, R.; Stredney, D.; Wiet, G. J. et al. “Building a VirtualEnvironment for Endoscopic Sinus Surgery Simulation” .Computers&Graphics, Oxford, v. 20, n. 6, p. 813-823, 1996.