8/7/2019 p53-gross

1/5

Computer Graphics in Medicine:From Visualization to Surge ry Simulation/Vlarkus H. GrossSwiss Federa l In st i tu te of Technology( E TH )AbstractMedicine is an extremely challenging field ofresearch, which has been -- more than anyothe r discipline - of fundamental imponlance inhuman existence.The variety and inherent com-plexity of unsolved problems has made it amajor driving force fo r man y natural and engi-neering science~. Hence, from the early days ofcomputer graphics the medical field has beenone of most important application areas withan enduring provision o f exciting research chal-lenges. Conversely.individual graphics tools andmethods have become increasinglyirreplaceablein modern medicine, where medical imagingsystems are only one prominent example.The purpose of the fo l lowing ar t ic le istwofold: Witho ut claiming completeness, thef irst pa rt gives a br ie f retrospective of thefruitful relationship between computer graph-ics and individual subareas of the medicalfield.We start w ith early imaging and 3D visu-alization and move via interactive, collabora-tive data analysis co the emerging field ofsurgery simulat ion.The second parr of th epaper presents a more detailed view on theinterdisciplinary field of virtual and simulatedsurgery which encompasses knowledge frommed ic ine , compute r g raph ics , compute rvision, mechanics, material sciences, robotic sand n umeric analysis. The autho r describesthe leading role of graphics and VR as coretechnologies and summarizes his personalvision of current and future research prob-lems, which have to be pursued for realizing

our vision of fully interactive and immersivesurgery simulation.Th e Past: InsightThroughVisualization3D medical imaging systems, such as X-ray(CT), magnet resonance (ME) or nuclear(SPECT) scanners, were revo lution ary devel-opments of modern diagnostics which con-quered th e f ield start ing in th e early '70s.These methods gave insight into almost e veryindividual sect ion of the human body andhave saved countless lives by ea rly diagnosisof tumors, heart diseases and others.

Th e data sets, usually defined on equallyspaced 3D grids, have been visualized interm s o f large sequences of individual slices,sometimes with the help of pseudocolor.Thesearch for more sophisticated wa ys of visual-izing this n ew type of volume data created anenormous push in a new subfield of comput-er graphics: volume rendering. Based on land-mark works of Bl inn and KajJya, manydifferent algorithms have been developed forth e eff ic ient and realist ic rendering o r vol-umes.The variety o f approaches ranges fromsimple back-to-front rendering to sophisticat-ed lighting and shading models implementedvia ray tracers.The produced images are high-ly realistic and ubiquitous.Besides direct approaches to volume ren-der ing, the geome t r i c r econs t ruc t ion o fanatomic structures gained increasing atten-tion, since it enabled th e fur ther processingand analysis of m edically imp orta nt features.In this c onte xt, data segmentation has provento be a critica l preprocessing step. In addition,higher-level volum e data analysis algorithms

have allowed the identif ication of individuaanatomic substructures.Generally. volume data feature extractioand interpretation is a paramount example oth e fruitful relationship and convergence ographics and v is ion. Here, the computevision community developed many differenstrategies, which partly belong to the repetoire of any graphics researcher working ithe medical field (for Instance, John Cannyfamous edge detector). Conversely, mangraphics methods are actually integral partof sophisricared computer vision technique(For instance, parametr ic polynomial surfaces).Many of the graphics and vision methoddesigned in the early days are presently weestablished and support advanced applications in medicine, such as rediatlon and opeat ion planning, prosthesis design, dentatreatment, education and training and manochers.

T h e Present: InteractiveExploration andT e l e c o l l a b o r a t i o nWhereas, in the early days,graphics and visualization algorithms were mostly designed apreprocessors producing st i l l images, themerging h igh performa nce comp uting angraphics hardware swiftly conquered the fieland changed the way medical data was analyzed. Nowadays, hardware-assisted real-timrendering of comp lex shapes and volumeallows for the interact ive explorat ion ananalysis of huge amounts o f m edical data.In addition, upcoming 3D input and outpudevices allow the development of virtua l rea

Figere I'A JAVA applet for distributedcompassion domain volumerendering(htq)'/Iwww.inf.elhz.ch/personal/lipperrJEVOLVF_).Seepage I01 ~r/m aL e in )CuBco/or.F'~nsre2" User fr~rface of ~e telecaopera~onsystem KANIEDIN.Imageprovided courtesy ar the CarnputerGreph~ Center,D a ~ Germany.See page I01 for Image in lu ll color.

Computer Graphics February1998 S

8/7/2019 p53-gross

2/5

i ty systems for interactive journeys throughthe human body. Highly immersive interfacesgenerate gorgeous i l lustrations and are cur-rently being investigated as enabling tools forthe next generation of medical training andeducation systems. In some other applicationscenar ios, VR systems are e nvis ion ed toreplace costly and dangerous invasive meth-ods.Yet, in most cases the lack of force feed-back turns out to be a critical issue because itseriously diminishes the degree of realism.In times where medical information sys-tems and databases are commonplace in mosthospitals, networked compression, visualiza-t ion and analysis methods gain more andmore impor tanc e . Ty pi c a l requ i r emen tsencompass fast searching and browsing oflarge CT or MR databases as well as scalabili-ty to the network performance and computa-tional pow er of the clients. As an example,Figure I presents a JAVA applet for rende ringremotely located volume data sets instanta-neously from a highly compressed file format.

Another vast subfield of graphics in medi-cine is collaborative diagnosis and telepres-ence. For instance in radiology, physiciansoften face the problem of having instanta-neous access to a specialist who is locatedsomewhere remote. Whereas in the pas timages wer e exchanged as hardcopies via sur-face mail, upcoming high-speed digital net-work s open t he door f o r r ea l - t imed i s t r i bu ted and c o l l abora t i v e d i agnos i s .Medical telecooperation projects and systemsare currently supported by various telecom-munications companies, and in some casesthey are already available as comme rcial prod -ucts. Figure 2 shows a user interface layout ofa typical telecooperation system for radiolo-gists.

Besides high-end CT and MR data sets,graphics researchers are discovering the lowerend of medical imaging systems. Here, ultrason-ic devices dominate the scene and variousapproaches for rendering and reconstruction o fultrasonic data sets can be found in contemp o-rary l i terature. Althoug h noise and alignmentproblems make a robu st segmentation and fea-ture ex t rac t ion much more di f f i cu l t , i t i sexpected that some of those methods will bepart of nex t generation's ultrasonic systems.Other research groups design high-endtelepresence systems, which enable the sur-geon to perform telesurgery using haptic androbotics interfaces. Additio nally, augmentedreality systems have been used for interactivediagnosis and medical check-ups. Images takenfrom a camera on the physician's head aresuper imposed with coincidental ly recordedultrasonic scans and are com posed in a head-moun ted display. Real-time projection s of 3 Dreconstruction of individual anatomic featuresallow the location and visual inspection of theinner part o f the human body.

Furthermore, 3D hardcopy devices, such asstereolithographs, have been widely and suc-cessfully used in medicine. Current researchactivit ies include the automatic computationand reproduct ion of prostheses or missingbones, based on geometric reconstruction. Inparticular, the symmetry of the human anato-my is exploited to compute missing pieces,wh i c h a re then r epro duc ed by m i l l i ngmachines and implanted by surgeons .Additionally, 3D hardcopies help the study ofanatomic deviations or malformations in theapproaches of complex surgical procedures,such as in facial surgery.

Besides mere geometric reconstruction ofanatomic structures, graphics researchers

developed models to simulate the physics ofthe human body. Presently, one of the mostchallenging and attractive subfields here is thefast and accurate computation of the mechan-ical behavior o f soft tissue, whi ch is an essen-t ia l prerequis i te for the vas t and most lyunexplored area of v i r tual and s imulatedsurgery. Al tho ugh we are s t i l l far f rom amature computational model, some existingsoft-t issue models perform surprisingly welland therefore can be considered a first steptoward s more sophisticated approaches. Anexample from facial surgery s imulat ion isgiven in figure 3, where pre- and postsurgicalfacial shapes of a case study are presented fora lowe r jaw bone repositioning. In this partic-ular application, highly realistic images of thepost-surgical appearance of the patient's faceare o f enormo us impor tan c e t o r e l axpatient's fear.The Future:VirtualOperation and SurgerySimulationUndoubtedly, future key applications of graph-ics in the medical field include operation plan-ning and surgery simulation. Here, we foreseefully immersive real-time simulation environ-ments in which surgeons can learn, plan andrehearse complex operations using individualpa t i en t mode l s and be ing s uppor ted bysophisticated input devices, such as virtualscalpels. The design of these types of simula-tion environments is a highly interdisciplinaryproject which requires input from a variety ofdisciplines including graphics, vision, robotics,numeric analysis, mechanics and material sci-ences, hardware design and, needless to say,medic ine. However, unl ike any other disc i-pline, comp uter graphics wil l be paramou nt,

Figure 3: Physics-based mode ling o f facial s0~ t/ssue:a) P resurgical facial shap e profile, b) Postsurgical profile, c) Postsurgical frontal view. Data source:Visible Hum an Project, Courtes yNational Library of Medicine. See page 101 fo r imag e in fu l l co/or.

54 February1998Com puter Graphics

8/7/2019 p53-gross

3/5

since i t wi l l take the role of the system archi-tect .

T h e c o n c e p t u a l c o m p o n e n t s o f a nadvanced surgery s imulat ion e nviron men t canbe summarized as follows: In a first step, rawdata sets have co be acquired by highly accu-rate 3D medical imaging or v is ion systems.S u b s e q u e n t p r e p r o c e s s i n g s t e p s e x t r a c tanatomic substructures and create geometricmodels of the pat ient au~ributed with respec-t i ve mater ia l parameters, s temming f rom anapprop r iate mater ial database.A sophisticatedmode le r a l lows the surgeon t o mod i f y th egeometry and the top olog y of indiv idual partsof the der ived model whi le s imulat ing cuts,bon e repos i t io ning, t ransplants etc . Forcefeedbac k as a f unc t i on o f t he under l y i ngmater ial has to be compu ted and interpola tedto mee t the high update rates which are nec-essary to beat the tem pora l resolut ion of thetacti le sensory channel. Tissue volume forcesand deformat ion f ie lds as wel l as col l i s iondetec t ion must be computed in real t ime,s ince they convey the parameters For v isualand fo rce feedback. In essence, Fast app roxi-mat ions of the under ly ing d i f ferent ia l equa-t ions have to be Found. Al thou gh fo r someapp lication s, such as facial surgery simulat ion,mor e e xpensive and accurate solut ion strate-gies c ou ld c ompute t he ex ac t de fo rmat i onf ields in batch m ode, the design o f ap propr i -ate real- t ime engines remains the most chal-lenging part o f the s imulat ion. As an op t ion,appropr iate render ing algor i thms might gen-erate p hotoreal is t ic st il l images of the pat ient.D a t a A c q u i s i t i o n a n d A n a ly s i sSince CT and MR scanners sti l l have l imitedresolut ion, h ighly accurate 3D sur face dataacquis i t ion is fundam ental For some appl ica-tio n scenarios. In this co ntex t, fast and robus t3D range f inding methods are necessary cosatisfy the resolution constraints imposed bythe s imu la t i on env i r onment . Fur t hermore ,color and texture samples must be recordedco equip subsequent render ing methods.Especially whe n using both volu me and sur-Face data, manifold re gistration and a lignme ntpro ble ms arise, inclu ding surface-surface, sur -face-volume and volume-volume registrat ion.Robus t (semi )automat ic methods are des i r -able.Since th e early days of medical imaging, seg-mentat ion and feature ex t rac t ion have los tn o n e o f t h e i r i m p o r t a n c e , a n d d e s p i tedecades of research, there is no robust andful ly automatic meth od in s ight- For matchingproblems, semiautomatic strategies might bean appropr iate al ternat ive and the col labora-tive effort of computer graphics and comput-e r v i s i on r es earc hers i s nec es s ary t oaccomplish this goal.

T h e H u m a n C o m p u t e r I n t e r fa c eThe design of a sophist icated natural humancomputer inter face is cer ta in ly one of thekeys to successful simulat ion.The i l lus ion of aful l medical work ing environ men t can only bec rea ted w i t h t he he lp o f adv anc ed v i r t ua lr e a l i t y h a r d w a r e . C o n t e m p o r a r y o u t p u tdev ices , such as head-moun ted displays,C a v e s , V i s d o m e s a n d W o rk b e n c h es arepromis ing tools to s tudy the behav ior andacceptance of medical users. Howeve r , weforese e smart, small, l ightweig ht and very highresolut ion eyeglass displays as the ul t imatedevices for mediat ing v isual in format ion tothe surge on.W e bel ieve that future d isplaytechnology wi l l prov ide a new generation ofsophisticated solutions.

Much more important and fundamental Forany surgeon than the display i tsel f is the pro-v is ion of h ighly accurate tac t i le and forcefeedback informat ion. Therefore, we requi rehighly accurate hapt ic interfaces, which cap-t u r e t h e r e s p o n s e s o f h u m a n t i s s u e t omechanical s t imul i . One of the major prob-lems, however, is that th e haptic device mustb e t h o r o u g h l y t a i l o r e d t o t h e u n d e r l y i n gappl icat ion. In facial surgery s imulat ion, fori ns tanc e , c omple te l y d i f f e ren t s e tups a rerequired than in laparoscopy. Moreover, com-p lex s urg i c a l procedures u s u a ll y e m p l o yextensive sets of individual mechanical tools.Al though var ious hapt ic inter faces are avai l -able in research labs or on the market andenable f i rs t expe r imenta l steps into the r ightd i rec t ion, we are s t i l l far f rom convenientsolut ions . Al l in al l, the design of indiv idu alforce feedback devices is strongly influencedby the appl icat ion context and by the under-lying physics. OptimJ7.arion to users' demandsrequires multiple iterations and design cycles.Therefore, successful solut ions can only bed e v e l o p e d i n a t i g h t c o l l a b o r a t i o n o fr es earc hers f r om m ed i c i ne , r obo t i c s andcomputer graphics.M e c h a n i c s a n d N u m e r i c A n a l y si sThe di f ferent ial equat ions governing any volu-metr ic t issue deformat ion have their roots inmechanics. In the past, l inear volu me tric strainand deForma tion models have been extensive-ly inves t igated and are w idely avai lable forv a r i o u s k i n d s o f m a t e r i a l s i m u l a t i o n . Insurgery simulation, however, large strains anddeformat ions occur and the mater ial tensorsbecome non-constant_The resul t ing phenom-ena are highly nonl inear and extremely di f f i -c u l t t o m o d e l . O t h e r e f f e ct s r e l a t e t ocompressibi l i ty of human soft tissue.The highpercentage of l iquid makes i t almost incom -pressible; local t issue force s generated dur ingsurgery force l iquid to stream ou t and to dis-to r t the mechanical behavior of the mater ial .

Besides mathematical models, appropmater ia l parameter databases are of gimpor tance in surgery s imulat ion. Moduelast ic ity or non- l inear i ty are usual ly funco f age, s e x , e t h n i c g r o u p a n d o t hUnfor t una te l y , i t i s ex t r emel y d i f f i c u lobtain the desired parameters and extenexper imental research has to be pursHere, we need the help of mater ial sc ienco provide us with appropr iate sets of pmeters and interpolat ion models For indual patients and tissue types.

The strategies for opt imal solut ions of t ial di f ferent ial equat ions are cr i t ical ly dedent on t he c omplex i t y o f t he under lmechanical model. In part icular , in ordeachieve real- t ime response we have co ance mathematical accuracy versus compt i o n a l e f f o r t . N u m e r i c a n a l y s t s hdeveloped var ious c lasses of solvers, wFEM is only one prom inent example. Usimpl i fy ing the physics, appropr iate solustrategies have to be designed in c lose cerat ion w ith numeric analysts. Here, hieraprogression an d localization might be somthe key word s to success.T h e P a r ~ n o u n t R o le o f C o m p u t e rG r a p h i c sTh e part icular at t ract iveness of highly cplex surgery s imulat ion environments fol ies in the versat i l i ty and depth of indivresearch p r o b l e m s t o u c h i n g o r c o v ealmost every impor tant subfie ld of co mpgraphics. The refore , we f inal ly i l luminateparam ount role of graphics in surgery s imt ion and br ief l y sketch some of the mresearch challenges: M odeling of geometry and topolo~. The bof any sophisticated surgery simulationtem is the efficient mathem atical descri

of the under ly ing geom et ry and topolHere, most o f the m ode l has co be descin a full volum etric settin ~ Sophisti,~tedcretJzations and ap pro xima tion modelsessential preprocessing steps For advanumeric procedures, which compute tidefo rmat i on . Con fo rm i t y and po l y nodegree of individual elements strongly athe complex i ty and accuracy of numsolut ions. The refore w e have to adaptmodels to subsequent numerical stratewhe re piecewise l inear approximat ionsmost ly insuff ic ient.As a consequence,challenge is to develop new generat ionh igher o rder v o l umet r i c approx imat iwhich do not impose unnecessary topocal restrictions on our model.

Ed/ting of complex structures:With app rate approx imat ion method s in place, ane r t a s k i s t o d e s i g n p o w e r f u l e d ia l lowing the user to modi fy the geom

ComputerGraphlcs February199

8/7/2019 p53-gross

4/5

and topo logy of anatomic structures. Againmere surface editing is certainly not suffi-cient for cutting and repositioning of indi-vidual pieces of soft tissue. Functions suchas a zoom into substructures are highlydesirable and enable a surgeon to operateon different scales in the full volumetricsett ing. Therefo re, elaborate data struc-tures have to be designed giving efficientaccess to, and maintaining individual primi-t i ves of the under ly ing approx imat ion.Furthermore, these data types will have tomaintain all necessary parameters of thephysics-based models, which in turn mustinteract with the e ditor in real t ime.

Simplification and adaptation of physical mod-els: One of our core challenges in surgerysimulation is to understand and simplify thephysics of the simulation process.The questfor real- t ime update rates in v isual andmechanical feedback forces us to developsophisticated solutions, which have to bal-ance computational complexity with accu-racy. To this end, we have to tho rou ghl yinvestigate and analyse the governing equa-t ions and to s tudy thei r er ror bounds .Based on a p ro found k nowledge andobservat ion of the physical phenomena,new approx imate s o lu t i ons mus t bedesigned. Many di f ferent strategies andparameters have to be investigated andevaluated against each other, such as finiteelement versus finite difference, flat versushierarchy, local versus global or polynomialdegree versus spatial resolution. Requiredupdate rates for visual and force feedbackrender ing might be decoupled from thesimulation engine and appropriate tempo-ra l i n t e rpo la t i on c ou ld c ompute t hedesired in-betweens. In addi t ion we wi l lhave to consider next generations graphicshardware to accelerate the env is ionedalgorithms.

Image generation: Generally, fast renderingalgorithms are irreplaceable in surgery sim-u la t i on . Howev er , w i t h t he des ign o fadvanced volumetric approximations newtypes of possibly unstructured graphicsprimitives wil l enter the scene and appro-pr iate hardware suppor t wi l l be h ighlydesirable. Thus both surface and volumerendering of higher orde r p rimitives wil l beof great interest.The required visual quality,however, depends highly on the applicationconte xt. The mostly lub ricant surfaces ofthe interior of human bodies, for instance,are highly specular and wil l ask for hard-ware support for more sophisticated shad-i ng mode l s . Fur t hermore , bump,displacement and texture databases orprocedural models for human skin and softtissue have to be created. Here, multipassmethods might be a promising approach.Apart from the real-time constraint, someapplications require extremely sophisticat-ed i l luminat ion models. In facial surgerysimulation, for instance, the rendering ofphotorealistically looking images might becomputed off-l ine by raytracing of the high-er order primitives envisioned earlier. Herewe need advanced reflection and scatteringmodels for facial skin, which will also varyas a function of age, sex and oth er parame-ters. In addition, high quality rendering ofhuman ha ir is still a wid ely ope n issue.

System Design and Layout: Computer graph-ics researchers wi l l undoubtedly be theprincipal designers of future surgery simu-lat ion systems. Thus, besides the meredevelopment, tuning and tai lor ing of indi-vidual algorithms graphics research wi ll alsohave to cover overall systems layout andopt imisat ion. In part icular , global designissues, such as the interactions of editor,render ing, s imulat ion engine and forcefeedback, are of fundamental importance inhighly interconnected, complex systemsand must also be considered by graphicsresearchers.

SummaryIn summary, from the earliest days, the med-ical field has been one of the most appealingareas for computer graphics research. Beingthe chief engineer of next generation's med-ical training and simulation systems, computergraphics wi l l help to improve our presenthealth system.Thus it will provide a significantcontribution to our society -- and to modernhuman life.Ma rku s H . Gro s s is full Professor at theComputer Science Department of the SwissFederal Ins t i tute of Technology (ETH) inZ~irich. He received a degree in Electrical andCom pute r Engineering in 1986 and a Ph.D. inComputer Graphics and Image Analysis in1989, bo th f r om the Un i v er s i t y o fSaarbr~icken, Germany. From 1990 to 1994,he was with the Computer Graphics Centerin Darmstadt , where he es tabl ished anddirected the Visual Com put ing Group. Hisc ur ren t research interests inc lude physicsbased and multiresolution methods for graph-ics wi th applications in the medical field.Markus H.GrossDepartment of Computer ScienceSwiss Federal Insitute o f Technology (ETH)CH 8092 Zurich, SwitzerlandT e l : +41-1-632-7114Fax: +41 -1-632- I 172Email:grossm@inf.ethz.chhttpd lwww.inf.ethz.chldepartrnent/ISIcgl

56 February1998Compu ter Graphics

8/7/2019 p53-gross

5/5

F R O M T H E, . . . .

E D I T O R

Gordon CameronSOFTIMA6E, Inc .When we look at the current state of comput-er graphics i t is easy to take so much fo rgranted. By doing so, we forg et th e rema rkableflurry of research and development undertak-en over the past quarter century (and more)which has led us to a present where comput-er-generated imagery mesmerizes us at thecinema, visual izat ions aid in our hospitals,graphical tools aid in the design of o ur homesand vehicles and countless other applicationsof computer graphics impact our daily life. I~ebeen involved in computers and graphics inone way or another for around 16 years (sincemy I~rst dabblings at the age of 13 on a neigh-bour's Apple II Europlus, and implem entation ofa painfully basic drawing program in 0.5K on aZ Xt il) . It is amazing to me how Par we havecome since '82, and equally amazing how wegot to that stage from preRy much nothingove r the preceding few decades.This year marks the official 25th birthdayof the 51GGRAPH conference (Ed/t0r's note:although the organization is older m see CarlMachover's column on page 25). To kick offth e celebrations, I asked FranTois 5illion (apioneer in graphics, particularly in the fields ofrendering and global i l lumination) if he wouldlike to guest edit an issue which would ask aselection of experts from a cross section ofthe community to offer their musings on thepast few co mpu ter graphic decades, as wel l aslook into their personal crystal balls to giveus thei r opinion on wha t lies in store m a'~orward-looking retrospective:' Much to mydelight, Francois decided co rise to the chal.lenge and collect and collate for us a fantasticselect ion of art ic les wri t ten by some of theleading researchers from a wid e range ofcomputer graphics disciplines. My thanks goout to all the authors and FranTois for pre-senting a fascinat ing view on the world ofcomp uter graphics -- past, present and possi-ble future!This February issue also sees the debu t of anew regular education column , as well as a stu-dent gallery to showcase the works of thosestudyin g in educational establishments aroundthe globe. Many thanks and best wishes ro thenew additions to the Computer Graphicscolum-nist family, Rosalee Wolfe, Jodi Giroux-Lang,Lynn Pocock and Ka ren Sullivan. On a column-

related note, the real-time column is on rata.t ion for this issue,but will return in Hay.Next time around, we investicate an areathat has had tremendous impact on societyand the computer graphics world over thepast decades -- compute r taming.To coincidewith the 25th anniversary, an earlyAugust issuewil l act as a special history document t iedspecifically to the conference.

The world of computer graphics continuesto evolve at a stardinI pace. I hope you enjoythis trip down memory lane and look to thefuture.

Gordon CameronSoftware DevelopmentSOFTIMAGE, Inc.3510 boul.$t-LaurentSuite 400Montreal, Quebec H2X 2V2CanadaTel: + I-514-845-1636 ex_3445Fmc + I-514-845-5676En~ l: Iordon_carnemn@s/araph.arg