Virtual Reality Techniques for theVisualization of Biomedical Imaging Data

M. A. Shaw, w. B. Spiliman Jr.,K. E. Meissnera, Gabbardc

aThe Optical Sciences and Engineering Research Center, VirginiaPolytechnic Institute and State University; of Physics,cResearch Graduate Studies, Virginia Polytechnic Institute and State

University

Abstract

The Optical Sciences & Engineering Research Center (OSER) at Virginia Polytechnic andState University investigates advanced laser surgery optics, biocompatible material forimplants, and diagnostic patches and other diagnostic and drug delivery tools. The Centeremploys optics to provide new biological research tools for visualization, measurement,analysis and manipulation. The Center' s Research into Multispectral Medical Analysis andVisualization techniques will allow human and veterinary medical professionals to diagnosevarious conditions of the body in much the same way that satellite information is used tostudy earth resources. Each pixel in the image has an associated spectra. Advanced imageanalysis techniques are combined with cross-correlation of the spectra with signatures ofknown conditions, allowing automated diagnostic assistance to physicians. The analysisand visualization system consists of five components: data acquisition, data storage, datastandardization, data analysis, and data visualization. OSER research efforts will bedirected toward investigations of these system components as an integrated tool for nextgeneration medical diagnosis. OSER will research critical data quality and data storageissues, mult-spectral sensor technologies, data analysis techniques, and diagnosticvisualization systems including the VT-CAVE,(www.cave.vt.edu). The VT-CAVE isVirginia Tech's configuration of Fakespace Systems, Inc Virtual Reality system.

Introduction

The Optical Sciences and Engineering Research Center (OSER) was recently formed atVirginia Tech. The center is a collaborative effort between Virginia Tech and the CarilionBiomedical Institute. OSER is specifically tasked with conducting research andengineering activities involving optics and other disciplines to create knowledge andtechnology to benefit the medical, biomedical and veterinary fields, while supporting thepractical goals of improving services and reducing the costs of health care.

Multispectral medical analysis and visualization can allow human and veterinary medicalprofessionals to diagnose various conditions of the body in much the same way thatinformation from satellites can be studied to yield information about the earth's surface(vegetation, pollution, and mineral resources, for example). Each pixel in the image has anassociated spectra. Advanced image analysis techniques are combined with cross-correlation of the spectra with signatures of known conditions, allowing automateddiagnostic assistance to physicians.

Biomarkers and Biological Spectral Imaging, Gregory H. Bearman, Darryl J. Bornhop,Richard M. Levenson, Editors, Proceedings of SPIE Vol. 4259 (2001)

© 2001 SPIE · 1605-7422/01/$15.0024

Multispectral Medical Vizualization

Advances in sensor technology using multispectral and hyperspectral data acquistionsystems can be combined with immersive data visualization techniques to produceenhanced reconstructions of a patient's condition for diagnostic purposes. These emergingsystems will require developments in sensors, data standards, data management,computational data analysis, and improved data visualization systems.

Advances in multispectral and hyperspectral technology produce data rich images. Theseimages can be combined using computational techniques to produce information richproducts. Data compression and transmittal standards and processes for these images mustbe developed to ensure quality is maintained. Furthermore, the computational algorithmscombined with immersive stereoscopic visualization techniques can result in enhanceddiagnostic capabilities. A rigorous information system is required to support this researchand development effort. The system must be modifiable, flexible, robust, scalable,maintainable, and have a set of supporting process in place to ensure data integrity, qualitycontrol, and availability.

The critical challenge is the development of techniques that will provide the clinician anenhanced diagnostic space. This will require development of new methods for informationportrayal derived from new sensor technologies. Display of enhanced three dimensionalimage elements within an immersive environment and interacive analysis techniques mustbe developed. Finally, this new diagnostic space must provide access to all availablepatient and other critical information within the work area to provide a complete decisionsupport capability.

Medical Visualization System Components

The goal of a multispectral data visualization system is to provide enhanced diagnosiscapabilities for use by the medical practitioner. The system consists of five components: 1.multispectral data acquisition; 2. data management; 3. data reduction; 4. data analysis; and5. stereoscopic visualization, (Figure 1.). The data acquisition and visualization systemswill provide enhanced capabilities for portraying multispectral data abstractions within anatural three-dimensional stereoscopic immersive display system.

Multispectral Data Acquisition

Multispectral imaging systems acquire full spectral data at each pixel of an image. Thus, athree-dimensional data cube is built with the axes being x, y and wavelength (X). Themultispectral data acquisition system used at OSER was custom built by OKSI (Torrance,CA). It consists of a TE-cooled, blue enhanced Silicon CCD array, a liquid crystal tunablefilter (LCTF), camera optics, and a laptop computer for control and storage. Either avisible LCTF (400nm — 720nm) or a near-infrared LCTF (600nm — lO5Onm) may be fittedon the camera system. The image acquisition time depends on the integration time at eachwavelength band, but is generally on the order of one minute. An image cube is 512 x 512x 33 (39) for the visible (NIR) and occupies approximately 17MB (20MB).

Data Management

The massive amounts of data generated by the enhanced sensor systems must be managedand stored in a manner that preserves the quality and unique identifying characteristics ofeach data set. The meta data describing the sensor data must include machine calibrationrecords, patient identification, patient condition at the time of the measurement, a complete

Proc. SPIE Vol. 4259 25

Figure 1 . Component Diagram of a multispectral medical data acquisition, analysis, andVisualization system.

record of data reduction and/or computational modification to the original data. Thisinformation is critical to ensuring data quality is maintained by preserving the chain ofevents and processes that result in the analytical or visualization product being evaluatedby the clinician.

The OSER system will support the research effort and therefore data provenance is criticalwhen evaluating data anomalies and determination of the efficacy of computationalalgorithms. The data maintained by the OSER data management system will be invaluablefor evaluating new diagnostic tolls as techniques are developed by the research team.

Also included in the data structure will be a complete set of patient records. Thisinformation will be available to the analyst and diagnostician during computationalanalysis and also within the virtual environment to provide an effective and completevirtual diagnostic space.

The availability of this information is critical to the proper analysis of factors that maycontribute to the variances in the sensing system. Changes in tissue response due to thephysiological state of the patient due must be available to the development team.

Data Reduction

Data reduction transforms raw multispectral sensor derived data into more useful forms.The techniques used during transformation will be preserved along with the resulting dataset to ensure data provenance is preserved for quality control. The data reduction productswill include data transformation into industry standard formats for data analysis, imagevisualization, and three-dimension virtual reconstruction.

Proc. SPIE Vol. 425926

Data Analysis

The data analysis component will consist of integrated commercial-off-the-shelf (COTS),open-source, and custom software to be used for computational analysis. The analyticalcomponent will be tightly coupled with the visualization system to allow interactivestereoscopic visualization. This capability will be available both at desktop workstationand with the VT-CAVE immersive environment.

The data analysis toolkit will include both COTS and custom computer applications.Appropriate software applications used by the remote sensing industry will be applied (andcustomized as necessary) to support the multispectral diagnostic development effort. It isexpected that research into the efficacy of applying these reliable remote sensingtechniques to medical multispectral data is a promising research area.

Visualization

Data visualization is the science of turning sets of complex data into high-density visualinformation and understanding. Current and emerging technologies allow the generation ofvirtual environments (VEs) that provide access to traditional data visualization analysistechniques and the ability to combine these capabilities with scaled digital models, video,and other multimedia within an interactive immersive, stereographic environment.

Advanced visualization has become a mainstream tool in applications where complex datasets need to be studied and manipulated. The ability to provide human-scale display,immersion, visual databases, spatial integration, and collaboration solutions will have asignificant impact on clinical and diagnostic procedures. The technology providesimmediate sensation of the spatial relationships between data. Data can be portrayed usingtechniques that can either abstract complex data into recognizable patterns or portrayinformation in a natural and therefore more understandable context.

The use of VEs for data fusion is an area of active research. The technology currentlyprovides capabilities for:

. interactive work environments with access to traditional data analysis techniques,S any combination of spatially related data (or abstractions of the data),

computational products, or data classifications,. 3D data models from other sources,. 3D models produced by voxelation of image slices. video,S images,. audio and 3D audio,. scale models of environments, structures, sensors, and devices,. interactivity at any scale or from any viewpoint,. textual labels, visual, and audio information that would help investigate,

understand, and communicate a medical investigation,. interactive devices that can be developed specifically to aid in the dissection of

discrete data images,• interactive tools to manipulate and navigate the VE scene graph, and,• interactive collaborative work sessions with remote sites.

The OSER will be using desktop workstations and Virginia Tech's VT-CAVE immersivesystem for medical visualization and diagnostic analysis. The CAVE(tm) is a multi-person,room-sized, high-resolution, 3D video and audio environment. In the current configuration,graphics are rear projected in stereo onto three walls and the floor, and viewed with stereo

Proc. SPIE Vol. 4259 27

glasses. As a viewer wearing a position sensor moves within its display boundaries, thecorrect perspective and stereo projections of the environment are updated by asupercomputer, and the images move with and surround the viewer. Hence stereoprojections create 3D images that appear to have a presence both inside and outside theprojection-room continuously. To the viewer with stereo glasses the projection screensbecome transparent and the 3D image space appears to extend to infinity

Specifically, the CAVE(tm) is a theater lOxlOx9 feet, made up of three rear-projectionscreens for the front, right and left walls and a down-projection screen for the floor.Electrohome Marquis 8000 projectors throw full-color workstation fields (1024x768stereo) at 96 Hz onto the screens, giving approximately 2,000 linear pixel resolution to thesurrounding composite image. Computer-controlled audio provides a sonificationcapability to multiple speakers. A user's head and hand are tracked with Ascension tetheredelectro magnetic sensors. Stereographics' LCD stereo shutter glasses are used to separatethe alternate fields going to the eyes. A Silicon Graphics Power Onyx with three InfiniteReality Engines is used to create the imagery that is projected onto the walls and floor.(http://www.sv.vt.edu/future/vt-cave/whatis/)

Conclusion

The goal of a multispectral data visualization system is to provide enhanced diagnosiscapabilities for use by the medical practitioner. The system consists of five components: 1.multispectral data acquisition; 2. data management; 3. data reduction; 4. data analysis; and5. stereoscopic visualization. The full spectra imaging system is combined with the dataanalysis and interactive visualization to provide means for evaluating sensor response,computational alogorithms, and efficacy of the portrayal techniques as a diagnostic tool.The data management component provides a platform for storage of raw and computationaldata and the resulting visualization products. This component also provides the patientinformation necessary to a comprehensive diagnostic space. Finally the data reductioncomponent provides the required procedures for standardizing and reducing the dataacquired from the sensor systems. The stereoscopic workstations and the VT-CAVE, amulti-person, room-sized, high-resolution, 3D video and audio environment, will be usedfor medical data visualizations and research into diagnostic procedures.

In conjuction with this data visualization work, we investigate an intriguing method foranalyzing hyperspectral data in talk 4259-10, "Cellular Automata for the Analysis ofBiomedical Hyperspectral Images", later in this session. Here, cellular automata are usedto rapidly scan hyperspectral images and quantify the extent of conditions of medicalinterest. This technique can lead to a large reduction in the computational time spentanalyzing large hyperspectral images.

Proc. SPIE Vol. 425928