Cluster Computing 4, 79–87, 2001 2001 Kluwer Academic Publishers. Manufactured in The Netherlands.

CAVEStudy: An Infrastructure for Computational Steering andMeasuring in Virtual Reality Environments

LUC RENAMBOT and HENRI E. BALDivision of Mathematics and Computer Science, Faculty of Sciences, Vrije Universiteit, De Boelelaan 1081, 1081 HV Amsterdam, The Netherlands

DESMOND GERMANS and HANS J.W. SPOELDERDivision of Physics and Astronomy, Faculty of Sciences, Vrije Universiteit, De Boelelaan 1081, 1081 HV Amsterdam, The Netherlands

Abstract. We present theCAVEStudysystem that enables scientists to interactively steer a simulation from a virtual reality (VR) environ-ment. No modification to the source code is necessary.CAVEStudyallows interactive and immersive analysis of a simulation running ona remote computer. Using a high-level description of the simulation, the system generates the communication layer (based on CAVERN-Soft) needed to control the execution and to gather data at runtime. We describe three case-studies implemented withCAVEStudy: soccersimulation, diode laser simulation, and molecular dynamics. In addition, we briefly describe a new technique of virtual measuring whichcloses the loop between simulation, immersive visualization, interaction in simulation domain and steering, giving the scientist deeperinsight into the simulated phenomenon.

Keywords: computational steering, virtual reality, virtual measuring, interaction simulation

1. Introduction

High-speed networks and high performance graphics openopportunities for completely new types of applications. Asa result, the world of scientific computing is moving awayfrom the batch-oriented management to interactive pro-grams. Also, virtual reality (VR) systems are now com-mercially available, but so far scientists mainly use them foroff-line visualization of data sets produced by a simulationprogram. This simulation runs on a remote supercomputerwithout any user-control. To become widely used by sci-entists, virtual reality environments should provide tools toconnect, visualize, and control on-going simulations. Fur-thermore, a scientist wants to learn from his virtual realityexperience. To facilitate this, the environments should pro-vide means to interactively identify and quantify features ofthe simulated data from the visual domain [28]. Scientistsshould be able to measure the data, using a variety of meas-urement paradigms. This gives rise to a number of researchquestions: interaction, collaboration, steering, and measure-ment in virtual reality become central issues in the design ofvirtual environments for scientific applications.

Networked virtual environments have rapidly developedover the last few years. Numerous toolkits are now available[3,16,24,25]. Each one has some specific features, such ascollaboration, portability, and distributed simulations. Themost general toolkit, and the one most used, is CAVERN-Soft [14]. It provides networking and database functional-ities needed in a virtual reality system. Different skeletonsof VR application are proposed. The system is open but stillrequires substantial effort to build new applications.

A related research area is the control of a running simula-tion, referred to as computational steering [22]. It is defined

as interactive control over a simulation during its execution.The scientist can control a set of parameters of the programand react to the current results. Computational steering en-hances the productivity of the scientist by giving a problem-solving environment [4]. However, existing systems, suchas SCIRun [22] and CSE [30], are used to build new ap-plications or require modifications to the source code of theapplication, and thus are unsuitable if the source code is un-available.

In this paper, we describe a system calledCAVEStudy,based on the CAVERNSoft toolkit, that allows the scientistto steer a program from a virtual reality system without re-quiring any modification to the program. It enables an in-teractive and immersive analysis of simulations running onremote computers.CAVEStudyallows non-experts in VR tocouple their simulation to virtual environments.

The scientist models the simulation as a set of input, out-put, and graphical objects. These objects are the input pa-rameters of the simulation and the data produced. Givensuch a description, our system generates a wrapper aroundthe simulation to control its execution on a remote comput-ing system, which usually is a supercomputer or a cluster.The data produced by the simulation is then packed and sentto the computer hosting the virtual environment. A proxyfor the remote simulation is built to receive the data gener-ated by the simulation. This proxy is plugged into a virtualreality environment, where it updates the objects describedby the scientist. CAVERNSoft provides the communicationand persistence layer needed by our infrastructure. Our in-frastructure consists of a description language, a graphicalinterface to create description files, a code generator, and avirtual reality environment. Thus, it is possible to visualize

80 RENAMBOT ET AL.

and control the program directly in the domain space of thesimulation.

The main contributions of our work are as follows:

• a high-level steering system that does not require modi-fications to the source code;

• a VR framework to immerse the scientist in the simula-tion space;

• an experimental evaluation of the steering system andVR framework using various systems, such as an IBMSP2, a 128-node Myrinet cluster computer, and severalCAVEs and graphic workstations;

• three real applications (collaborative soccer, laser simu-lation, molecular dynamics visualization) used as case-studies.

This paper is structured as follows. In section 2, we sur-vey related research. In section 3, we introduce our system.Using a small example, we show how it facilitates the cou-pling of a simulation to a virtual environment. Some applica-tions built with our infrastructure are described in section 4.Section 4 briefly describes work on virtual measurement. Fi-nally, we present our conclusions.

2. Related work

Our work is related to a wide and active research area re-ferred to as theGrid, Metacomputing, or the next genera-tion high-performance computing infrastructure [4]. Sys-tems such asGlobus[5] or Legion [8] provide languages,tools, and environments to create new applications that werenot conceivable before, such as world-wide collaboration invirtual reality or real-time data-mining of large data sets. Be-low, we discuss networked virtual environments and steeringsystems.

2.1. Networked virtual environments

Interactive and collaborative visualization radically changethe way scientists use computer systems [4]. With interac-tive visualization, a user can interact with a program in its vi-sual domain. Distributed collaboration allows multiple usersat different geographic locations to cooperate, by interact-ing in real-time through a shared application [14,26]. Sev-eral toolkits have been designed to provide networked virtualenvironments. They provide functionalities such as com-munication, shared-state management, collaboration mecha-nisms, and 3D graphics.

NPSNET [16] focussed on large-scale virtual environ-ments for battle simulations. The entities simulated are, forinstance, tanks, missiles, or soldiers. The database is repli-cated and split into areas of interest, which permits to reducethe communication. The DIS (Distributed Interactive Sim-ulation, which allows to model remote entities) protocol isused in conjunction with multicast to achieve a better scal-ability. BrickNet [25] is a toolkit that supports graphical,behavioral, and network modeling of virtual worlds using a

shared object paradigm. The network medium is a set ofinterconnected servers. Objects can be shared by severalworlds. The main goal was to build collaborative designenvironments. MR Toolkit [24] maintains a shared worldin a distributed manner, using frequent UDP packets to up-date the world. Using callback functions, the state updatesare propagated across the whole data set. DIVE [3] focuson some network isssues to obtain large-scale virtual en-vironments, using mainly reliable multicast and spatial de-composition of the virtual world. Massive [7] is an object-oriented environment that provides multicast, region of in-terests, 3D graphics support and real-time data streaming.Bamboo [32] focusses on dynamically extensible and real-time networked environments which should configure itself,allowing to add new functionalities or to discover new envi-ronments at runtime. Avocado [29], an object-oriented sys-tem based on the Performer library, replicates and updatesthe scene graph among the remote users using a group com-munication approach. The shared-data structures are keptconsistent by some global operation such as deletion or in-sertion. CAVERNSoft [14] provides networking and data-base functionalities needed in a virtual reality system. It alsocomes with different skeletons of VR applications calledtemplates. The system is open but remains very basic. Itstill requires a substantial effort to build a new application.

2.2. Steering systems

Many existing applications restrict the interaction to the vi-sualization process (e.g., the direction of view, the zoom fac-tor). A more advanced form of interaction, referred to ascomputational steering, allows the user to interact with thesimulation process. Several systems support steering [18],but they typically provide only low-level interactions and re-quire users to monitor or change the application program’sinternal variables. We do not address the issue of fine grainsteering offered by some systems [11], but we focus on thecomplete execution of a simulation for a given set of inputs.However, we also examine simulations that produce resultsduring the execution (i.e., iterative methods), shipping theintermediate results as output.

SCIRun [22] is a problem-solving environment for com-putational science. It provides an integrated framework toconstruct, steer and study large applications. It has beenused in domains such as medicine or geophysics. The sci-entist constructs the simulation by connecting a set of mod-ules using a data flow paradigm. Each module is a com-piled function written in C, C++ or Fortran wrapped in aTcl/Tk interface. A central module schedules and synchro-nizes the system. SCIRun allows the scientist to modify theparameters of the simulation, which triggers the evaluationof the connected modules. Some 3D visualization modulesare provided to present the data. Such a system is designedto build and test, in a closed loop, new algorithms, but it isnot well suited to study existing applications. Moreover, thegraphic output is used to visualize results, but not to steer theapplication. VASE [11] allows the programmer to design a

COMPUTATIONAL STEERING AND MEASURING IN VIRTUAL REALITY 81

steerable program through the specification of the programstructure. This structure is a graph made of blocks of sourcecode and arcs representing the control. The program can bestopped at run-time between blocks and some user scripts(C-like) are executed. At these points, data can be retrievedand modified. The visualization is managed by an externallibrary such as IRIS Explorer. Magellan [31] requires theuser to annotate the source code to produce an abstract viewof the program, consisting of steering parameters and outputdata. These objects can be probed, modified at run-time, orprovide a breakpoint. A master program can control severalsteered program. A graphical interface can generate moni-toring and steering commands. The visualization has to bedone through an external library. CUMULVS [10] is de-signed to study PVM-based parallel applications. Within theapplication, data distribution and steering parameters haveto be declared. Steering commands are synchronized andapplied to all tasks. Moreover, CUMULVS allows severalusers to connect to the same application. In such a col-laboration mode, users can lock some steering parameters.VIPER [23] allows the steering of massively parallel appli-cations, for instance computational fluid dynamics. VIPERrequires the parameters of the simulation to be annotated inthe source code. During the execution, the system extractsand transfers data from the application to a remote multi-threaded server. A graphical interface is provided to managedata gathered. Finally, CSE [30] focuses on existing applica-tions. It allows several programs to be integrated within onedistributed steering environment. An application is anno-tated to specify input and output objects, and then connectedto a server. This server acts as a data manager for all thecomponents. A component can subscribe to the output of an-other component. The server manages the data exchange andthe synchronization of the whole system. Data managementand visualization components are available. Moreover, aneditor is available to design graphical data representations.Such a representation is used as direct steering mechanism,instead of a simple classical visualization.

Related to the idea of a virtual workspace for the scien-tist, the concept of avirtual laboratory is explored in re-search projects like VIDL [20] and Cactus [1]. The authorstry to provide a collaborative and problem-solving environ-ment for large-scale simulations (as computational fluid dy-namics or relativistic astrophysics). Nevertheless, our mainconclusion is that no environment provides all the function-alities needed for an interactive and immersive steering en-vironment, given that we do not want to modify the sourcecode. We designedCAVEStudyin that spirit.

3. CAVEStudy

Our goal is to build a system that combines the power andthe functionalities of computational steering and virtual re-ality. Such an environment, combining both the control overa simulation and the immersion in the data space, does notexist yet as far as we know.

Figure 1. Software layers.

CAVEStudymainly consists of two parts: a code genera-tor and a VR framework, as shown in figure 1. To minimizethe programming for the control of the simulation and thedata management, the user has to describe the simulation bya description file. This file is processed to generate two ob-jects, a proxy and a server. The simulation is wrapped intoa server object to control its execution. The server’s inter-face provides methods to start, stop, pause, and resume thesimulation. The data generated by the simulation are auto-matically propagated to the proxy object. This object can beseen as a local copy of the remote simulation. Through thenetwork, it reflects the input values and the commands to theserver. Furthermore, it manages the incoming data from thesimulation and presents them to the VR framework.

3.1. Simulation description

To model a simulation, we designed a description languagewhich allows the user to describe the input parameters andthe output data of the simulation. It can be seen as a light-weight and dedicated CORBA-like interface description lan-guage. Several sections must be present in such a file: a de-scription of the simulation program, a set of input parame-ters, and a set of output data. Graphical objects can also bedescribed. An example is presented in figure 2. A graphicalinterface is available to describe such a file.

To be able to start the simulation, the user must give in-formation about the executable such as its name and direc-tory. This is specified in theSimulation section. The way tofeed the simulation with the input parameters should also bespecified, for example as a command-line or with an inputfile. The system should know how to acquire the output dataproduced, for instance on the standard output or from a file.Data manipulated by the simulation can be described usingeither the provided built-in types or by some types definedby the user. The basic types defined in the system are:long,floating-point, string, 2D-point, and3D-point. New typescan be described by combining basic types. An example canbe seen in figure 2 (typePointTime). All these types can beused in scalar, list, and matrix objects.

Input parameters are described by a name, a type, a di-mension (scalar, list, or matrix) and an optional input value

82 RENAMBOT ET AL.

Project{ Name BouncingBall}Struct PointTime{ vector3D point

float time}Simulation{ Name bounce

Executable bounce.exeDirectory SimulationProcesses 1InputType commandlineInputValue ""OutputType fileOutputValue position.data

}Input vector3D{ Name init_position

Value [ 1.0, 0.0, 0.0 ]}Input int{ Name nb_step

Value 1000}Output PointTime{ Name current_position}Graphic{ Name the_ball

Type sphereValue current_positionColor [ 1.0, 0.0, 0.0, 0.0 ]

}

Figure 2. A sample description file.

when applicable (scalar object). Any number of input para-meters can be specified by severalInput sections. Outputobjects are given in a similar way usingOutput sections.Finally, it is possible to specify graphical objects. Such anobject is the graphical representation of an output data pro-duced by the simulation, and its type can be selected amongseveral ones (sphere, line, surface, etc.). The user is able tointeractively modify the values of input parameters to steervisually the simulation.

3.2. Code generation

We wrote a code generator for the description files. It gener-ates C++ code for the CAVERNSoft [14] network layer, asshown on the right part of figure 1. We selected CAVERN-Soft because it provides functionalities to build networkedvirtual environments and because it is widely used in the VRcommunity. CAVERNSoft, based on the Nexus communi-cation library from Globus, uses a “publish and subscribe”paradigm. A site can define keys to publish its own data,and a remote site that subscribes to these keys will auto-matically receive the data through callback functions. Thismechanism can be used for small data (tracker data) to largedata sets (data-mining) using different policies. CAVERN-Soft supports persistence, thread management, and networkprotocols (TCP/UDP/multicast). It also provides some fa-cilities for avatars and 3D models using the SGIPerformerlibrary. Currently, our code generator can produce programsfor CAVERNSoft version 1 and version 2 (also known asG2). The source code generated can be used on SGI Irix,Linux, IBM Aix, and Microsoft Windows.

Figure 3. A resulting architecture.

Figure 4. Server and proxy objects.

Our code generator produces C++ classes for the serverobject and the proxy object. Each of these objects containsa threaded IRB (network object of CAVERNSoft) and de-fines a set of keys with their associated callback functions.For each input or output object, a key is defined to transmitthe value. The marshaling code for all the types is gener-ated to be able to use our system in a heterogeneous envi-ronment. A set of keys is also created for the control ofthe simulation (initialize, start, stop, pause, resume, shut-downmethods). It is therefore possible to manipulate proxyand server entities as C++ objects, without dealing withnetwork issues. Figure 3 presents the resulting architecturecorresponding to the description file of figure 2. It showsthe relation between the server object (see in figure 1) andthe proxy object. Figure 4 shows an example of the result-ing programs generated byCAVEStudy. We present a cleanand simple interface of the two components to the program-mer. All the communication aspects are hidden within theimplementation of C++ objects. The only public methodsare those concerning the setting of new values for the in-put parameters and those to get the values from the simu-lation. For the server program (cf. figure 4, on the right),we create an instance of the server class and enter an end-less loop waiting for remote method invocations. The proxy(cf. figure 4, on the left) connects to a given server (machinename plus port number) and sets the values of the input para-meters. It is then possible to control the execution of the re-mote simulation with some method invocations (start, stop,etc.). The data produced by the remote simulation are sentand stored automatically to the proxy object, and can be read

COMPUTATIONAL STEERING AND MEASURING IN VIRTUAL REALITY 83

TPC CAVEStudy TCP CAVEStudythroughputlatency latency throughput (Mbytes)

(ms) (ms) (Mbytes) Char Double

Ethernet 100 Mbits 0.20 1.64 7.9 5.1 2.7Internet 3.71 7.78 1.7 1.5 1.3

Figure 5. Performance measurement.

at any time by local method calls (without any communica-tion). This proxy object is embedded into our VR frame-work.

We present in figure 5 some preliminary performanceresults produced by the current (non-optimized) prototype.The round-trip latency measurement is performed using theTCP protocol with a one-byte buffer in a RPC manner (trig-gering a function which provides the data received). Thethroughput test is done with an 8 Kbytes buffer using ei-ther character data or floating-point values (requiring mar-shaling and network-independent representation). Figure 5shows the latency and the throughput between two nodes ofa Linux cluster using 100 Mbits Ethernet, and the same ex-periments between two computers within Amsterdam usingInternet. The callback mechanism and the data marshalingof CAVERNSoft introduce some performance loss, mainlyon the local network. Using Internet, theCAVEStudylatencyis approximatively twice the one using a low-level program-ming approach. The differences are smaller for the through-put: it decreases from 7.9 Mbytes/s using a low-level TCPsocket to 5.1 Mbytes/s withCAVEStudyusing Ethernet. Ap-plying a packing algorithm to deal with heterogeneous sys-tems (big-endian and little-endian representation) gives usa throughput of 2.7 Mbytes/s using Ethernet. Between twosites interconnected by Internet, theCAVEStudyoverhead re-mains minimal. Even though our main focus in this work isnot the performance, we consider these results encouragingfor a first implementation.

3.3. VR framework

Besides the code generator, we developed a virtual realityframework to steer and visualize the data produced by thesimulation. It is built on top of thePerformer library toexploit the high performance of SGI workstations and ontop of theCAVELib library to control VR devices (multi-screens and tracker management), as shown on the left partof figure 1. The architecture is described in figure 6, fo-cussing on the VR framework as referred in figure 1. It con-sists of a shared-world where the objects of the simulationare represented. These objects are updated through the gen-erated proxy object. An interaction module allows the userto send commands to the proxy or to directly manipulate theobjects of the simulation to steer it. This framework is func-tional but still under development, following the needs ofthe applications described in the next section. For instance,we developed a menu system to control the execution of thesimulation. Using sliders and buttons it is possible to modifyparameters values. The interaction modules allows the userto scale, rotate and translate objects. The graphic part in-

Figure 6. VR framework.

Figure 7.CAVEStudylaboratory.

cludes basic objects (lines, point, spheres) using color, trans-parency and textures. A board provides also textual informa-tions. More specific details are discussed in the applicationsection. Figure 7 shows the default virtual environment pro-vided byCAVEStudy. It consists of a virtual laboratory witha manipulation box where data can be inserted, and a boardto present some text information.

The current version of our VR framework was tested andused with two distinct CAVEs, an Immersadesk, SGI work-stations, and finally Linux PCs (usingCAVElib and Per-former ports on Linux). A new version is under develop-ment to support Microsoft Windows, using eitherDirectX orOpenGL.

4. Applications

To evaluate our approach of coupling a simulation and avirtual-reality environment, we implemented three differentapplications usingCAVEStudy: Interactive Soccer, DiodeLaser Simulation, andMolecular Dynamics. We use theseto illustrate the ease of incorporating an existing application,the usability of such a method, and the added value for theuser.

84 RENAMBOT ET AL.

4.1. Interactive RoboCup

RoboCup (Robot Soccer) is a standard problem from Artifi-cial Intelligence [12]. Its goal is to let teams of cooperatingautonomous agents play a soccer match, using either real ro-bots or simulated players. We constructed a VR environmentin which humans in CAVEs at different geographic locationscan play along with a running RoboCup simulation in a nat-ural way. A central role is played by the so calledSoccerServer, which keeps track of the state of the game and pro-vides the players with information on the game. The playersare individual processes that can request state informationfrom the server and autonomously compute their behavior.The server also enforces the rules of RoboCup and ignoresinvalid commands from the players.

Our RoboCup VR system uses the unmodified existingserver software. The players communicate with the serverby sending soccer commands. The commands are expressedin a simple language, consisting of accelerations, turns, andkicks. The server discretizes time into slots and only the lastcommand of a player within a time slot is executed. Also,the kick command requires the player to be close to the ball.We run such a system on a Myrinet cluster computer.

The CAVE program allows the user to be immersed inthe game and to interact with it. We implemented a proxywhich uses the same information and communication as theexisting 2D visualization. The data set described in ourCAVEStudyconfiguration file consists mainly of the playerpositions and the ball position. From successive states of thegame, the visualization system computes several quantitiessuch as direction, velocity and acceleration of the players.We built a virtual stadium (figure 8) and a parameterized soc-cer player whose movements are interpolated between threedifferent modes: standing still, walking, and running. Wedeveloped software to track the behavior of a human in theCAVE. One tracker is connected to the viewing glasses andmonitors positional changes of the human player inside theCAVE. The second tracker is connected to the wand (a 3Dmouse), which is used for global movements over the soc-cer field. The third tracker is attached to the foot of thehuman player and is used to recognize a kick. We converttracker changes into soccer commands and transmit these tothe server. Finally, we coupled two CAVEs located in Am-sterdam and Stockholm [27]. Each CAVE is connected tothe same proxy, so the two humans participate in the samegame.

The most difficult problem in realizing a virtual RoboCupsystem is caused by the latency of the simulation program.If the human player moves over the virtual soccer field, thesemoves happen almost instantaneously for the human. In con-trast, the soccer server will require some time to process thechange of position. Also, the wide-area (Internet) connec-tion causes a substantial delay. This problem is a typicalexample of how a delay introduced by a simulation pro-gram can harm a natural and real-time interaction [21]. Weare currently developing accurate and low-bandwidth algo-rithms for the navigation (walking across the soccer field)

Figure 8. Interactive RoboCup.

and the interaction (kicking detection) that generate com-mands to the server.

The first implementation of this system was done with-out CAVEStudy. We had to program the communicationbetween the different components, which is tedious and er-ror prone. UsingCAVEStudy, the communication is auto-matically generated. More generally, the case study withRoboCup shows the applicability ofCAVEStudyon the largeclass of agent/server systems.

4.2. Diode laser simulation

Simple systems with only a few degrees of freedom can ex-hibit chaotic dynamical behavior, even when the evolution isdeterministic. A precise knowledge of the initial conditionsallows us, in principle, to calculate exactly the future behav-ior of the system. One problem in understanding a chaoticsystem is the reconciliation of the apparent conflict betweenrandomness and determinism. An application of this classwe implemented is the visualization of a diode laser behav-ior, referred to as theSisyphus Attractor[17]. Numericalsimulations are performed for a semiconductor diode laser,subject to optical feedback. Due to the feedback, the result-ing dynamical system has infinite degrees of freedom. Theexploration and investigation of such a large data set callsfor the immersion of the user into a representation of theparameter space. A simulation run generates a trajectory insuch a space. In the 3D space provided byCAVEStudy, wedecided to focus on the most natural phase space from thephysical point of view (the output power, the inversion, andthe phase difference) as shown in figure 9.

A previous study [17] on the visualization of this simu-lation already gave a better insight into the dynamical be-havior of the laser, but suffered severely from lack of inter-action. WithCAVEStudy, we linked the simulation runningon an IBM SP2 to our CAVE. In a first step, the simulation

COMPUTATIONAL STEERING AND MEASURING IN VIRTUAL REALITY 85

Figure 9. Diode laser simulation.

computes some fixed points in the phase space for a givenset of parameters. The user can interactively set the valuesof selected parameters using sliders. The fixed points serveas starting point of the simulation. These points are visual-ized, and the scientist can directly select one of these pointsto start the simulation. The computed trajectory is sent in-crementally to the CAVE. The trajectory is visualized andcan be manipulated by the scientist. The simulation can bestopped and re-started using a new starting fixed-point or dif-ferent parameter values.

CAVEStudy’s benefits are many-fold in this case; it is eas-ier to use than a previous approach (batch-processing andoff-line visualization); the study of the initial-condition sen-sitivity of the laser is enhanced by the ability to modify theparameters of the simulation interactively; since our systemdoes not require modifications of simulation code, we candeal very easy with the changes of a code still revised fre-quently; the interactive way in which physicists could testhypotheses and investigate the behavior of the diode laserhelped them to gain a better insight in this complex system.

4.3. Interactive molecular dynamics

Our third application concerns the coupling of a moleculardynamics (MD) simulation to a virtual reality system. Mole-cular modeling tools are essential to design and study newmolecules. For example, when steering a molecular dynam-ics simulation, the user can express external forces to helpthe system to overcome energy barriers, or can help in thesearch for likely geometric configurations in docking prob-lems. Many studies focus on numerical simulations or visu-alization tools, and some of them describe the advantages ofconnecting molecular dynamics simulation to visualization.For instance, NAMD [19], which includes state-of-the-artserial and parallel algorithms, has been designed as a flex-ible program, incorporating many options such as control

Figure 10. Interactive molecular dynamics.

integration methods, force field parameters, and restart ca-pabilities. VMD [9] is a visualization environment for struc-tural biology which has been widely used for plain visual-ization, docking studies, structure refinement or trajectoryanalysis. An important feature is that it is possible to cou-ple a NAMD running simulation to VMD to study trajecto-ries of molecules. Moreover, some local forces can be ap-plied using either a 2D graphical interface or a 3D hapticfeedback tool [9]. In [13], the authors describe how theylink their molecular simulation to VMD. The user can ex-press external forces to help the system to overcome energybarriers between states. Another example is the moleculardocking simulation program described in [15]. A user in avirtual reality environment can interact with a genetic algo-rithm running on a parallel computer to help in the searchfor likely geometric configurations.

VR allows the scientist to gain a deeper understanding ofthe complex conformations in 3D. Moreover, modifying 3Dstructures or expressing forces is intrinsically a 3D process,for which the use of an immersive virtual environment is aperfect match. As a feasibility study (and for later exper-iments on interaction and measurement), we wrapped themolecular dynamics NAMD [19] simulation and visualizedit in a VR environment.

Our current implementation allows a remote simulationrunning on the DAS parallel cluster computer [2] to be vi-sualized in the CAVE. The input parameters we selected arethe name of the molecule on which the simulation will beapplied, the number of time steps of the simulation, and thetemperature. It corresponds to the minimal set of parame-ters among the large possibilities of NAMD. We did notimplement the interactive parameter selection yet, but pa-rameters can be modified at starting time. As output of thesimulation, we use the PDB description files produced as in-termediate result during the execution. These files, whichcontain the position and velocity of all the atoms, are readby the proxy process and sent continuously to the visualiza-tion, showing the dynamics of the molecule. Several classicmolecule representations are available (wireframe, sphere,backbone). An example is shown in figure 10.

86 RENAMBOT ET AL.

The coupling of MD simulation to visualization has al-ready been done before, but always by modifying sourcecode. Usually, they are made by the original developersmodifying their own software and using simple TCP/IP con-nections, and in a very application-specific manner. UsingCAVEStudy, we were able to very quickly couple NAMDto our virtual environment. WithCAVEStudy, we can easilyswitch between several simulation packages. Furthermore,to steer such a simulation adequately, 3D forces should beexpressed, which can efficiently be done in a 3D VR envi-ronment implemented byCAVEStudy.

5. Measuring

Existing methods for measuring in virtual reality are basedon ideas like probing the visual domain, deriving quantitiesfrom the simulation prior to visualization or altering the sim-ulation to generate required data directly. However, thesemethods are too restrictive, rely on advanced programmingskills of the scientist (modification of the source code), orare batch-oriented by nature (extensive search).

We are working on a system based onCAVEStudyformeasuring and steering using virtual reality environments[6]. It will allow non-expert users to use virtual reality en-vironments as a measuring tool next to a visualization tool.A few examples are: measuring the pressure directly ontothe visualization during a gas simulation, measuring fieldsor electric forces within a molecular dynamics simulation,or measuring distances in pre-surgery medical applications.This system will profit from all the benefits ofCAVEStudy(no modification of the simulation program and interactionwith the simulation’s original input and output). The meas-uring process will be done interactively by the final user,within the VR world. The measuring methods will be pro-vided by our VR environment, operating on the data pro-duced by the steered simulation. Such a system raises nu-merous research issues, among them virtual measurementparadigms, comparison with other steering methods, calibra-tion and validity of the measurements.

We are currently studying two relatively simple simula-tions, a charge-recombination simulation, and an ideal-gassimulation. For both case-studies we try to show that it ispossible to measure physical quantities and derived quanti-ties from data in the visual domain.

The coupling of a steering system to some measuringtechniques will produce a high-level and easy-to-use steer-ing environment, putting the scientist within an “instrumen-tation loop”.

6. Conclusion and future work

In this paper, we described theCAVEStudysystem that al-lows the scientist to interactively steer a simulation, withoutrequiring any modification to the original program. It en-ables an interactive and immersive analysis of a simulationrunning on a remote computer.

A set of input, output, and graphical objects are speci-fied by the scientist in a description file. These objects rep-resent the input parameters of the simulation and the dataproduced. Using such a description, our system generates aserver to control the simulation and to send data producedto the virtual environment. These data are used to updatethe graphical objects, which can be manipulated to steer thesimulation. Thus, it is possible to visualize and control theprogram directly in the domain space of the simulation. Werun simulations on various systems, such as an IBM SP2and a Myrinet cluster computer, and coupled them to sev-eral CAVE VR systems. Moreover, three real applications(soccer, laser simulation, molecular dynamics visualization)have been studied. The lessons learned from these appli-cations are that different types of applications can easily beimplemented usingCAVEStudy, that CAVEStudyis easy touse and to maintain compared to previous methods, and that3D interactive visualization and steering help to gain a betterinsight in a complex system.

In our future work, we will developCAVEStudytowardsseveral directions. First, we plan to incorporate a betterdata management allowing the user to deal with data fluxescoming from the simulation. With the applications alreadyimplemented, we observed several patterns: data generatedonce, small data produced continuously, large data sets sentseveral times during the simulation, several data fluxes pro-duced synchronously. It would be useful to the scientistto visualize these data sets independently of the simulationtime. We can imagine that the user will be able to “walk”through the simulation time in a VCR manner (forward,backward, rewind, etc.), while the remote program generatesthe data at its own speed. We will also design and implementthe measuring techniques sketched in the last section, toachieve a higher level of steering withCAVEStudy. This en-vironment will provide a generic set of interaction, selection,and measuring functions. We will apply these on some newapplications like medical simulations and robot exploration.

The wholeCAVEStudysystem will be available on ourweb site:http://www.cs.vu.nl/ ∼renambot/vr .

References

[1] G. Allen, W. Benger, T. Goodale, H.-C. Hege, G. Lanfermann,A. Merzky, T. Radke and E. Seidel, The Cactus Code: A problemsolving environment for the Grid, in:Proceedings of the Ninth IEEEInternational Symposium on High Performance Distributed Comput-ing, Pittsburgh, PA, August 2000 (IEEE Computer Society Press,2000).

[2] H. Bal, R. Bhoedjang, R. Hofman, C. Jacobs, K. Langendoen, T. Rühland F. Kaashoek, Performance evaluation of the Orca Shared ObjectSystem, ACM Transactions on Computer Systems 16(1) (February1998) 1–40.

[3] C. Carlsson and O. Hagsand, DIVE – A platform for multi-user virtualenvironments, Computers and Graphics 17(6) (November–December1993) 663–669.

[4] A. Foster and C. Kesselman,The Grid: Blueprint for a New ComputerInfrastructure(Morgan Kaufman, 1998).

COMPUTATIONAL STEERING AND MEASURING IN VIRTUAL REALITY 87

[5] I. Foster and C. Kesselman, Globus: A metacomputing infrastructuretoolkit, The International Journal of Supercomputer Applications andHigh Performance Computing 11(2) (1997) 115–128.

[6] D. Germans, H.J.W. Spoelder, L. Renambot and H.E. Bal, High-levelsteering: Measuring in virtual reality environments, Technical report,Vrije Universiteit, Faculty of Sciences (September 2000).

[7] C. Greenhalgh and S. Benford, MASSIVE: A collaborative virtualenvironment for teleconferencing, ACM Transactions on Computer–Human Interaction 2(3) (1995) 239–261.

[8] A.S. Grimshaw and W.A. Wulf, The Legion vision of a worldwidevirtual computer, Communications of the ACM 40(1) (January 1997)39–45.

[9] W.F. Humphrey, A. Dalke and K. Schulten, VMD – visual moleculardynamics, Journal of Molecular Graphics 14 (1996) 33–38.

[10] G.A. Geist II, J.A. Kohl and P.M. Papadopoulos, CUMULVS: Provid-ing fault tolerance, visualization and steering of parallel applications,The International Journal of Supercomputer Applications and HighPerformance Computing 3(11) (1997) 224–235.

[11] D.J. Jablonowski, J.D. Bruner, B. Bliss and R.B. Haber, VASE: Thevisualization and application steering environment, in:Proceeding ofSupercomputing’93(1993) pp. 560–569.

[12] H. Kitano, M. Veloso, P. Stone, M. Tambe, S. Coradeschi, E. Osawa,I. Noda, H. Matsubara and M. Asada, The RoboCup synthetic agentschallenge 97, in:15th International Joint Conference on ArtificialIntelligence, ed. M. Pollack (1997) pp. 24–29.

[13] J. Leech, J.F. Prins and J. Hermans, SMD: Visual steering of mole-cular dynamics for protein design, IEEE Computational Science andEngineering 3(4) (Winter 1996) 38–45.

[14] J. Leigh, A.E. Johnson, T.A. DeFanti and M. Brown, A review oftele-immersive applications in the CAVE research network, in:IEEEVirtual Reality’99(1999) pp. 180–187.

[15] D. Levine, M. Facello, P. Hallstrom, G. Reeder, B. Walenz andF. Stevens, Stalk: An interactive system for virtual molecular dock-ing, IEEE Computational Science 4(2) (April/June 1997) 55–65.

[16] M.R. Macedonia, M.J. Zyda, D.R. Pratt, P.T. Barham and S. Zeswitz,NPSNET: A network software architecture for large-scale virtual en-vironment, Presence 3(4) (1994) 265–287.

[17] C.R. Mirasso, M. Mulder, H.J.W. Spoelder and D. Lenstra, Visualiza-tion of the Sisyphus attractor, Computers in Physics 11(3) (May/June1997) 282–286.

[18] J.D. Mulder, J.J. van Wijk and R. van Liere, A survey of computa-tional steering environments, Future Generation Computer Systems13(6) (1998).

[19] M.T. Nelson, W.F. Humphrey, A. Gursoy, A. Dalke, L.V. Kalé, R.D.Skeel and K. Schulten, NAMD: A parallel object-oriented moleculardynamics program, The International Journal of Supercomputer Ap-plications and High Performance Computing 10(4) (1996) 251–268.

[20] U. Obeysekare, F.F. Grinstein and G. Patnaik, The visual interac-tive desktop laboratory, IEEE Computational Science and Engineer-ing 4(1) (January 1997) 63–71.

[21] K.S. Park and R.V. Kenyon, Effects of network characteristics on hu-man performance in a collaborative virtual environment, in:IEEEVirtual Reality’99(1999) pp. 104–111.

[22] S.G. Parker, M. Miller, C.D. Hansen and C.R. Johnson, An integratedproblem solving environment: The SCIrun computational steeringsystem, in:Hawaii International Conference of System Sciences(Jan-uary 1998) pp. 147–156.

[23] S. Rathmayer and M. Lenke, A tool for on-line visualization and in-teractive steering of parallel HPC applications, in:Proceedings of the11th IPPS ’97(1997) pp. 181–186.

[24] C. Shaw, M. Green, J. Liang and Y. Sun, Decoupled simulation invirtual reality with the MR toolkit, ACM Transactions on InformationSystems 11(3) (July 1993) 287–317.

[25] G. Singh, L. Serra, W. Png and H. Ng, BrickNet: A software toolkitfor network-based virtual worlds, Presence 3(1) (1994) 19–34.

[26] S. Singhal and M. Zyda,Networked Virtual Environments: Designand Implementation(Addison-Wesley, 1999).

[27] H.J.W. Spoelder, L. Renambot, D. Germans, H.E. Bal and F.C.A.Groen, Man multi-agent interaction in VR: a case study withRoboCup, in:Poster Session – IEEE Virtual Reality’00(2000).

[28] H.J.W. Spoelder, Virtual instrumentation and virtual environments,IEEE Instrumentation and Measurement Magazine 3(3) (1998) 14–19.

[29] H. Tramberend, Avocado: A distributed virtual reality framework, in:IEEE Virtual Reality’99(1999) pp. 14–21.

[30] J.J. van Wijk and R. van Liere,An Environment for ComputationalSteering(Computer Society Press, 1997).

[31] J. Vetter and K. Schwan, High performance computational steeringof physical simulations, in:Proceedings of the 11th IPPS ’97(1997)pp. 128–132.

[32] K. Watsen and M. Zyda, Bamboo – A portable system for dynami-cally extensible, Real-time, networked, vitual environments, in:IEEEVirtual Reality Annual International Symposium (VRAIS’98)(1998).

Luc Renambot is currently a Postdoctoral Re-search Associate of the Vrije Universiteit in Am-sterdam, The Netherlands. He received in 2000a Ph.D. in computer sciences from the Univer-site de Rennes 1, France, within the INRIA Re-search Institute. His research interests includehigh-performance and parallel computing, compu-tational steering, virtual reality environments, andcomputer graphics.E-mail: renambot@cs.vu.nl (main contact)

Henri E. Bal received a M.Sc. in mathematicsfrom the Delft University of Technology in 1982and a Ph.D. in computer science from the VrijeUniversiteit in Amsterdam in 1989. His researchinterests include parallel and distributed program-ming, cluster computing, and programming lan-guages and interactive visualization. At present,Dr. Bal is a full professor at the Vrije Universiteit.He is a member of the ACM and IEEE CS.E-mail: bal@cs.vu.nl

Desmond M. Germansis a Ph.D. student at thePhysics-Applied Computer Science group of theVrije Universiteit of Amsterdam. He earned MSin physics at the same university in 1998. His re-search interests include high-performance graphicsand interactive applications in virtual reality.E-mail: desmond@nat.vu.nl

Hans J. W. Spoelderstudied experimental physicsand received the Ph.D. degree in biophysics in1987, at the Vrije Universiteit in Amsterdam, TheNetherlands. Since then, he has been a FacultyMember of the Physics Applied Computer ScienceGroup, Division of Physics and Astronomy, VrijeUniversiteit. In 1991, he was visiting scientist asIBM T.J. Watson research Center, Hawthorne, NY.Dr. Spoelder is presently an associated Professor ofthe Physics Applied Computer Science Group. Hisresearch interest is in the field of interactive visual-ization, virtual realities, high performance comput-ing and computational steering.E-mail: hs@nat.vu.nl