Situational Visualization

David M. Krum, William Ribarsky, Christopher D. Shaw∗, Larry F. Hodges, and Nickolas FaustGVU Center, Georgia Institute of Technology,

801 Atlantic Drive, Atlanta, Georgia, 30332-0280, USA{dkrum,ribarsky,cdshaw,hodges}@cc.gatech.edu, nick.faust@gtri.gatech.edu

Phone: 404-894-6328, Fax: 404-894-2970

ABSTRACTIn this paper, we introduce a new style of visualization calledSituational Visualization, in which the user of a robust, mo-bile visualization system uses mobile computing resources toenhance the experience and understanding of the surround-ing world. Additionally, a Situational Visualization systemallows the user to add to the visualization and any under-lying simulation by inputting the user’s observations of thephenomena of interest, thus improving the quality of visu-alization for the user and for any other users that may beconnected to the same database. Situational Visualizationallows many users to collaborate on a common set of datawith real-time acquisition and insertion of data. In this pa-per, we present a Situational Visualization system we aredeveloping called Mobile VGIS, and present two sample ap-plications of Situational Visualization.

KeywordsMobile users and collaborators, dynamic databases, syn-chronized databases, real-time acquisition and insertion ofdata, location-specific services, location and time-specificuser input.

1. INTRODUCTIONMany tasks require a detailed knowledge of the local en-

vironment as well as an awareness of rapidly changing andinteracting events. This awareness is termed situationalawareness or situation awareness. It can be formally definedas:

“the perception of the elements in the environ-ment within a volume of time and space, thecomprehension of their meaning and the projec-tion of their status in the near future.” [5]

This type of spatio-temporal knowledge and awareness is im-portant in many tasks, including aircraft piloting, military

∗Contact author

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.VRST’01,November 15-17, 2001, Banff, Alberta, Canada.Copyright 2001 ACM 1-58113-427-4/01/0011 ...$5.00.

maneuvers, law enforcement, fire fighting, on-site construc-tion engineering, and even the daily automobile commute.

In this paper, we define a set of techniques to providecomputer-based support for situational awareness in aug-mented and virtual reality systems. We wish to bring large,diverse sets of data into such systems, and manage the lim-ited visualization resources of the system and the limitedperceptual and cognitive resources of the user. We describethis body of techniques as Situational Visualization. Situ-ational Visualizations are responsive to a user’s location inspace and time, as well as her actions and situation. Theyintegrate real-time and static data from a variety of sourcesand allow users to view time-varying events and objects andto project trends into the future.

The difference between such Situational Visualizations andother types of visualizations (e.g. information visualizationand scientific visualization) is that the user is situated in theenvironment and experiencing the situation firsthand and inreal time. These elements of Situational Visualization be-come especially significant for mobile users who can movefrom place-to-place and can encounter new and unexpectedsituations. The user must be able to receive timely informa-tion associated with the situations encountered.

From the user’s perspective, this implies that the systemmust be:

Unobtrusive: The user must be able to attend to the en-vironment without being distracted by the system in-terface.

Context-Aware: The system must be aware of the user’slocation and circumstances.

Time-Aware: The system must help the user anticipateevents, objects, and trends.

Priority-Aware: The system must help the user prioritizeand attend to the most important events and objects.

Users can also act as data collectors, having a hand in theconstruction of the database. It is now possible for users tocarry a variety of sensors (cameras, various environmentalsensors, etc.) as they move about. The user may activelydirect these sensors, or allow them to operate automatically,providing a rich stream of information to be absorbed intothe database. Furthermore, a user may develop higher levelinterpretations and understanding about the environment.The user can make precise observations and annotationsbased on what she sees. To help capture and share thatinformation, the system must support annotation (the capa-bility to annotate and edit the database) and collaboration

(the ability to communicate and work with remotely locatedusers).

These high level capabilities and attributes translate intothe following system design requirements:

Real-time Data Acquisition: To update the situationalpicture, the latest information must be placed in thedatabase. Real-time acquisition and insertion of datamust be possible from a variety of sources.

Dynamic Data: Because the database is continually beingupdated with spatial data, dynamic data structuresmust be capable of changing on-the-fly as pertinentinformation is received.

Synchronized Data Structures: Synchronized data struc-tures are required, since there may be multiple, net-worked databases and multiple collection points.

User Geo-Location: A key situational variable is user lo-cation, so that users needing situational informationcan see who contributed it, and where it comes from.Users must be geo-located, and must be able to in-sert their orientations and movement vectors into thedatabase. The database will be enriched with these an-notations and, in turn, will permit more accurate userlocations and orientations to be derived using sensorfusion techniques.

Global Geo-spatial Structure: There must be a global,geo-spatial structure capable of handling all types ofgeo-spatial data from all over the Earth. The databasemust be able to handle data from any location, andmust therefore be robust enough to address any loca-tion at any reasonable resolution.

Time and Location-specific Services: This presumes adatabase structure appropriate to the location-specificand temporal nature of the user’s situation and work.With this structure, one can have location-specific ser-vices, which also depend on the time at which relevantevents occur, on what the user is doing, and on whereshe is and where she is going.

Situational Visualization can share many properties ofVirtual Environments. The application may take the form ofan immersive and highly interactive 3D virtual environment.Situational Visualization is also an Augmented Reality ap-plication in that one is simultaneously navigating the realworld and a corresponding annotated 3D virtual replica ofthe world. There are issues of view registration in terms ofaligning a virtual view in the database with the real viewseen by the user. Although we do not do so in this paper,one could carry these issues all the way to the simultaneousalignment of virtual geo-located objects and real ones in thesame scene, as is usually done in Augmented Reality. Be-cause of the overlap, we treat the issues of immersion and in-teractivity in Situational Visualization from the standpointof Virtual Reality.

This paper will examine the design of data and synchro-nization structures, as well as collaboration and awarenessfeatures to support Situational Visualization. We also ex-amine some example Situational Visualization applications.

2. RELATED WORKSome aspects of Situational Visualization are seen in sys-

tems that present information based on the user’s location.A number of tour guide and map navigation applicationshave demonstrated location-specific information for smallgeographical regions. Georgia Tech’s Cyberguide [1] is anindoor and outdoor tour guide based on the Apple Newtonhandheld computer. The indoor version presents informa-tion about various demos using an array of infrared trans-mitters for pinpointing location within a large lab. Theoutdoor version uses GPS to present information about lo-cal establishments in Atlanta. A travel diary is created andused to suggest other locations that a user might be inter-ested in visiting.

The Touring Machine [7] is a wearable augmented re-ality system that presents multimedia information aboutthe Columbia University campus with access to departmentweb pages. It combines see-through augmented reality withtablet input and output. The ”map-in-the-hat” system de-veloped by Thomas, et al. [17] is a navigational aid system.It uses a see-through head mounted display to display way-points and a compass for walking direction. The main focusof ”map-in-the-hat” is to guide the users in an orienteeringtask from one point to another.

Nokia’s Context Compass [16] is a wearable computer ap-plication that presents a see-through 2D overview map ofan urban area with roads, buildings, and virtual objects.Virtual objects can represent buildings, other users, or mes-sages, photographs, or waypoints, for example. Virtual ob-jects are displayed on a linear compass and selected by facingthe object, which centers the object in the compass. Eachvirtual object has an associated URL with a web page, im-age, audio, or video.

Our focus is to extend the notion of providing navigationand informational aids with real time data feeds and largedata sets covering large (up to whole Earth) geographic re-gions. Various projects for spatialized hypertext such asWorldBoard [15] and the Real-World Wide Web [11] will uti-lize large information data sets over worldwide geographicregions, but our system provides an emphasis on situationalawareness and visualization, requiring a more powerful setof geo-located objects and services than hypertext and otherweb media. It also permits continuous visual navigation toany point on the earth.

The collaborative and information annotation aspects ofSituational Visualization are related to work at Columbia onindoor and outdoor interfaces for MARS [9]. This follow-onwork to the Touring Machine adds a collaborative interfaceto their mobile augmented reality system that allows station-ary remote experts to monitor and assist mobile users. Theusers can create virtual objects and annotations to pointout interesting objects and direct attention. Another ap-plication that demonstrates context tagged annotations isStickePad on a GPS equipped PalmPilot, which allows usersto create geo-located and time-stamped “stick-e notes” forecology fieldwork [13].

Another project that is closely related to our descriptionof Situational Visualization is the Battlefield AugmentedReality System (BARS) developed at the Naval ResearchLaboratory [10]. It also combines a detailed map of theuser’s surroundings overlaid with a context and situationaware information display. This AR system presents dataabout the soldier’s environment such as sniper positions or

navigational waypoints based on the importance of each ofthe soldier’s current tasks and needs. However, it currentlyis designed to address the needs of the dismounted soldier inurban landscapes. We are developing a more general frame-work of services to assist a wider variety of users.

Our system is built on our global geo-spatial data or-ganization and visualization system, VGIS (Virtual Geo-graphic Information System) [3, 4, 6]. VGIS treats theterrain as a single, connected surface for rendering usinga continuous, view-dependent Level-Of-Detail (LOD) repre-sentation. Although there are similar approaches in the lit-erature, VGIS is the first system to support real-time view-dependent, pixel-accurate multi-resolution rendering of highresolution global datasets.

The VGIS system has the following capabilities:

• It accepts and integrates diverse types of geo-spatialdata into a global framework.

• It is scalable to very large data stores and to dis-tributed environments.

• It provides an interactive visualization framework sothat the data store can be navigated as one coherentwhole.

• It supports discovery of new information via naviga-tion in context with unfolding detail.

Our approach starts with a hierarchical structure for op-timal interactivity for data exploration. We use a ”forestof quadtrees” with nested coordinate systems for handlingglobal-scale terrain data and permitting timely paging ofcollections of objects in support of real-time navigation.

We have built a quadtree and shared cache data structurethat together constitute the basic components of our globaldata model. The global structure is divided into 32 zones,each 45 degrees × 45 degrees on the Earth’s surface. Eachzone has its own quadtree and all quadtrees are linked sothat objects or terrain crossing quadrant boundaries can berendered efficiently. On top of this structure we have added256 x 256 coordinate systems within each quadtree – onecoordinate system per node, eight levels below each rootnode – resulting in a 1 mm worst case precision.

3. DYNAMIC AND SYNCHRONIZED DATASTRUCTURES

An attribute of Situational Visualization is that data canbe received all the time, either in discrete or streaming fash-ion. Our interactivity requirement means that these datamust be readily available when received. One data struc-ture that can effectively accommodate these attributes isan appropriately tailored universal hierarchy. In the caseof geo-spatial data, the choice is the VGIS hierarchy de-scribed above, which we have shown can be tailored to han-dle a wide variety of data including terrain (from low tohigh resolution), phototextures and maps, GIS information,3D buildings, and even time-dependent, volume data (e.g.,weather) [3, 4, 6, 14]. The forest of quadtrees is used at thetop level for each of these types of data and although eachhas its specific structure at higher levels of detail, the specificstructures are quadnode aligned. The quadnode alignmentpermits the handling of dynamic data in the same generalfashion, regardless of the type. As the data are received

they are organized to fit the quadtree hierarchy. This en-tails building links to data chunks at appropriate levels ofthe hierarchy and updating quadnodes with the appropri-ate averages, ranges, error values (for LOD management),and links. For streaming data this process may exceed theallotted time budget, in which case the dynamic data areorganized approximately in the hierarchy. The system canthen complete the organization during idle times.

For Situational Visualization, data must also be in ”syn-chronized data structures” in the sense that any user maybe collecting data and putting it in her own database, whichmust then be synchronized with databases of her peers.With geo-spatial data, one of the peers may be quite a bitlarger than the others (e.g., a data server), so the networkingstructure and overall data organization will have aspects ofboth peer-to-peer and client-server structures. Our general,global structure allows us to think in terms of a completeglobal virtual hierarchy. This is the hierarchy for the wholeEarth down to very high resolution (currently 1mm). Noone, not even the servers, has a complete set of data; ev-eryone has some piece of it. But if anyone receives a newpiece of data for any place on Earth, he knows where toput it in the hierarchy and it is a unique location. The treestructure is unique but properties such as links to individ-ual blocks of terrain are not. Thus one peer may have aset of elevation posts at a quadnode that differs from thoseat the same node of another peer. Furthermore, these nodecontents may change over time.

Figure 1: Distributed databases. The two mobilesystems on the right have overlapping areas of in-terest. This figure is reproduced in color on page201.

The distributed set of databases needs a synchronizationmechanism. (see Figure 1.) If a mobile system, for example,moves into a new area and finds it doesn’t have the datait needs, it looks for these data amongst its peers (see Fig-ure 1). On the other hand, it can also be collecting data thatit wants to transmit to its neighbors. To support efficientsynchronization we define 3 types of peers, each with differ-ent attributes: neighbors, servers, and collaborators. Neigh-bors are the set of all peers who may communicate; serversare neighbors with large repositories of data at defined lo-cations, and collaborators are neighbors who are in activecommunication. Note that these groups are not mutuallyexclusive and could also change over time. The servers arethe first systems queried for data in their areas of interest,

although the rest of the neighbors could also be queried forhigh priority requests. Collaborators directly share annota-tions or acquired data and also such things as interactionsand viewpoints. Servers also get any updates for their ar-eas of interest. As long as a peer gets a fresh download ofdata from the servers when it enters the collection of neigh-bors (but only for those regions in its area of interest thatare marked newer on the appropriate server), it will be syn-chronized. Thus the synchronization is restricted to areasof interest for all peers. The meaning of areas of interest formobile users will be discussed further below.

For efficiency the querying peer can construct an exact ad-dress for the data block it needs. This is one of the attributesof the virtual global hierarchy. For example, a terrain foot-print can be addressed by first obtaining the zone it is in (i.e.,the quadtree from the forest of quadtrees), then finding theleaf cell at the quadlevel appropriate to the footprint size.These are both bin sort operations and quite fast. A uniqueaddress can then be constructed. This querying procedureneeds to take account of the distributed environment. Thusthe footprint covers not only the scene in direct view butalso areas likely to be covered in further navigation. Thisnavigational footprint (and the resolution needed) will bedifferent depending on, for example, whether a mobile useris walking, in a vehicle, or in an airplane. The system con-structs two navigational footprints around the user’s currentlocation (see Figure 2). The outer one defines the region tobe retrieved from the server, if necessary. However, this onlyhappens if the user moves outside the inner footprint in herexplorations. This arrangement insures that thrashing willbe minimized as the user moves back and forth; once the usermoves outside the inner footprint, a new set of footprints isset up centered around the new current position (Figure 2).This arrangement is similar to the one used in [12], exceptthat now it is done in the context of a complete global hier-archy. The footprint address and resolution needed is thentransmitted to, say, the relevant server. The server findsthe appropriate parent node and then does a breadth firsttraversal across child nodes at the appropriate resolution. Alist of data blocks is constructed for transmission (for exam-ple, terrain elevation blocks, or 3D blocks of buildings). Thelist is ordered so that blocks in the immediate vicinity of theuser are sent first, minimizing the delay in receiving data ofimmediate interest. The global hierarchy permits these tobe found quickly. The individual blocks are large enoughthat they contribute significantly to a given view yet smallenough that they are transmitted quickly. Several blockscan be in a view, depending on the viewpoint location. Wehave found that this balance in size makes out-of-core dataretrieval efficient for a variety of data [4, 3].

3.1 Streaming DataSome of the data may stream in continuously. These may

be time-stamped 3D weather data acquired from sensorssuch as Doppler radars, computer simulations, streamingvideo, or other data. A special server collects and organizesthese data, but in addition, mobile users may also collectand transmit streaming data (e.g., video). The overall orga-nizing structure for the streaming data is the same virtualglobal hierarchy discussed above. However, the data are alsoorganized for fast queries of the form, ”Tell me about thisevent at this time and location.” To handle these queries weare in the process of developing an event-based hierarchy for

Figure 2: Inner and outer footprints used to man-age retrieval of terrain and imagery data from aserver. Spatial locality of reference helps minimizethe amount of data that needs to be transmitted.

the sequence of steps in the streaming data (e.g., time stepsor image frames) aligned with the quadtree structure. Oneissue we are studying is at what quad level (i.e., what levelof spatial granularity) to insert the event structure. Withthis structure we will be able to ask questions such as, ”Howmany tornadoes occurred in North Georgia during the Sum-mer from 1995 to 1999, where, and with what severity,” or,”At what times and in what locales did this atmospheric pol-lutant exceed this level during Spring and Summer, 1998.”Queries such as this are relevant to the weather applicationinvolving streaming 3D Doppler radar data that we describebelow. The above dynamic structure is discussed more fullyin [8].

4. AWARENESS AND COLLABORATIVESTRUCTURES

Our implementation of Situational Visualization extendsthe spatial model of interaction proposed by Benford andFahlen [2]. In the spatial model of interaction, there arethree regions of note: the focus, aura, and nimbus. Thefocus is the area of interest, or region in which the useris aware of activity and objects. This could be the areadefined by the limit of visibility. The aura is the regionwhere interaction is possible. When the auras of two usersintersect, the users can interact. For example, when twopeople are face to face and within a few feet of each other,they can easily converse. The nimbus is the region in whicha second user can be aware of a first user. For example,on a crowded city sidewalk, a tall person has a much largernimbus than a shorter person. They are easily noticed byothers from longer distance.

There are four ways to modify the regions of focus, aura,and nimbus. The first is implicitly by making changes inlocation or orientation. The second is by explicit modifica-tion of key region parameters, such as radius. The third isby adapter objects, such as a telephone, telescope, or otherreal or imaginary object. The fourth is by boundaries inspace such as walls.

This interaction model is employed in networked collab-orative virtual environments such as MASSIVE and DIVE.However, there are several key differences between those vir-tual environments and a networked collaborative mobile en-vironment, such as a Situational Visualization. The spatialmodel of interaction was developed for virtual environmentwhere users have one virtual location and can easily movein virtual space. Some participants in a Situational Visual-ization are stationary clients. These clients could present anessentially unmodified spatial interaction model. However,many clients will be mobile. In these mobile clients, userswill have both a real and a virtual location that may or maynot coincide. The real or physical location corresponds tolocation reported by GPS. Virtual location corresponds tothe user’s cursor or viewpoint in the visualization. Since auser will often rapidly navigate through the visualization,virtual location will change faster than physical location. Auser will need to be aware of events and objects surroundingtheir physical location, while they closely examine an areathat may be a hundred miles away. Moreover, the user maywish to monitor several regions without explicitly examiningthose regions. A user may also want to be notified of futureevents that may have significance.

Modifications to the concept of focus would be to createfocus (or awareness) regions for both the user’s real andvirtual locations. The user would also be able to defineadditional regions of focus, in a sense, dropping pieces ofawareness across the terrain or attaching them to particularusers or moving objects. Users who make modifications tothe database should transmit updates to other users whohave an overlapping focus region. Since the spatial modelof interaction defines different region extents for differentmedia (sight, sound, communication, etc.), it follows thatusers should be able to register focus for particular classes ofobjects as well, (weather, traffic, virtual objects, other users,etc.) To deal with warnings about future events, a predictionprocess would determine if objects, such as weather eventsmight intersect a focus region in the future, even if the focusregion is in motion with the user.

In our implementation, auras determine interaction suchas personal communication and database communication.Since physical location typically changes less rapidly thanvirtual location, the concept of aura may be split in two.Transmitting auras will be attached to virtual body and re-ceiving auras will be attached to physical body. When a userwishes to initiate contact, the user will navigate the virtualbody in proximity of another user’s physical body and es-tablish a two-way communication channel. Since users maywish to stay in contact even if users move apart, the channelmust be explicitly closed. In our implementation, personalcommunication is currently text messaging, with voice com-munication to be implemented in the near future. Auraswill also apply to database communication. Some users maycarry updated static terrain, building, or other virtual ob-ject information. For example, one user may have an anno-tated database with locations of wheelchair ramps, handi-capped parking spaces, automatic doors, and stairways. Asecond user can initiate a database synchronization in muchthe same way as a communication channel. This allows thesecond user to fetch and add all the new information fromthe first user’s database. The second user can also havea previously enabled database connection that fetches thisinformation from the first user as needed. Some special pur-

pose stationary servers such as a weather information serverwould project a location quite different than it’s physicallocation. Its database communication aura coincides withthe weather footprint.

Mobile Situational Visualizations will have nimbus regionsfor virtual location as well as physical location. Nimbus isthe visibility profile of a user. In a standard mode of opera-tion, users would only be aware of the nimbus surroundingthe physical location of other users. Users would only seethe physical locations of other users. If a group of userswere collaborating closely, they may wish to see an avatarrepresenting a user’s virtual location.

5. SITUATIONAL VISUALIZATION APPLI-CATIONS

There are many potential Situational Visualization ap-plications. One class has to do with awareness, being no-tified of time-evolving events based on one’s location, di-rection of movement, or actions. The nano-forecaster de-scribed below is one such application. Another would be a“traffic-wizard” that gives one advice about traffic on po-tential routes as one approaches. Another class would in-volve geo-referenced services, such as telling a mobile userwhere and in what direction restaurants, bus stops, theaters,or other services are based on her current location. Otherapplications could involve annotations and updates of thesynchronized database as determined by real world obser-vations. These might include finding and noting handicapaccess points, noting positions of trees or shrubs and thenplacing appropriately scaled models, noting incompletenessor inaccuracies in building models and then capturing im-ages to be linked to the models, capturing geo-located andoriented video, and many other applications. There are alsocollaborative applications, such as two mobile users investi-gating a large building site and coordinating their informa-tion collection so that it is complete. Other collaborationsmight involve mobile users and those at some central lo-cation or command post. The user at the central locationmight act as an overall coordinator, directing the mobileusers and sending or receiving relevant information.

Our prototype Situational Visualization system consistsof a set of central computers and a mobile system. The cen-tral computers include both desktop machines and a virtualworkbench. The latter enhances collaborations since multi-ple people can be at the workbench and more than one caninteract. All systems are connected by an 802.11b WaveLANwireless network. To reach mobile users, WaveLAN anten-nas have been mounted on the exteriors of two buildings(Figure 3) providing a communication path of about a half-mile through a central part of the Georgia Tech campus. Anominal 11 Mb Ethernet runs on the wireless network. Theoutdoor portion of the network is subject to interruptionsand points where communication may be blocked. This,however, permits us to try out predictive tracking, use ofcached events, or other ideas for handling interrupted ser-vice. The mobile computer is an IBM 850 MHz Thinkpadwith 512 MB RAM and an ATI Mobility 128 AGP graphicschipset with 16 MB video RAM. This configuration runs theVGIS global geo-spatial visualization system at interactiverates.

To evaluate the Situational Visualization concept and be-gin building the needed infrastructure, we have identified

Figure 3: The Georgia Tech wireless testbed for Sit-uational Awareness. To reach mobile users, Wave-LAN antennas have been mounted on the exteriorsof two buildings providing a communication path ofabout a half-mile through a central part of the Geor-gia Tech campus. This figure is reproduced in coloron page 201.

two prototype applications. The first is an awareness appli-cation, the Nano-Forecaster. The second is an annotationapplication, the Wheelchair Assistant.

5.1 The Nano-ForecasterUp to now the best weather reports that a person can

receive is a set of ”micro-forecasts” developed for variousareas of a geographic region. These are not specifically cor-related to a viewer location and do not provide accurateinformation such as ”When is a storm going to hit me, fromwhat direction, and with what intensity?” However, suchinformation can be made available to a user if integratedin the above Situational Visualization context with appro-priate geo-spatial model and weather data. We have anaccurate model for North Georgia with high resolution in-sets for parts of Atlanta including the Georgia Tech campusand the weather data (in this case, real-time streaming 3DDoppler data from multiple radars covering the northernhalf of Georgia). Similar models have been or could be de-veloped for other locales. We have previously shown how3D, time-dependent data can be inserted on-the-fly into thegeo-spatial hierarchy [8]. The Doppler radars are operatedby the National Weather Service and are distributed acrossthe country, especially around urban or tornado-prone ar-eas. Thus the components are available that could makepossible wide use of the nano-forecaster.

The mobile visualization system is shown in Figure 4,where the user has navigated to a ground level view as-sociated with her location. We have found that a GPS-controlled icon, as shown in Figure 5, gives reasonably ac-curate and easy to follow position updates in an overview asa user walks through the same area. Further, the system canbe made to automatically navigate to a view encompassingthe user’s current position if its viewpoint is initially some-where else. The moving position information is enough todynamically update nano-forecaster queries. Since the time-dependent, 3D Doppler data are in the global geo-spatial

Figure 4: Prototype laptop Situational Visualiza-tion system. Mobile VGIS is running on the lap-top, showing a location on Georgia Tech campusplus buildings in midtown Atlanta. This figure isreproduced in color on page 201.

hierarchy, the parts in the user’s current area of focus canbe quickly located on the streaming data server. Associatedwith these 3D data is a set of storm analyses that are gen-erated at the same time as the data are collected. Thesetrack windfield intensities, reflectivity, and spectral width.These quantities can be transformed into severe storm signa-tures (including tornadoes), and precipitation intensity andtype (e.g. , rain or hail). For signature above certain val-ues, features called mesocyclone cells are generated, whichhave a location, width, height, and average values for thesignatures. These cells move over time and can be tracked.Certain cells that have the signatures of tornadoes, for ex-ample, can be flagged. The tracking permits a simple, butaccurate phenomenological model to be applied for predict-ing cell motion. A severe storm passing over Atlanta, withboth 3D radar data and mesocyclones, is shown in Figure 6.

In the current nano-forecaster prototype, the forecastingagent looks only at the subset of mesocyclones, with theirpredicted paths, in the current area of focus. It determineswhether the path of any mesocyclone will pass over the user’sprojected position (taking into account trajectory informa-tion from previous positions). The area of focus includes atime frame of interest; if the intersection is within that timeframe and the mesocyclone signatures are above a selectedthreshold, the user is notified. Notification can be as sim-ple as an arrow, indicating the direction of the approachingweather, labeled with type and severity of storm and time ofarrival, as shown in Figure 7. Alternatively, the user couldrequest a detailed view of the 3D Doppler data in her area offocus, as in Figure 6. The user’s area of focus and trajectoryand the storm’s development are all continuously updated.Since the key notification involves very little data, it couldeven be displayed on a PDA with minimal display capabil-

Figure 5: Screenshot of Mobile VGIS tracked byGPS. The arrow at the center indicates the user’sreal position. This figure is reproduced in color onpage 201.

ities, perhaps just showing a fixed image of the user’s areawith her current position and the storm’s labeled vector su-perimposed.

5.2 Wheelchair AssistantThe Wheelchair Assistant is an application that allows

wheelchair users to navigate appropriately through an out-door urban landscape. Users can annotate the building andterrain database with icons denoting locations of wheelchairramps, handicapped parking spaces, automatic doors, andstairways. Users will avoid having to double back because ofan outdoor staircase or use a heavy door when an automaticdoor is readily nearby.

The user can add wheelchair annotations by selecting anannotation type and clicking in the VGIS window for thelocation. The user can also move to the proper locationand then select an annotation type. After a short delay, theannotation will be made. One of the advantages of display-ing a highly detailed set of satellite phototextures and 3Dbuilding models, as in VGIS, is that a user can accuratelyplace an annotation even though GPS location informationis often inaccurate. The user can input the position of awheelchair ramp in relation to the building models or itemsdepicted in the satellite photo. Furthermore, the user canadjust the GPS position to more accurately reflect the user’sactual position.

Through database synchronization, users can share theseannotations to create a complete picture of wheelchair accessfor an area. This allows other wheelchair users to benefit.It also allows an organization like a business, school, or cityto visualize and assess how wheelchair users are constrainedor assisted.

A similar application mapping locations of emergency tele-phones, police stations, and manned security desks mightalso be a part of crime prevention and assessment projects.

6. CONCLUSIONSIn this paper, we have introduced a new style of visual-

ization called Situational Visualization, based on our mo-

Figure 6: Overview of Doppler Radar readings overNorth Georgia. The column of cylinders near At-lanta to the upper left is a series of mesocyclones,or large circular wind patterns. This figure is repro-duced in color on page 201.

bile Virtual Geographic Information System, VGIS. Ourapproach to issues surrounding how to deploy augmented in-formation spaces to mobile users is to start from the basis ofa rich source of information about the real world, such as ter-rain elevations and aerial photography supplied by variousgovernment agencies. This allows us to build an augmentedinformation source that already has a great deal of groundtruth. This Situational Visualization system allows the userto enhance the visualization and any underlying simulationby inputting the user’s observations of the phenomena ofinterest, thus improving the quality of visualization for theuser and for any other users that may be connected to thesame database. The example applications show how such asystem can afford a rich information space that can be usedto enhance the user’s situational awareness.

Figure 7: Closer orbital view of Georgia Tech andmidtown and downtown Atlanta with a single meso-cyclone plus its predicted path over the next 5 min-utes. This figure is reproduced in color on page 201.

7. ACKNOWLEDGMENTSThis work is supported by a grant from the NSF Human-

Computer Interaction Program and the DoD MURI pro-gram administered through the Army Research Office. Thiswork is also supported by a grant from the NSF Large Sci-entific and Software Data Visualization Program.

8. REFERENCES[1] G. D. Abowd, C. G. Atkeson, J. Hong, S. Long,

R. Kooper, and M. Pinkerton. Cyberguide: A MobileContext-Aware Tourguide. ACM Wireless Networks,3(5):421–433, 1997.

[2] S.D. Benford and L.E. Fahlen. A Spatial Model ofInteraction in Large Virual Environments. InProceedings of Third European Conference on CSCW(ECSCW’93), pages 109–124, September 1993.

[3] D. Davis, T.Y. Jiang, W. Ribarsky, and N. Faust.Intent, Perception, and Out-of-Core VisualizationApplied to Terrain. IEEE Visualization ’98, pages455–458, October 1998.

[4] D. Davis, W. Ribarsky, T.Y. Jiang, N. Faust, andSean Ho. Real-Time Visualization of Scalably LargeCollections of Heterogeneous Objects. IEEEVisualization ’99, pages 437–440, October 1999.

[5] M.R. Endsley. Situation Awareness Global AssessmentTechnique (SAGAT). In Proceedings of the IEEE 1998National Aerospace and Electronics Conference,volume 3, pages 789–795, May 1998.

[6] N. Faust, W. Ribarsky, T.Y. Jiang, andT. Wasilewski. Real-Time Global Data Model for theDigital Earth. In Proceedings of the InternationalConference on Discrete Global Grids, March 2000.

[7] S. Feiner, B. MacIntyre, T Hollerer, and A. Webster.A Touring Machine: Prototyping 3D Mobile andAugmented Reality Systems for Exploring the UrbanEnvironment. In Proceedings of the First InternationalSymposium on Wearable Computers (ISWC 1997),pages 74–81, October 1997.

[8] B. Hannigan, R. Mitchell Parry, Nickolas L. Faust,T. Y. Jiang, and William Ribarsky. HierarchicalStorage and Rendering of Real-Time 3D Data. In NickFaust, editor, Visualization of Temporal and SpatialData for Civilian and Defense Applications III,volume 4368A, 2001.

[9] Tobias Hollerer, Steven Feiner, Tachio Terauchi, GusRashid, and Drexel Hallaway. Exploring MARS:Developing Indoor and Outdoor User Interfaces to aMobile Augmented Reality. Computers and Graphics,23(6):779–785, 1999.

[10] Simon Julier, Marco Lanzagorta, Yohan Baillot,Lawrence Rosenblum, Steven Feiner, Tobias Hollerer,and Sabrina Sestito. Information Filtering for MobileAugmented Reality. In Proceedings of IEEEInternational Symposium on Augmented Reality(ISAR 2000), pages 3–11, October 2000.

[11] Rob Kooper and Blair MacIntyre. The Real-WorldWide Web Browser: An Interface for a ContinuouslyAvailable, General Purpose, Spatialized InformationSpace. In Proceedings of the 2nd InternationalSymposium on Mixed Reality, March 2001.

[12] Michael R. Macedonia, Michael J. Zyda, David R.Pratt, Donald P. Brutzman, and Paul T. Barham.Exploiting reality with multicast groups. IEEEComputer Graphics and Applications, 15(5):38–45,1995.

[13] J. Pascoe. Adding Generic Contextual Capabilities toWearable Computers. In Proceedings of the SecondInternational Symposium on Wearable Computers(ISWC 1998), pages 92–99, October 1998.

[14] Christopher D. Shaw, R. Mitchell Parry, WilliamRibarsky, Anthony A. Wasilewski, and Nickolas LFaust. Interactive Volume Rendering of Non-UniformData. In Nick Faust, editor, Visualization of Temporaland Spatial Data for Civilian and DefenseApplications III, volume 4368A, 2001.

[15] J. C. Spohrer. Information In Places. IBM SystemsJournal, 38(4):602–628, 1999.

[16] Riku Suomela and Juha Lehikoinen. ContextCompass. In Proceedings of the Fourth InternationalSymposium on Wearable Computers (ISWC 2000),pages 147–154, October 2000.

[17] B. Thomas, V. Demczuk, W. Piekarski, D. Hepworth,and B. Gunther. A Wearable Computer System withAugmented Reality to Support Terrestrial Navigation.In Proceedings of the Second International Symposiumon Wearable Computers (ISWC 1998), pages 168–171,October 1998.