Collaborative Visualization Jason Wood, Helen Wright, Ken Brodlie

School of Computer Studies University of Leeds, Leeds. LS2 9JT. UK

ABSTRACT Current visualization systems are designed around a single user model, making it awkward for large research teams to collectively analyse large data sets. This paper shows how the popular dataflow approach to visualization can be extended to allow multiple users to collaborate - each running their own visualization pipeline but with the opportunity to connect in data generated by a colleague. Thus collaborative visualizations are ‘programmed’ in exactly the same ‘plug-and-play’ style as is now customary for single-user mode. The paper describes a system architecture that can act as a basis for the collaborative extension of any dataflow visualization system, and the ideas are demonstrated through a particular implementation in terms of IRIS Explorer.

1. INTRODUCTION Scientific visualization is now a key component in computational science and engineering, allowing research teams to analyse large datasets that would otherwise be intractable. A number of powerful software systems have emerged over the last decade to support this work: notably, IRIS Explorer [3], IBM Data Explorer [6], AVS [I] and Khoros [7]. However each of these systems was designed under the premise that visualization is a single-user activity: collaborative working was not envisaged.

Today that premise seems increasingly false. Modem research is rarely a one-person task; it is carried out by large teams, often multi-disciplinary, each member of the team bringing different skills to the table. The team will quite likely be geographically spread. Yet all members of the team will need to come together over the analysis of the research results: surely, then, visualization must be a collaborative activity.

The current visualization systems mentioned above all follow a common model. They are sometimes known as modular visualization environments (MVEs). A library of modules is supplied and the user applies visual programming techniques to connect modules together in a dataflow pipeline; data is read in at the initial module of the pipeline, and successively transformed by each module in the pipeline, the visualization being finally rendered as an image, or animation, at the final module. The systems were typically designed in the late 1980s when distributed processing was important: thus different modules in the pipeline can run on different processors, allowing a computationally intensive process to run on a compute server,

O-81 86-8262-0197 $10.00 Copyright 1997 IEEE.

with lighter weight modules running on the user workstation. However there is a single user interface from which everything is controlled: thus the machine processing is distributed, but not the human processing. The model for collaboration is for all participants to cluster around a single workstation at a single location.

The world however is changing. The dramatic growth in Web computing raises the expectations of scientists and engineers to be able to collaborate effectively over networks and the visualization community needs to provide tools that allow scientists to work as in Figure I, each person able to take part in the collective analysis of data from their own offtce.

Progress has been made. One generic approach is to extend an existing single-user application by embedding it in a shared interface shell: the output from the application is broadcast to all collaborators, but input is limited to a single person holding a token. The token can be exchanged between participants. An example of this in the UNIX environment is “Shared x” [IO], while in the PC area, Intel ProShare [9] provides a similar facility. This approach has the advantages of generality and simplicity; but it is inflexible in the context of scientific visualization. It imposes the constraint that there just be one pipeline of modules - each collaborator must follow the same visualization idiom. It is also very demanding of bandwidth if a large number of images are being created.

This generic idea has been made special to visualization in the work of Pagendarm and Walter [8], and Wierse et al [l2]. Purpose-built collaborative visualization systems have been created, with participants having alternate control of the pipeline. However the same lack of flexibility and performance concerns persist. Moreover, scientists require to learn a new system, rather than make use of their existing experience with a single-user system.

An important variation on the shared interface approach is suggested by Grave [4] as part of the EC PAGEIN project. Each collaborator runs their own instance of a pipeline, and so there is not the performance overhead of transferring images across the network. Collaboration is achieved in the following way: parameters which control the operation of modules in the pipeline (for example, selecting the value of the isosurface to be extracted

u Nclwork

Figure 1 Collaborative Visualization

253

Proceedings of the 8th IEEE Visualization '97 Conference 1070-2385/97 $10.00 © 1997 IEEE

from volume data) are grouped in a specially written, shared interface which is layered on top of the MVE. Thus the collaborators drive their visualizations in a common way. The model extends to allow some parameters to be shared, and hence made public, while others can remain private to each participant. However even this approach lacks flexibility: the division of parameters between private and public needs to be made a priori; there is no opportunity to exchange data between pipelines (for example, to allow just one participant to run a simulation module and transfer results then to all collaborators); and the scientist has some additional learning to do in order to use the new system.

A further example is provided by FAST expeditions, devised by Watson et al [2] at NASA Ames, as an extension of the FAST CFD visualization system. Here an internal audit trail of a session is maintained, in such a way that it can be played back later by a collaborator. This extends to a live collaborative session, where the actions of one participant are used as a script to drive the visualization seen by another - their analogy is with “pilot” and “passenger”. Performance is good compared with the Shared X approach because each participant runs their own visualization processing, and the script data that needs to be exchanged is quite small relative to images, say. Again however there are limitations: there is an implicit assumption that each participant has the same dataset to start with, and that they wish to carry out the same visualization.

In this paper we describe a new approach to collaborative visualization in which we try to overcome some of the limitations of these other approaches. In section 2 we set out our design objectives and a reference model that supports the sort of flexibility we seek. Section 3 describes the resulting system architecture, and its realisation as an extension of IRIS Explorer. In section 4, we run through a sample collaborative session, to illustrate the way the system is used. Finally in section 5 we give some conclusions in respect of our design objectives.

2. DESIGN OBJECTIVES AND REFERENCE MODEL

Our aim is to create a collaborative visualization system with the following characteristics:

. Multiple independent participants - Each participant should run a separate instance of a visualization system, with freedom to collaborate as much as they please within a larger collaborative environment. The design should allow an arbitrary number of participants who should be able to join and leave as the session progresses.

. Data exchange - Each participant should be able to transmit, or receive, data from a collaborator. This data may be at any level of abstraction: raw data, geometry or images. In this way, collaborative sessions can be configured to match different computing scenarios - as mentioned above, it may be effective to have just one participant perform a computationally intensive simulation, and then broadcast the results to the others.

. Shared control - Any participant should be able to control any parameter of any module on the pipeline of any collaborator. This allows scenarios where different participants are allocated control of different parameters in a pipeline, according to their individual skills and expertise (for example, a CFD expert might control simulation module parameters, while a visualization expert might control the rendering parameters).

. Dynamic collaboration - The nature of the collaboration

I Figure 2: Visualization Pipeline Reference Model

should evolve, not be predetermined. Decisions to join a collaboration, and in what way to collaborate, should be allowed as the visualization analysis proceeds - indeed, the nature of collaboration can be driven by the visualization.

. Instructor driven - It should be possible to work in a mode where one participant is the instructor, able to launch and connect modules in a collaborator’s pipeline - indeed launch entire pipelines for a collaborator to use.

. Ease of learning - There should be no change of paradigm when moving from single-user to multi-user working.

O+-------+co Shared Control =-==w Shared Data

User A

User B

Figure 3: Extended Visualization Pipeline Reference Model

254

Proceedings of the 8th IEEE Visualization '97 Conference 1070-2385/97 $10.00 © 1997 IEEE

. Application mode - The current visualization systems allow end-user applications to be constructed from pipelines, by extracting key parameters into a single interface - from which the user then drives the application without recourse to the pipeline. The collaborative system should support that style of working also, in order to provide shared applications with simple interfaces.

The current MVEs have evolved from a common reference model, proposed independently by Upson et al [ 1 I] and by Haber and McNabb [5]. We start from the original model in which the visualization process is seen as a pipeline of processes, each pipeline being essentially the sequence: Data Input, Filter, Map, Render, Image (Figure 2).

In order to support multiple independent participants, we need to envisage such a pipeline associated with each collaborator. For data exchange, we need the ability to transmit and receive data at arbitrary points on the pipeline. For shared control, parameter settings need to be controlled either locally by the user, or externally by a collaborator. Dynamic collaboration requires that the pipelines and sharing between them is an interactive process, with instructor mode allowing the launching of modules and entire pipelines to be externally initiated. Finally ease of learning and application sharing drives us towards a solution that achieves collaboration as a simple extension of existing systems - something which is facilitated by our extending the original reference model to a collaborative scenario [ 131 (Figure 3).

Machine C

El- User C

i

Machine E

3. SYSTEM ARCHITECTURE AND IMPLEMENTATION

The previous section has shown how the visualization reference model extends quite readily to group working. We now develop from this model a system architecture that acts as the basis for the development of a collaborative visualization system - as an extension of IRIS Explorer.

The starting point is the architecture of a modular visualization environment. From the outset we assume that the environment has a set of supplied modules and pre-defined datatypes, but is open in the sense that new modules can be created and used within the system. It is also assumed that the visualization system can be driven by a scripting language as an alternative to using visual programming.

The elements of the collaborative architecture (Figure 4) which extend existing MVEs are as follows:

. Collaboratively aware modules (CAMS): These modules embody the essential links between pipelines. When a participant launches a CAM it causes a corresponding CAM to be launched automatically in the collaborators’ environments which they can connect in to their pipeline as they wish. There is a direct communication channel between each CAM and the central server. As a minimum, CAM modules must be provided which allow data and parameter sharing between pipelines, one for each datatype supported in the environment. in the environment.

. . Advisor module: This module allows the selection of a Advisor module: This module allows the selection of a module, or set of modules, from a pipeline on the local module, or set of modules, from a pipeline on the local machine; it generates the appropriate scripting commands to machine; it generates the appropriate scripting commands to

Central Server Central Server Participant List

Control Data Route Control Data Route Group 1 Data Route Group 1 Data Route Group 2 Data Route

Group n Data Route Group n Data Route

Machine D Machine D

t4 User D

I Visual Editor

*CAM - Collaboratively Aware Module

6 CAM’

I Visual Editor

L

Machine A Figure 4: Collaborative Visualization Architecture

255

Machine B

Proceedings of the 8th IEEE Visualization '97 Conference 1070-2385/97 $10.00 © 1997 IEEE

Figure 5: Collaborative Session Using IRIS Explorer

duplicate this pipeline fragment and sends these commands via the central server to the other participants.

. Local server: This requests entry to a collaborative session and manages the involvement of a participant in it, via a link to the central server. This local server launches appropriate modules, wires up connections and sets module parameters at the request of the central server.

. Central server: This element manages the collaboration and has three main features:

. Participant list: This keeps a record of multiple participants in the session, dynamically adding and deleting them as they join and leave.

. Control data route: This route passes control messages about new module launches to local servers. It also acts as the communication channel for any connected advisor modules.

. Group data routes: These routes are initiated by the connection of new CAMS and distribute data from one group member to all others. The group ID is common to all group members and acts as a unique way of identifying the end points of the data route. These unique identifiers also facilitate the construction of end-user collaborative applications by guaranteeing automatic connection of their multiple constituent CAMS.

This generic design has been implemented for the special case of IRIS Explorer, running in a UNIX environment. Figure 5 shows a screen shot from a collaborative session. The local server has been implemented as a module along with the advisor module and the collaboratively aware modules. These behave like any other IRIS Explorer module: their names are listed in the library window (left of screen) and they are launched into the Map Editor (the IRIS Explorer visual programming environment) and connected to other modules exactly as normal. There is no need for the user to learn any additional skills: the transition from single user to multi-user environment is seamless.

A collaborative session begins when one party launches a local server module and enters the contact details of the central server. Any collaboratively aware modules subsequently launched use information set by their local server to find and connect to the central server by means of a UNIX socket. The act of connection initiates a unique data route within the server and causes the automatic launch of corresponding modules in the other connected sessions; together these modules form a group. The architecture is designed to allow participants to join and leave during the lifetime of the session; if a party connects to an already running session then the central server automatically sends commands to the new local server enabling the new party to catch up on the current set of CAMS.

Once a CAM is connected to a data route, any data passed into it from the local session is forwarded to the central server which then distributes this data to all other group members. Participants can identify shared data by means of the unique group ID attached to each module which is consistent between all participants. In addition to being able to join and leave a running session participants may disconnect single modules From their group to allow periods of independent work on a subset of the pipeline while remaining in contact with the rest of the session. Disconnected modules may be re-connected subsequently to their group and the results shared once more.

Moving from a one-to-one model to the current multi-participant architecture has raised a number of issues in respect of scalability and efftciency. The chosen system uses a reliable protocol, TCP, and routes all data through the central server. Theoretically as participant numbers increase the centralised routing could become a bottleneck, with data queuing to get through the server while some large data object is transmitted. To date this has not been a problem, because the architecture encourages replication of the visualization process rather than duplication of the visualization data. If necessary, however, a mechanism for resolving this issue

256

Proceedings of the 8th IEEE Visualization '97 Conference 1070-2385/97 $10.00 © 1997 IEEE

would be to create a multi-threaded server, with each data route occupying its own process.

As participant numbers increase, so the required bandwidth increases with multiple copies of the data being transmitted. Also the use of a reliable protocol such as TCP incurs an overhead as the system must wait for an acknowledgement of each packet. One way of reducing the consequent network load would be to use a cheaper protocol such as UDP or multicast, the latter being capable of making a significant reduction. However both of these protocols are unreliable. While this is acceptable for certain types of data streams such as live video or audio, it is not suitable for sending scientific data. Here the loss of any part of that data can leave the systems out of synchronisation, or leave some participants with corrupted data to visualize. Until reliable low- bandwidth protocols become the norm, our architecture provides a means to tailor the collaborative session to suit different network and computational capabilities.

4. DEMONSTRATION In this section, we run through a simple demonstration of the collaborative IRIS Explorer system. The scenario - totally hypothetical - is that of two doctors collaborating over the analysis of medical image data. One doctor, whom we shall call Dr Bone, has access to CT data of a 2D cross-section through a patient’s head; the other doctor, Dr Blood, at another hospital, has access to SPECT data ftom the same 2D cross-section. CT data

shows structural information such as bones and soft tissue; SPECT data shows functional informatioln such as blood flow.

The scenario begins with Drs Bone and IBlood each studying their respective datasets (Figure 6). Bone calls Blood over the videophone and suggests a collaborative look at the patient data: each launch their local server module,, and enter the machine name and port number of the central server. Blood launches a ShareLat module in order to send his blood data to Bone; a corresponding ShareLat module is automatically launched in Bone’s map editor, and she wires it directly in to her map.

Bone can now see Blood’s data alongside her own and the same procedure can be followed to allow Blood to view Bone’s data (Figure 7). In order to allow them to discuss features in the images, using the videophone for voice dialogue, it proves very useful for each to have a pointer. Figure 8 shows Bone using a pointer to indicate an anomaly in the patient data: notice the pointer geometry is connected to a ShareGeom module so that Blood can receive it, and see where Bone is pointing.

Some progress is made, but they agree they need a better visualization to correlate the two sets of data. Bone suggests a height field plot, where the two datasets appear together: CT as surface height, SPECT as surface colour. Bone, being a more experienced IRIS Explorer user, knows that DisplaceLat is the appropriate module. She launches the module in her own map editor, and wires Blood’s SPECT data in alongside her own.

Dr Blood Dr Bone

Dr Blood

Figure 6: Each Separately View Their Own Data

Dr Bone Figure 7: Each View All Data

257

Proceedings of the 8th IEEE Visualization '97 Conference 1070-2385/97 $10.00 © 1997 IEEE

Dr Bone Figure 8: Pointer Sharing To Identify Specific Features

Dr Blood Dr Bone Figure 9: Sharing Control Parameters On A Height Field Plot

Figure 9 shows the resulting pipeline, and visualization. By means of the advisor module, Bone is able to launch the appropriate corresponding modules in Blood’s map editor in order for him to see the visualization too.

Finally, using a ShareParam module wired to the scale port of the DisplaceLat module in each pipeline they are able to jointly control the surface height (also shown in Figure 9). Blood however, being the SPECT expert, has sole control of the colour map used to represent the SPECT data. They are now able to collaborate quite closely, each contributing their particular expertise.

5. CONCLUSIONS This paper has described a new approach to collaborative visualization. It allows existing modular visualization environments to be extended in a very natural way to accommodate multi-user working. The design aims set out in section 2 have all been met, as illustrated in the example presented in section 4:

. Multiple independent participants: Drs Blood and Bone each worked with their own instance of the MVE, and chose when and how to collaborate - i.e. this is collaborative programming. A third participant could have joined the collaboration if needed.

. Data exchange: In the scenario, both raw data and geometry data were exchanged between Blood and Bone

. Shared control: In the final sequence, control of the appearance of the height field plot was shared by the two doctors.

258

Proceedings of the 8th IEEE Visualization '97 Conference 1070-2385/97 $10.00 © 1997 IEEE

Dynamic collaboration: The form of collaboration was not pre-defined; Blood and Bone only decided on the height field collaboration after the image comparison was initially tried. Instructor mode: Bone, more experienced in visualization, was able to launch modules for Blood. Ease of learning: The two doctors, already knowing IRIS Explorer, had little extra to learn in order to work in collaborative mode. Application mode: Having defined a useful visualization pipeline, the doctors could then encapsulate it as an application for colleagues to use who may not know IRIS Explorer.

This approach promises to radically change the way research groups analyse their results. It is no longer necessary for the members of the team to come together at a single location: they can work independently, coming together over the network in a collaborative session as they please. The flexibility of the approach means that it can handle different forms of collaboration: the participants might be teacher and student, or vendor and customer, just as easily as joint researchers. In short, visualization can now be properly supported as ,a collaborative activity.

REFERENCES [I] H.D.Lord “Improving the Application Development Process

with Modular Visualization Environments”, Computer Graphics, 29(2): 1 O-1 2, 1995.

[2] J.Clucas, “Interactive Visualization of Computational Fluid Dynamics Using Mosaic”, Proceedings Second International WFI’/w Conference ‘94, Chicago, October 1994.

[3] D.Foulser. “IRIS Explorer: A Framework for Investigation.” Computer Graphics, 29(2):13-16, 1995.

[4] CSCW in AVS and IRIS Explorer. ONERA : Office National d’Etudes et de Recherches Aerospatiales. http://visu-www.onera.fr/onera/cscw.htm, 1995.

[5] R.B.Haber and David A. McNabb, “Visualization Idioms: A Conceptual Model for Scientific Visualization Systems”, Visualization in Scientljic Computing, IEEE, pp 74-93, 1990.

[6] G.Abram and L.Treinish, “An Extended Data-Flow Architecture for Data analysis and Visualization”, Computer Graphics, 29(2): 17-21, 1995.

[7] M.Young, DArgiro and S. Kubica. “Cantata: Visual programming environment for the Khoros system.” Computer Graphics, 29 (2):22-24, 1995.

[S] H.G.Pagendami and B.Walter, “A Prototype of a Cooperative Visualization Workplace for the Aerodynnamicist”, Computer Graphics Forum, 12(3), EC 93 Conference Issue, pp 485-496, 1993.

[9] ProShare Personal Conferencing, Intel Corporation, http://www.inteI.com/proshare/index.htm, 1997.

[lo] W.Minenko, “The Application Sharing Technology”, The X Advisor, Volumel( 1) June 1995, Discovery Publishing Group,

r1 II

[121

[131

http://landru.unx.com/DD/advisor/docs/jun95/jun95.minen ko.shtml

CUpson, T.Faulhaber, D.Kamins, D.Laidlaw, DSchlegal, J.Vroom, “The Application Visualization System: A Computational Environment for Scientific Visualization.” IEEE Computer Graphics and Applications, 9(4):30-42, 1989.

A.Wierse, U.Lang, RRhtile, “Architectures of Distributed Visualization Systems and their Enhancements”, Eurographics Workshop on Visualization in Scient@ Computing, Abingdon, 1993.

J.D.Wood, H. Wright, K. W.Brodlie, “CSCV- Computer Supported Collaborative Visualization” in proceedings BCS Displays Group International Conference on Visualization and Model&g, Leeds, 1995. To be published by Academic Press, 1997.

ACKNOWLEDGEMENTS This research was carried out as part of the COVISA project, funded by the UK Engineering and Physical Sciences Research Council. A number of people have contributed to this work by testing and commenting upon intermediate versions of the software we have developed - in particular we would like to thank Alison Tomlin and Justin Ware. We would also like to thank Bill Crum of our Medical Physics Department for the data used in the demonstration and Jeremy Walton and Robert Iles at NAG Ltd for helpful advice.

259

Proceedings of the 8th IEEE Visualization '97 Conference 1070-2385/97 $10.00 © 1997 IEEE