Component Based Video Communication Toolfor Collaborative Virtual Environment

8/3/2019 Component Based Video Communication Toolfor Collaborative Virtual Environment

1/6

Component Based Video Com munication Toolfor Collaborative Virtual Environment

Hirotatsu Sakamoto Yoshihiro Okada Toshihiko Shimokawa Kazuo UshijimaGraduate School of Information Science and Electrical Engineering, Kyushu University

6-10-1 Hakozaki, Higashi-ku, Fukuoka, 8 12-8581, JapanPhone: +8 1-92-642-3872, Fax: +81-92-583-7632

[email protected] [email protected] p

AbstractThis paper treats a component based video communica-

tion tool or collaborative virtual environments. Especiallythe authors propose a new concept and its realizationmechanisms o r easy construction ofdistributed 30 graph-ics applications using video comm unications , e.g., networkmeeting, conference, and training in a 30 virtual space. I fa software compon ent is represented as a visible, manuallyoperable object, users can make its copy and transfer itto another computer easily and rapidly. I f a facility thatmanages video/audio data is realized as such a object,even end-users can easily and rapidly build networkedvideo communication environments through the copy-and-transfer operation.

To clarify an availability of this concept, the authorsemploy IntelligentBox system as a research plagorm. Theauthors introduced a video communication facility as asoftware component into the IntelligentBox system. Intel-ligentBox has provided a network comm unication fa cil i5as a software component. Using these components, it willbe possible to build collaborative virtual environments,which support video communications, through copy-and-transfer operations.1 Introduction

At present, advances of information network technolo-gies made it possible to use broad band networks. Then ,in the near future, high quality multimedia communi-cations will be com mon and inevitable to use. Ma nyresearches on multimedia communications have alreadybeen done. Most of them focused on the development ofthe multimedia comm unication systems such as networkmeeting systems, teleconference systems, and networklecture systems[2][3][5][8]. S om e of them focused on mul-timedia com munications in a 3D virtual environment[ ].Distributed 3D virtual environments are sometimes called

[email protected] [email protected]

collaborative virtual environments or distributed virtualenvironments. Many researches on collaborative virtualenvironments have been do ne so far because advances ofcomputer hardware technologies made it possible to gen-erate realistic 3D images in real-time. Then we are alsointerested in collaborative virtual enviro nmen ts. Especiallywe are currently focusing on multimedia comm unicationsin a collaborative virtual environment since videolaudiosupports are necessary for smooth com munications in var-ious applications.

Our research purpose is to establish the essential mech -anisms that enable even end-users to develop collaborativevirtual environments. In this paper, w e propose a new con-cept called copy-and-transfer operatio n and its realizationmechanisms for easy construction of collaborative virtualenvironments that support video communications. If asoftware component is represented as a visible, manuallyoperable object, users can make its copy and transfer itto another computer easily and rapidly. This copy-and-transfer operation is similar to the copy-and-pastelcut-and-paste operations, which a re standard GUI operationsbased on using a mouse-device. If a facility that managesvideolaudio data is realized as such a visible, manuallyoperab le object, even end-use rs would be able to build net-worked video com munication environments only throughth e copy-and-transfer operation.

To clarify availability of the copy-and-transfer oper-ation, we employ IntelligentBox system[6] as a researchplatform. IntelligentBox i s a component ware for devel-oping interactive 3D graphic s applications. IntefZigentBoxprovides various software components as visible and man -ually operable 3D objects called boxes. IntelligentBoxalso provides a dynamic data linkage mechanism calledslot-connections to enable users to co nstruct 3D graphicsapplications by co mbining existing boxes through directmanipulations on a computer screen.

IntelligentBox has already provided video m anaging fa-cilities as boxes called MovieBox an d VideoBox. However

3750-7695-0951-7/0110.000 001 EEE
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


2/6

these boxes were not available through network. Then w eimproved them in order to support video communicationsvia network. IntelligentBox has also provided a networkcommu nication facility as a bo x called RoomBox [ 7 ] .Mul-tiple RoomBoxes on different computers share specificusers-operation events with each other. Those RoomBoxesvirtually provide multiple users with a shared 3D space. Itis possible to import any already developed boxes into theshared 3D space without any modifications. Therefore, it ispossible to construct a distributed 3D virtual environm entby making a copy of on e RoomBox existing on a certaincomputer and transferring it to another com puter.

Using these boxes, i.e., RoomBox, MovieBox, an dVideoBox, it will be possible to build collaborative vir-tual environments, which support video comm unications,only through copy-and-transfer operations. In this process,end-users will not have to w rite any text-based programs.

In this paper, we clarify availability of this conceptof copy-and-transfer operation by discussing developmentcosts and performances with show ing possible, practicalapplication examples.[Related works]

One of the collaborative virtual environment system s us-ing multimedia communications is a teleconference system.Very popular produ cts are Cornell Universitys CU-SeeMeand Microsoft NetMeetingTM. Famous free software arevic and vat[ 1 I]. Many teleconference systems besidesthem have been made so far. Hond a, et a1.[4] propo sed avirtual office based on a shared room concept. Yoshida,et al.[ 101 proposed a virtual space teleconference system.This system supports intuitive interaction for creative andcooperative work. Although these are very powerful sys-tems, it is not easy to use their essential mechanisms whendeveloping distributed 3D graphics applications. Our re-search purpose is to clarify software architecture that mak esit easier to develop distributed virtual reality applications.This is major difference between o ur research and others.

The remainder of this paper is organized as follows:Section 2 explains essential mechanisms of IntelligentBoxan d RoomBox. In section 3, we explain realization mech-anisms of MovieBox an d VideoBox. Section 4 discussesdevelopment costs and performances with showing possi-ble, practical application examples. Finally, we concludethis paper in section 5 .

2 Essential mechanisms of ZnteZZigentBoxIntelligentBox employs the following essential mech-

anisms inherited from IntelligentPad [9], which is a 2Dsynthetic media system since IntelligentBox is an extensio nof IntelligentPad to 3D graphics applications.

2.1 MD (Model-Display object) structureAs show n in Figure I , each bo x consists of two objects ,

a model and a display object. This structure is called anM D (Model-Display object) structure. A model holds statevalues of a box. They are stored in variables called slots. Adisplay object defines how the bo x appears on a computerscreen and defines how the bo x reacts to user operations.

Figure 1 also shows messages between the display ob-ject and, the model. This is an examp le of a RotationBox.Through d irect manipulations on a box, its associated slotvalue changes. Furthermore, its visual im age simultane-ously changes according to the changes of the slot value.Then a box reacts to the user manipulations according toits function.2.2 Message-sending protocol for slot connec-tions

Figure 2 illustrates a data linkage concept betweenboxes. Each bo x has multiple slots. Its one slot can beconnected to one of the slots of other box. This connectionis called a slot connection. T he slot connection is carried outby three messages when a parent-child relationship existsbetween two boxes. Three messages are a set message, agimme message and an update message.

A (value) in the format @ represents any value, anda (slotname) in the formats @ an d @ represents a user-selected slot of the parent box that receives these twomessages.A se t message writes a child box slot value into itsparent b o x slot. A gimme message reads a parent b o x slotvalue and sets it into its child box slot. Update messagesare issued from a parent box to all of its child boxes to tellthem that the parent box slot value has changed. In thisway, these three messages connect a child box slot and aparent bo x slot, and com bine their two functionalities .2.3 A shared-copy and a distributed model-sharing

As previously mentioned, each box consists of tw oobjects, i.e., a model an d a display object. T he M Dstructure allows more than one bo x to share the samec ommon model. This mechanism is called model-sharingand the operation that generates a copy of a display objectsharing a c ommon model is called shared-copy. A bo xgenerated by shared-copy shares all slot values. Afterone of model-shared boxes is transferred to other host vianetwork, new corresponding box is generated in that hostand the bo x has the same slot values and keep them alwaysby messages via network to conserve consistency of slotvalues. T his makes it possible to realize model-sharing vi anetwork.

376


3/6

0 PICFigure 1: An MD structure of abox and its internal messages Figure 2: Standard messagesbetween boxes

2.4 RoomBox for collaborative virtual environ-ments

This section briefly describes an idea of a shared 3Dspace and a functionality of the RoomBox. As shown inFigure 3, the RoomBox has a slot name d 'event' whichholds a current user-operation event on its descendantboxes. Som e specific user-operation even ts generated in aRoomBox are always stored in this slot until the next event isgenerated. As shown in Figure 3 , IntelligentBox provides adistributed model-sharing mechanism. I t enables multiple,distributed RoomBoxes to virtually sha re a commo n modelwith each other through messages passed via network.By the distributed model sharing mechanism, multipleRoomBoxes can share user-operation events with eachother. H ere, descendant boxes of a RoomBox are treated a scollaboratively operable 3D objects.

In the example case shown in Figure 3, there are twoRoomBox models existing separately on a different com-puter. These two models are kept in the same state bymessages passed via network. This linkage is built easilyand rapidly by making a shared copy of a RoomBox onone comp uter and by transferring it to the other com puter.When a user operates one box in the RoomBox on thecomputer A, his operation event is sent to the RoomBoxmodel and subsequently set in its event slot. Furthermore,this event is sent to the other RoomBox model existingon the computer B by a message. After these processesare completed, the operation event is applied to the cor-responding box on the computer B. In this way, by usingdistributed RoomBoxes , user-operation events are sharedamong several computers.3 MovieBox and VideoBox for video commu-nications in collaborative virtual environ-

mentsAs previously mentioned, with using RoomBoxes, it ispossible to build collaborative virtual environ ments easily

and rapidly. Based on RoomBoxes, several distributed 3 D

0 3Dwaudve ol$'ecls(my Bores)ComputerAI1 I :

M&l of K"Bm ModelofRoOmBm

F i g u r e 3: Message flow between twoRoomBoxes for network collaborationgraphics applications have been m ade so far. If video com-munication facilities become available in the IntelligentBoxsystem, these applications become more and more prac-tical than now. Originally video managing facilities, i.e.,MovieBox an d VideoBox, have already existed. MovieBoxhas a function to play a movie file, and VideoBox has afunction to get each imag e from a video camera on live andto display it o n a computer screen. However both boxescould not support video communications. Then we ex-tended their functions to be applicable to various networkapplications that need video communications.3.1 The design of M o v i e B o x and V i d e o B o x

Both MovieBox and VideoBox use the texture-mappingtechnique to display a 2D image 'in a 3D virtual space.Strictly speak ing, a texture image is mapped on the surfaceof a 3D object, i.e., a box. Then each frame of a m ovie filestored in a storag e or each frame of video data captured bya video camera appears on a computer screen. As shownin Figure 4 both MovieBox and VideoBox have a 'frame'slot in its model. A texture image is loaded and stored inthis slot. To periodically update content of the slot allowsus to see an animation.3.1.1 MovieBox

The left figure in Figure 4 llustrates data flow concern-ing the MovieBox. Indeed the MovieBox has the followingslots in its model: 'moviefile ' , 'currentNo', ' increment','TRIGGER', and 'frame'.

A 'moviefile' slot value is a name of a movie filerequested to play. A 'currentNo' slot stores an indexnumber of a current displayed frame. The next frame tobe displayed is determined by the next 'currentNo' slotvalue. The next 'currentNo' slot value becomes a sumof a current 'currentNo' slot value and an 'increment'slot value. Whenever a 'TRIGGER' slot is accessed, a'currentNo' slot value is updated and the next frame willbe loaded. Actually a TimerBox is used to access the'TRIGGER' slot periodically by a slot-connection since a

377


4/6

MovieBox 1 1 VideoBox I1 - I t i l I i I I itrigger

I I I IDisplayObject m Model

F i g u r e 4: The mechanism of MovieBox andVideoBoxTimerBox has a slot named 'time', and the 'time' slot valueis automatically increased by a timer process. Finally a'frame' slot stores a binary 2D image data.3.1.2 VideoBox

The right figure in Figure 4 illustrates data flow concern-ing the VideoBox. Indeed the VideoBox has the followingslots in it s model : 'DriverNo', 'TRIGGER', and 'frame'.A 'DriverNo' slot value indicates a video capture driversince multiple video capture drivers can be installed. Whe n-ever a 'TRIGGER' slot is accessed, the VideoBox capturesnew video fram e data and stores it into the 'frame' slot. Aswell as the case of MovieBox, a TimerBox is used to accessthe 'TRIGGER' slot periodically. Finally a 'frame' slotstores a current frame, which is a binary 2D image data.

Figure 4 shows the mechanism of MovieBox an dVideoBox. The difference between two boxes is theirsource where a texture image data is generated, i.e., incase of MovieBox, its source is a mov ie file and in case ofVideoBox, its source is a video capture camera.

3.2 Extentions of MovieBox and VideoBox forcollaborative virtual environmentsBefore introduction in this paper, MovieBox an dVideoBox have already been implem ented. However their

distributed shared copies were not possible because the'frame' slot has not a texture image data but an indexnumber indicates a content of a texture table for savingmemory area. This time, to realize video communica-tions, we modified the ' frame' slot to be able to storean image data itself. Then it became possible to make ashared-copyof th e MovieBoxlVideoBox and transfer it toanother computer to build a video commun ication environ-ment as shown in Figure 5. Furthermore we imp roved theMovieBoxlVideoBox in order to cope with the RoomBox.Remem ber that the RoomBox treats user-operation eventsof its descendant boxes and multiple shared copies of aRoomBox provide virtually a collaborative environment.We made it possible that the RoomBox treats the event ofupdating the 'frame' slot value of the MovieBoxlVideoBox

Q- -PI I

Display obie~ fa do l

Figure 5: The mechanismof video communication

as the same way as other user-operation events. Then videocomm unication s becam e possible in collaborative virtualenvironments built based on the RoomBox.3.3 AudioBox for audio communications

For developing actual applications, audio and videocommu nications are both necessary. We have almostfinished implem entation of software com ponen ts, i.e., theVideoBox an d MovieBox. For audio communications,we have not yet implemented. IntelligentBox provides aparticular box called SoundBox for playing a soun d file butthis bo x is not available on network. We are suppo sed toimprove the SoundBox to make it available on network, andwe are trying to introduce a new bo x called AudioBox fo raudio communications. Essential m echanisms of AudioBoxwill be the same as VideoBox. After realizing these boxes, itwould be possible to develop practical applications shownin the next section.4 Discussion

This section discusses possible, practical applicationexamples, their development costs and performances.4.1 Practical application examples(1)Teleconference systems

Figure 6 shows a screen image of a teleconferencesystem. A conference attendant is represented as a robot.Each robot's h ead is the comb ination of a CameraBoxand a VideoBox . In this example, conference attendantsare five but four robots are seen since the screen imageof the figure is the camera view of the remaining robotcontrolled by the user using the computer of this screen.Al l boxes are software components so that even end-users can construct this application example only throughdirect manipulations, i.e., copy-and-transferoperations, ona com puter screen. Strictly speaking, in this example,each of multiple attendants makes four copies of his ownrobot and mutually transfers each copy to each of other

378


5/6

Figu re 6 : A teleconference system examplecomputers. Then a video com munication environment likethis will be built.(2)Tutoring systems

Figure 7 shows a screen image of a tutoring system. Themodel appearing on the main window of IntelligentBoxmeans a printer consisting of a top cover, a back cover,a paper tray and a toner cartridge. In this exam ple case,the user using the computer of this screen is a learnerand the person whose face appearing on the screen is atutor. The tutor teaches the learner a component structureand the way to manipulate a printer, e.g., open a to pcover and remove a toner cartridge from the insid e, draw apaper supply, and open a back cover to remove a jammedpaper. User-operation ev ents are shared by using RoomBoxand users voices and facial images are shared by usingVideoBoxlAudioBox so that the tutor can teach the printeroperatio ns by live demonstrations with oral explanations.4.2 Development costs

We discribe as follows the simplest way to createan application for video comm unication using VideoBox,RoomBox an d TimerBox,

to compose a composite box from a TimerBox an d aVideoBox,to connect the time slot of the TimerBox and theTRIGGERslot of the VideoBox by a slot-connectionthrough a men u selection,to define the composite box as a descendant of aRoomBox, andto make a copy of the RoomBox and transfer it toanother computer.

In this way, construction process for video commun ica-tion, based on a copy-and-transferoperation, is very simpleand easy for even end-users. The application examples ex-cept videolau dio communication facilitie s presente d in thispaper were actually developed in less than a few hourswithout writing text-based progra ms. Tho se were con -structed only through direct manipulations on a computerscreen using a mou se device.

Figure 7: A tutoring system example4.3 Performances

Performance of video communication depends on suchmeasures as frame rate, frame imag e quality, and networkbandwidth. Th e frame rate is the number of fram es to bedisplaye d per a unit time.

The fram e image quality is determined by two factors,imag e sizelresolution and information quantity of one pixelcolor. Of course, higher frame rate and higher frame im agequality are both desirable if network bandwidth allows.However such a situation is rare. Therefore we introducedthe protocol slot into MovieBox an d VideoBox as on esolution so that users can choose a protocol, TCP orUDP, according to the network bandw idth they can use.Furthe rmor e we will make it possible that users can directlychoose the frame rate and frame image quality accordingto application types.

Here we consider the relation among the frame rate,frame image quality and the network band width. In In-telligentBox , nformation quantity of one pixel color of atexture image is four bytes, RGBA. R , G , an d B mean red,green, and blue color valu es respectively. A means a valuewhich is a transparency factor.

In fact, the a value is not used. Then informatio nquantity of one pixel color is reduced into three bytes. Inthe case that users desire the real-time comm unicatio n, i.e.,30 fps, and the higher resolution of one frame image, i.e.,25 6 x 256 pixels, the necessary bandwidth is calculated asfollows:

25 6 x 256 x 3 x 8 x 30 = 45 Mbps(bit per second)Generally we are not allowed to use a high bandwidth

network like this. But if the frame rate requirem ent isdecreased to 20 fps, the necessary bandwidth becomes30 Mbps ( = 256 x 256 x 3 x 8 x 20). Additionallyif the resolution requirement is decreased to 128 x 1 28pixels, the necessary bandwidth becomes 7.5 Mbps ( =12 8 x 1 2 8 x 3 x 8 x 20). Furthe rmor e if informationquantity of one pixel color is decreased to one bye, i.e.,using a packed color, the necessary bandwidth becomes2.5 Mbps .

Currently we are experimenting real-time network col-laborations using an ATM communication network. Wecan use it in up to 30M bps bandwidth as a virtual dedicated

379


6/6

commun ication line. W e will check that real-time com-munications including video/audio communications arepossible without any stress. We will report these findingsin the near future.

We have been developing three different versions ofIntelligentBox corresponding to machine environments,i.e., a Windows version, a FreeB SDL inux vers ion, andan SGI (IRIX OS) version. Now we are developingmainly Windows and FreeB SDL inux vers ions. In thecase of the FreeBSDLinux vers ion, X window systemseems to be a. bottleneck since the Windows version isfaster than the Free BS DL inu x version as for its renderingspeed. T his reason is obvious. X Free86, X window sys-te m of our FreeB SDL inux environment , does not supportOpenGL nor Mesa3D hardware accelerations. Especiallythe texture-mapping technology strongly needs a hardwareacceleration support.

IntelligentBox employ s the unicast comm unication. Incase of multiple-attendants commun ication, loads of com-puters become h eavy and traffic of network becomes high.Then we are trying to introduce the multicast communica-tion instead of the unicast. W e will report perform ances indetail after realizin g this.

5 Conclusing remarksThis paper presented the component based video com -

munication tool for collaborative virtual environments.Especially the new conce pt, i.e., the copy-and-transfer OQ-eration was proposed and its realization mechanisms wereexplained. If a software component is represented as avisible, manually operable object, the copy-and-transferoperation to such an object becomes possible. Then if afacility that mana ges vid eo/audio data is realized as such avisible, manually operable object, even end-users can easilyand rapidly b uild networked video communication environ-ments through the copy-and-transfer operation. To clarifyan availability of this concept, we employed the Intelligent-Box system and introduced a video communication facilityas software components. This paper clarified the avail-ability of the copy-and-transfer operation by discussingdevelopment costs and performances of applications withshow ing possible, practical application exa mples .

Acknowledgements

References[ I ] Banus JW, Waters RC, Anderson DB, Locals and Bea-cons:Efficient and Precise Support For Large M ulti-User

Virtual Env ironm ents, Proc. of IEEE Virtual Reality An-nual International Sym posium, Santa Clara, C A, pp. 20 4-213. 1996.

[2] Fahlen LE, Stahl 0, Brown CG, Carlsson C, A SpaceBased Model for User Interaction in Shared SyntheticEnvironments, ACM INTERCHI 93 New York: pp.43-48. 1993.

[3] Gajewska H, Kistler J, Manasse MS, Redell DD, Argo:ASystem for Distributed Collaboration, ACM Multimedia94: pp. 433-440, 1994.

[4] Honda, S . , Tomioka, H., Kimura, T., Ohsawa, T., Okada,K. and Matsushita, U ,, A Virtual Office EnvironmentBased on a Shared Room Realizing Awareness Spaceand Transmitting Awareness Informatio n, Proc. of ACMUlST97, pp. 199-207, 1997.

[5 ] Ishii H, Kobayashi M, Grudin J, Integration of Interper-sonal Space and Shared Workspace:ClearB oardDesign an dExperim ents, ACM Transactions on InformationSystems,Vol. 11, NO .4, pp. 349-375.

[6] Okada,Y. and Tanaka,Y., 1ntelligentBox:A ConstructiveVisual Software Development System for Interactive 3DGraphic Application s, Proc. of Com puter Animation ! Q5,IEEE Com puter Society Press, pp. 1 14-125, 1995.

[7] Okada,Y. and Tanaka,Y., Collaborative Environments inIntelligentBox for Distributed 3D Graphic Applications,The V isual Computer (CGS special issue), Vol. 14, No. 4,pp. 140-152, 1998.

[SI Streitz NA, GeiBler J , Haake JM, Hol J , DOL-PH1N:Integrated Meeting Support across LiveBoards,Local and Remote Desktop Environments, Proc. ofACM Conference on Comp uter-Supported CooperativeWork(CSCW 9 4), New York: pp. 345-358, 1994.

[9] Tanaka, Y . , Meme Media and a World Wide MemePool, Proc. of ACM M ultimedia 96, pp. 175-186, 1996.

[ IO ] Yoshida, M., Tijerino, A. Y ., Abe, S. and Kishino, F.,A Virtual Space Teleconferencing System that SupportsIntuitive Interaction for Creative and Cooperative Work ,Proc. Of ACM Symp osium on Interactive 3D Graphic s, pp.115-122, 1995.

We would like to thank all members of our laboratoryfor their advices and suggestions. W e also thank allmembers of a special interest group on networks at KyushuUniversity.

[ I 11 http://www-nrg,ee.lbI.gOv/

380

Documents

Component Based Video Communication Toolfor Collaborative Virtual Environment