elsaddik @ site.uottawa.ca abed @ mcrlab.uottawa.ca ...elsaddik/abedweb/teaching/elg5121/pdf/13_MM… · Multimedia Communications ... ØOriginally, the term referred to 'Immersive

Page 1

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om

101_Introduction(c) elsaddik

Multimedia Communications

Multimedia Technologies & Applications

Prof. Dr. Abdulmotaleb El SaddikMultimedia Communications Research Laboratory

School of Information Technology and EngineeringUniversity of Ottawa

Ottawa, Ontario, Canada

elsaddik @ site.uottawa.ca

abed @ mcrlab.uottawa.ca

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Interacting with computers

ComputerWorld

Page 2

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


New Paradigm for VR

Conventional Computer

ComputerWorld

VR-based Computer

ComputerWorld

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Virtual RealityØ The idea:vVR is based on the idea of generating “virtual worlds” that

contain computer-generated three dimensional (3D) objects. Optimally, a human being could interact with such world as naturally as with the real world.

ØCharacteristics: vVirtually real : events not occurring in the real world, but from

a human-observer point of view, there is no practical difference between the computer generated objects and the objects in the real world vA VR user could touch, see, hear and smell the VR objects as

he/she would in the real world.

ØApplications:vA superficial analysis of such system will quickly lead to the

conclusion that the application areas of such system would be numerous:

• entertainment,• military,• education, medicine, • business, architecture • and so on.

Page 3

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


VR and Multimedia

From the Multimedia Communication point of view, virtual reality can be thought of as just another “medium” with all the usual issues related to it:ØQuality of Servicevsystem response, vperception quality (touch, vision, ...), vresolution of objects (shape, color, lighting,

...)ØNetworkingØSynchronizationØSharing environmentsvVery important issue since a virtual world will

bea shared media most of the time.ØDatabase management

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


VR as a media

All medium attempt to create the “virtual presence”.ØTheater, poem, fine arts, novel, telephone,

movies, TV, ...

VR vs. existing mediaØimmersiveØinteractiveØ3-DØmultimodalØmediatedvInformation is not sent back and forth.vMediated environments are created and then

experience.

Page 4

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Virtual Reality

Virtual Reality or Real Virtuality?ØOriginally, the term referred to 'Immersive Virtual

Reality.' vthe user becomes fully immersed in an artificial, 3-

D world that is completely generated by a computer.

ØThe head-mounted display (HMD) was the first device providing its wearer with an immersive experience (1965)

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Virtual Reality

ØDesktop VRvTwo dimensional medium , used for 3-D

ØAugmented Realityvthe ability to see the real world and the virtual

world at the same time. ØImmersive VRvthe user becomes fully immersed in an artificial,

three-dimensional world that is completely generated by a computer

Page 5

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


CAVE

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


CAVE Immersion

Page 6

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Distributed Virtual Environments

Øa simulated world runs not on one computer system, but on several. ØThe computers are connected over a networkvpossibly the global Internet and vpeople using those computers are able to:

• interact in real time, • sharing the same virtual world.

ØOn each host there are a number of "entities“ (things in the virtual environment) that communicate their changing state by sending "update messages".

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


---------------------------------------------

--------------------------------------

-------

CVE

Real-time Interaction-Between the customers-Between customers and salesmen-Between customers and goods

Shared Environment-Global coordinate system-Static background-Dynamic entities

Mutual awareness-Customers can “see” and “talk” to each other in the CVE

--------------------------------------

-------

Server

Distributed Virtual Environment

Page 7

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Collaborative Virtual Environments (CVE)

ØDistributed Virtual WorldsØUsers represented there by their “avatars”

(=incarnations)ØCollaborative work performed in CVE

ØA Collaborative Virtual Environment requires an unconventionally broad range of:vnetworking, vdatabase and vgraphics capabilities

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


CVEsØVirtual Reality spaces with special emphasis on

collaboration.ØCollaboration: Multiple users performing closely-coupled

tasks.ØApplications:vtele-surgeryvcollaborative design/engineeringvtele-trainingventertainment

Page 8

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Shared Virtual Environments for Training http://www.vetl.uh.edu/overview/overview2.html

Virtual Environment Technology Laboratory, Univ. of Houston andNASA Johnson Space Centre, US

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Shared Space: Collaborative Augmented Realityhttp://www.hitl .washington.edu/projects/shared_space/

Human Interface Technology Laboratory (HITL), Univ. of Washington, US

Page 9

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Greenspace Project

http://www.hitl.washington.edu/projects/greenspace/

The project is working on the problems of people communicating with machines and machine arbitrated representations of people

Human Interface Technology Laboratory (HITL), Univ. of Washington, US

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


TransVision: A Collaborative Augmented Reality Testbed

http://www.csl.sony.co.jp/person/rekimoto/transvision.html

•Augmented reality for collaborative designing

•The system uses the palmtop video-see-through display instead of bulky head-mounted displays. The user can see a computer-generated 3D model superimposed on the real world view

Sony Computer Science Lab., Sony Co. , Japan

Page 10

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Virtual Society Projecthttp://www.csl.sony.co.jp/project/VS/index.html

The overall goal of the Virtual Society project is to investigate how the future online community will evolve

Sony Computer Science Lab., Sony Co. , Japan

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


SICS - Swedish Institute of Computer Science, Sweden

Multi-user Virtual Workplace Conferencing

Page 11

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Virtuosi Project, UK.

Fashion PilotØapplication in fashion design. Virtuosi is looking at

ways in which all the people involved can collaborate on the design without leaving their own offices

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


i-Land - FHG-IPSI, Darmstadt – Germany

ØRoomware: to integrate computer components into a roomvDynawall

• drag / drop• throw• ...

vInteracTable

• no strictup / down

• place andmanipulate

• …

Page 12

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


i-Land (cont.), FHG-IPSI, Darmstadt – Germany

ØRoomware: to integrate computer components into a room

vCommChair• integrated computers or laptop “docking”• communication via

wireless LAN• user profiles

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


CVE Data Types

• real-time audio and video data:

• data carrying the video or audio streams in the environment.

• object/scene description data: • data describing an object and its attributes; e.g.,

description of an avatar.• CSCW data:

• this represents traditional 2D collaborative data such as whiteboards.

• control data: • used by the system to perform tasks such as

consistency control, management, and so on.• update messages:

• such as avatar/vehicle motions.• synchronous collaborative update messages:

• Update messages representing a synchronous collaborative task.

Page 13

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


CVE Issues

Øhuman-factors research in VR has traditionally focused on the development of natural interfaces for manipulating virtual objects and traversing virtual landscapes.ØCVEs require the consideration of

vhow participants should interact with each other in a shared space, in addition to how co-manipulated objects should behave.

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


CVE issues

Other issues include: Øhow participants should be represented

in the collaborative environment; Øhow to effectively transmit non-verbal

cues that real-world collaborators so casually and effectively use; Øhow to best transmit video and audio via

a channel that allows both public addressing as well as private conversations to occur; and Øhow to sustain a virtual environment

even when all its participants have left.

Page 14

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Causality in CVE

Time

Simulator A(tank)

Simulator B(target)

“ fire”

“ targetdestroyed”

Simulator C (observer)

eventmessage

real world simulated world

the observer should see the tank fire before the target is destroyed.

• “Things” happen in the real world in a certain order (e.g., cause & effect).• It should appear that events in the simulated world happen in the same order as the real world actions that they represent.• If the message for the “fire” event is delayed in the network, the observer will “see” the target is destroyed before the tank fired upon it!

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


VR Standards: Rendering

OpenGLØ Introduced by SGI in 1992ØMultiplatformØHardware acceleration

DirectXØCreated by MicrosoftØSupports Win9x, WinNT (DirectX 3) and Win2000ØHardware acceleration

VRMLØVRML 1.0 based on SGI’s Open InventorØVRML Architecture Group created VRML 2.0 in 1996Ø ISO standard in 1997ØBrowsers based on OpenGL or DirectX

Java3DØ Introduced by Javasoft in 1998ØMultiplatformØBuild on top of OpenGL and ActiveX

Page 15

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


VRML Browsers

VRML-Browsers are software tools that generate views from a VRML file.

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


VRML & JAVA3D

ØVRML is a file format vIt enables authors to control shapes and

some animation and interaction vFor more control, authors may write short

programs in Java or JavaScript vTo gain greater control, something more is

needed...

ØJava3D is a Java Application Programmer's Interface (API) vJava3D is a set of standardized classes for

use by Java programmers vIt enables authors to control shapes,

animation, and interaction, PLUS rendering, input devices, and lots more vIt is (roughly) a Java-based superset of VRML

Page 16

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


CVE Standards: MiddlewareDISØDistributed Interactive Simulation (DIS) protocol

Living Worlds (LW)ØDesigned by the VRML ConsortiumØAims at providing collaboration features for VRMLØVRML 3.0 = VRML 2.0 + Living Worlds

Open Community (OC)ØProposed by MitsubishiØSPLINE (Scalable Platform for Large Interactive Networked

Environments) is a OC compliant implementationØSPLINE uses ISTP (Interactive Sharing Transfer Protocol)

High Level Architecture (HLA)ØDeveloped by the U.S. Defence Modeling Simulation Office

(DMSO)ØRTI (Run Time Infrastructure) is an HLA compliant

implementationØHLA is in the process of becoming an IEEE standard

Java Shared Data Toolkit (JSDT)ØPart of Java Media API developed by JavasoftØJava provision for real-time sharing of applets and/or

applications

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Distributed Interactive Simulation (DIS) protocol

ØDIS, and its predecessor SIMNET, arose from real-time interactive computing. ØDIS is a government/industry initiative to define an

infrastructure for linking simulations of various types at multiple locations to create realistic, complex, virtual worlds for the simulation of highly interactive activities.ØDIS is a virtual environment within which humans

may interact through simulation at multiple sites that are networked using compliant architecture, modeling, protocols, standards and databases. ØDIS exercises involve the interconnection of a number

of simulators in which simulated entities are able to interact within a computer generated environment. ØThe simulators may be present in one location or be

distributed geographically and the communications are conducted over the network.

Page 17

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


DIS Standards

ØIEEE Std 1278.1-1995 (DIS V 2.0): IEEE Standard for Distributed Interactive Simulation- Application Protocols.ØIEEE Std 1278.2-1995: IEEE Standard for Distributed

Interactive Simulation- Communication Services.ØIEEE Std 1278.3-1996: Recommended Practice for

Distributed Interactive Simulation- Exercise Management and Feedback.ØIEEE P1278.4: Project, Recommended Practice for

Distributed Interactive Simulation - Verification, Validation and Accreditation.ØIEEE P1278.5: Project, Standard for Distributed

Interactive Simulation: Fidelity Description Requirements

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


LiveParticipants

SupportUtilities

Interface

Interfaces to Live Players

Simulations

Runtime Infrastructure

Federation Management Declaration ManagementObject Management Ownership ManagementTime Management Data Distribution Management

• OMG, IEEE, DoD standard

High Level Architecture (HLA)

Page 18

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Example of Distributed Virtual Environment over HLA

federate Afederate B

HLA MiddlewareTCP/UDP/IP Network

federate C

federate A

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Open CommunitySPLINE

Page 19

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Open Community, SPLINE

SPLINE (Scalable Platform for Large Interactive Networked Environments) is a software library compliant with O.C. which provides an API for C (and promises of Java soon).

ISTP (Interactive Sharing Transfer Protocol) is a hybrid protocol used by SPLINE which supports the sharing of information about a Virtual World among a group of user processes.

OC (Open Community) is a proposed standard which provides many essential services necessary to make real-time multi-user cooperative environments possible.

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


OC (SPLINE) Functionality

Network Communication issuesØISTP

Real-time audio transport ( no video)

Application neutral transport of large or complicated objectsØVRML models, sound files, etc.

Filtering of Regions of Interest

Scalability to large Virtual World and large number of users allowing users in not so powerful stations to participate.

Page 20

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


OC World Model - High Level Overview

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


SPLINE: Scalability

World is partitioned in regions called locales.ØEach process is supposed to render a

reduced number of locales (1- It’s own locale, 2- Neighbour locales)

Each locale has its own multicast addressesØEach process only deals with relevant

messages/audio

Each locale has its own coordinate systemØDifferent parts of the VW can be designed

independently

Page 21

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


SPLINE: Locales

Each locale has a list of adjacent localesØThis list includes coordinate transformations

as wellØPortals (non-Euclidean motion)

Locales may overlap, be enclosed or contain empty spaces

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


CVE Projects @ MCRLab U.of Ottawa

ØDIVERTIONS : Distributed Virtual Environments with Training Applications [NCIT]ØDISCOVER: Distributed Information System for Collaborative Virtual

Environment Research [CITO ]Ø VIRTUAL THEATER: Tele-immersion using Internet Technologies and

Distributed Virtual Environments for Training and Telecollaboration[Stentor New Media Fund]Ø VEHICLE: Virtual Environments for Human Interaction,

Communication and Learning [ORDCF]Ø VELVET: Very Large Virtual Environments [CITO]ØDistributed Virtual Environments in Industrial Training

[ALCATEL/Newbridge]ØDistributed Virtual Environment User Interface and Intelligent Agents

for Electronic Commerce [CITR] ØDistributed Interactive Virtual Environment (DIVE) over CA*net II: QoS

Management, IPv6 and Performance Tools [CANARIE] Ø Virtual Reality for promoting Children's Pedestrian Safety [Ontario

Neurotrauma Foundation]

http://www.mcrlab.uottawa.ca/research/Application to E-commerce, Tele-training, Virtual Theater

Page 22

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


VIRTUAL THEATER for Industrial Training: A Collaborative Virtual Environment

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


CVE for Industrial Training

ØReduces Training ExpensesvWaste of equipment which likely gets

damagedvMaintenance of Training FacilitiesvTraining of users in remote locations

ØVery little cost associated with creating virtual objectsvIt can be replicated as many times as

necessaryØAny damage done to the object does not incur in

financial lossØMaintenance fee is very lowØActual objects may be used for its intended use

Page 23

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Industrial Training Prototype

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


DirectoryServer

VideoServer

C++ (dll)

Java3D

JNI

Java 2

JavaSwing

VoiceRecognition

(ActiveX)

JNI

HeadTracking(ActiveX)

JNI

To other clients

VideoDecoder

System modules & Connectivity

Page 24

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


ØBased on Java3DØ3D objects designed with

PowerAnimator and later on optimized to make them treatableØVRML objects and

behaviours are imported into Java3D

3D Rendering

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Trainer shows how to perform the operation while trainee watches it

Trainee is then able to perform the same operation under supervision of trainer

Replacing a Faulty Card

Page 25

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Second View allows trainee to see details (other point of view)

H263 video shows relevant information and/or details

2nd View & Video Streaming

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Users may control the scene by three aproaches:Ø Point-and-ClickØ Menu SelectionØ Voice Commands

Speech recognition has been shown to be a natural choice among users of the system.

Pre-defined commands such as “Go to the table” allows easy navigation by novice users.

Commands defined by SAPI grammar.

Speech Recognition

Page 26

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Ø Head tracking allows natural way of controlling the avatar’s head.

Ø Users have a better awareness of others while listening to theirvoice and watching their avatar’s animation

Ø Extension to body motion detection allows Virtual Theater like collaboration

Ø 3D Scene is projected in a wall for stereoscopic view through 3D shutter glasses and more natural experience for the users.

Head Tracking

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Stereoscopic View

Users can view and interact with a stereoscopic 3D version of the training room at the computer’ screen as well as projected in the wall through the use of shutter glasses

Combined with voice recognition and gesture tracking allows a very natural and effective interaction.

Page 27

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Voice-enabled VRML User Interface

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Page 28

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Object Management

RTI Interface

Java Object ControllerObject Management

RTI Interface

Java Object Controller

HLA RUN-TIME INFRASTRUCTURE

Collaborative E-Commerce by CVE

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Virtual Theater project

VisualFeedback(s)

Machine/ Human Interface

TactileFeedback(s)

ForceFeedback(s)

VideoSensor(s)

TactileSensor(s)

ForceSensor(s)

StructuredLight

Human/Machine Interface

Object Recognition

Motion TrackingVirtual Object Manipulation

Object Shape &Behavior ModelsObject Interaction Models

Computer Generated Models Animation Script

Pseudo-Random Encoding

VIRTUAL SCENE / THEATER (where virtual objects interact)

Page 29

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


VEHICLE: Virtual Environments for Human Interaction Communication and Learning

• Use of emotional and behavioral agents for managing a CVE

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Mobile Collaborative Augmented Reality

Computer Supported Collaborative Work (CSCW)

ØAllows a computer to be used as a medium for human communication

Mobile Collaborative Augmented Reality

ØThe combination of augmented reality, mobile computing and CSCW promises exciting applications for industrial training

Page 30

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Augmented Reality (AR) Prototype

A trainee works with a wearable computer

nWhere and which virtual objects should be superimposed on the real world?

nA marker with a square region and some pattern inside of the square region is used

nSupports multiple trainees and trainers distributed in different locations

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


AR Prototype (cont’)

(a) A physical board with virtual chips around

(c) A virtual chip is laid on the board

(d) The virtual chip is rotated by remote trainee B

(b) Local trainee A is moving a virtual chip

Page 31

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


(PHANTOM)

Ø Incorporate SensAble’sPHANTOM

Ø Write Java Native Interface to PHANTOM libraries

ww

w.s

ite.u

otta

wa

.ca/

~els

addi

kw

ww

.el-s

add

ik.c

om


Conclusion

ØCollaborative Virtual Environments open new ways for human interaction, collaboration and learning ØMany Applications:

vMedicine, vTraining, vTelecollaboration, vE-commerce,….

ØPhychological and social issues: an open area for study.

Documents

elsaddik @ site.uottawa.ca abed @ mcrlab.uottawa.ca ...elsaddik/abedweb/teaching/elg5121/pdf/13_MM… · Multimedia Communications ... ØOriginally, the term referred to 'Immersive