04/08/2304/08/23 uri="http://www.globalmmcs.org" email="uri="http://www.globalmmcs.org" email="gcf@indiana.edu"gcf@indiana.edu"

11

Design and Implementation of Audio/Video Collaboration System

Based on Publish/subscribe Event Middleware

PTLIU Laboratory for Community GridsGeoffrey Fox, Wenjun Wu, Ahmet Uyar, Hasan Bulut

Computer Science DepartmentIndiana University, Bloomington IN 47404

http://www.globalmmcs.org

gcf@indiana.edu

CTS04 San Diego 19 January 2004

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 22

A/V Collaboration Systems H323

H.323 is defined as an umbrella standard specifying the components to be used within an H.323-based environment.

SIPThe Session Initiation Protocol (SIP) defines how to establish, maintain and terminate Internet sessions including multimedia conferences

Access Grid enhanced Mbone A/V tools ( VIC, RAT )

Internet 2 network ( Multicast support )

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 33

XGSP Collaboration Framework To integrate heterogeneous systems into one

collaboration system• A unified, scalable, robust “overlay” network is needed to

support A/V and data group communication over heterogeneous networking environments.

• A common A/V signaling protocol has to be designed to support interactions between different A/V collaboration endpoints.

• Different A/V endpoints should collaborate in the same collaboration session.

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 44

Group Communication in Collaboration Collaboration applications usually need a group

communication service for both multipoint data and control information delivery• Centralized conferencing systems usually depend upon a

single conference server

• Distributed conferencing systems use IP multicast

Access Grid uses Internet2 multicast for audio/video transmission.

• Problems:

Centralized conferencing systems don’t have good scalability

IP multicast has not become ubiquitously available

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 55

Narada Broker Network

Database

Resource

Broker

Broker

Broker

Broker

Broker

Broker

Software multicast

(P2P) Community

(P2P) Community

For message/events service(P2P) Community

(P2P) Community

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 66

Narada Brokering Narada Brokering from the Community Grid Labs is

adopted as a general event brokering middleware• supports publish-subscribe messaging model with a dynamic

collection of brokers

• provides services for TCP, UDP, Multicast, SSL and raw RTP clients

• Supports the communication through firewalls and proxies

• operate either in a client-server mode like JMS or in a completely distributed JXTA-like peer-to-peer mode.

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 77

Narada Brokering

Covers the heterogeneity of network transportation and provides unified multipoint transportation API• Software multicast • Communication over firewalls and proxy boundaries • Communication over multiple transports • Application level Quality of Service

Filter messages to slow (collaborative/real-time) clients Provides robust, scalable and high efficient multipoint

transportation services • Availability and scalability • Efficient routing and bandwidth utilizations

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 88

A/V Collaboration over publish/subscribe Middleware video streams (VS) VS {v1, v2, … vm }

audio streams (AS) AS { a1, a2, … an }

A/V endpoints: E1, E2, … En

Each endpoint may send a single or multiple video streams, but only send an audio stream

Different types of A/V endpoints have different collaboration capabilities. • Multicast endpoints are able to receive multiple video and audio

streams, display all the video streams in their screens, and mix all the audio streams by themselves

• Unicast endpoints like Polycom Via Video can only receive and play a single video and audio stream

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 99

Publish/subscribe of A/V A stream in VS and AS is regarded as a “topic” Each RTP packet from this stream is regarded as an

“event” for this topic Only the sender of this stream can “publish” A/V

events to this topic Other endpoints need to subscribe to this topic in order

to receive the stream data Create mixed video and audio stream topics for unicast

endpoints

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 1010

Major Events for each stream topicEvent Name Change in the stream

NewStreamEvent This stream has been created

ByeEvent The stream gives a BYE RTCP packet, indicating it has left the session

TimeOutEvent This stream has not send any RTCP packet for a long time, indicating it may have left the session

Active-To-Passive The stream has stopped sending RTP packet

Passive-To-Active The stream resumed sending RTP packet

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 1111

A/V Collaboration Architecture

NaradaBrokeringPublish/Subscribe Messaging

AVSessionServer

SIPGateway

H.323Gateway

H.323Client

SIPClient

AccessGrid

Multicast

XGSPWeb

Server

AudioMixer

VideoMixer

ImageGrabber

Media ServersAudioMixer

VideoMixer

ImageGrabber

RTPLink

RTPLinkRTPLink

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 1212

RTP packets encapsulation RTPLink: RTP Packets ~ Narada Brokering Event Every legacy A/V client needs one corresponding

RTPLink to be set up at a broker within broker network.

Unicast RTPLink: Integer Topic Numbers for RTP and RTCP

Multicast RTPLink:

A reflector between NaradaBrokers and multicast groups, encapsulating raw RTP packets from a multicast IP address to RTP events, publishing these events to NaradaBrokers, and forwarding the data it receives from broker network on the same IP address

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 1313

Audio Mixer, Video Mixer, Image Grabber Audio Mixing

The audio mixer creates a mixed audio stream from all the audio streams in the session

Video Mixing Video mixing makes the unicast users watch the pictures of

multiple participants in a meeting through one video stream Video Thumbnail visualize the VS set in the session, embedded into the control panel

of each endpoint, which Image grabbers capture video streams and save them as static JPEG

files.

All the media processing components can be distributed among the pool of the media servers connected to NaradaBrokering infrastructures.

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 1414

H.323, SIP Gateway Servers, A/V Session Server H.323 and SIP gateway transform these protocol

specific messages into XGSP signaling messages so that H.323 and SIP A/V endpoints could communicate with the XGSP A/V session server

The session server implements session management logics • creating/destroying A/V sessions

• allowing endpoints to join/leave session

• Allowing users to make audio/video selection, managing A/V application components

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 1515

Implementation Java Media Framework (JMF) : Openh323 is basis of H323 Gateway NIST SIP stack is basis of SIP Gateway NaradaBrokering is open source messaging from

Indiana

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 1616

Demo Pics ( I )

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 1717

Demo Pics ( II )

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="gcf@indiana.edu"uri="http://www.globalmmcs.org" email="gcf@indiana.edu" 1818

Demo Pics ( III )

04/08/2304/08/23 uri="http://www.globalmmcs.org" email="uri="http://www.globalmmcs.org" email="gcf@indiana.edu"gcf@indiana.edu"

1919

Thank you!

Please watch our demo at 5:30 PM !