Conference a Pi Workflow

7/27/2019 Conference a Pi Workflow

1/23

ConferenceAPI Workflow


2/23

Object Model

Conference API

Conference

Venue

Participant

Application

Rtp API (High Level)

RtpSender

VideoCapabilitySender VideoCapabilityViewer

DirectShow API

Managed DShow

RtpListener

RtpSession

RtpStream

FilterGraphManager

Video Capture Compress Network Render

FilterGraphManager

Network Source Decompress Video Render

Venue

Service


3/23

Start Bar UI

Conference API

Conference

Venue

Participant

Application


RtpSender


DirectShow API

Managed DShow

RtpListener

RtpSession

RtpStream

FilterGraphManager


FilterGraphManager


Venue

Service

Display CapabilitySenders

In MenusDisplay Venues


4/23

Start BarUI

FMain_Load()

Attach local objects and event handlers to the Conference object

Conference.ctor

Conference.Install()

Check to see if ConferenceAPI and its depdendencies are properly installed

Perform installation if not

This allows ConferenceAPI to be used in XCopy Deployment even though

ConferenceAPI has COM dependencies that must be registered and statethat must be initialized on the client only once like Performance Counters and

Event Logs

Load A/V Devices & other CapabilitySenders into the Options menu

FMain.DisplayVenues()

Conference.Venues.get

Conference.LocalParticipant.getRtpSession.ctor

Set local RtpSession properties, like CName and Name, for this Participant

MSR.LST.ConferenceXP.Identity.Identifier()

Gets email address from Messenger or creates one from Windows credentials

VenueService.GetVenues( Identity.Identifier )

Makes a Web Service call to the Venue Service set in the app.config file to load a

list of venues from a central database

foreach (Venue in Conference.Venues)Add Venue to Listbox


5/23

Join Venue, RtpListener.ctor

Conference API

Conference

Venue

Participant

Application


RtpSender


DirectShow API

Managed DShow

RtpListener

RtpSession

RtpStream

FilterGraphManager


FilterGraphManager


Venue

Service

Venue.Join()

Network

Start sending SDES RtcpPackets

announcing new Participant in Venue


6/23

Join Venue, RtpListener.ctorClick on .NET & Grids Venue, Clear Listbox & adjust UI (buttons, menus)

FMain.listView_ItemActivate

Venue.Join()

RtpListener.ctor()

Randomly choose a RtpSession SSRC (AKA heartbeatSSRC)

rtpNetworkListener == NetworkListener.ctor()

Start listening for incoming Rtp Packets

rtcpNetworkListener == NetworkListener,ctor()

Start listening for incoming Rtcp Packets

rtcpNetworkSender == NetworkSender.ctor()

Get ready to send outgoing Rtcp Packets telling other clients youre here

CheckFirewall()

Check the Firewall to ensure the proper ports are open

InitializePerformanceCounters()

InitializeThreads()

Continued on next slide


7/23

Join Venue, InitializeThreads

FMain.listView_ItemActivate

Venue.Join()

RtpListener.ctor()

Continued from previous slide

InitializeThreads()

RtpListeningThread()

Receives incoming Rtp packets from rtpNetworkListener

Creates new RtpStream objects as neededCalls RtpStream.newPacket to forward RtpPackets onto

the RtpStreams RingBuffer

RtcpListeningThread()

Receives incoming Rtcp packets from rtcpNetworkListener

SDES (Stream Description, describing Rtp traffic)

BYE (Stream Leaving, when Rtp traffic is stopping)

Others (SR - Sender Report, RR - Receiver Report, APP)

Creates new RtpSession objects as needed from SDES

Associates RtpStream objects to RtpSession objects from SDESRemove RtpStream and RtpSession objects if BYE

MaintenanceThread()

Send Rtcp SDES packets for this RtpSession on rtcpNetworkSender

Send Rtcp RR packets for this RtpSession on rtcpNetworkSender

Remove RtpStream or RtpSession objects if timed out

Rtp Stream Local

1.1 1.2 1.3 2.1 2.2 2.3 3.1 3.2

Rtp Stream Remote

7.1 7.2 7.3 8.1 8.2 8.3 9.1 9.2

Rtcp Session Remote

SD RR

Rtcp Session Local

SD RR

Rtcp Session Local

SD RR


8/23

Join Venue, ParticipantAdded

Conference API

Conference

Venue

Participant

Application


RtpSender


DirectShow API

Managed DShow

RtpListener

RtpSession

RtpStream

FilterGraphManager


FilterGraphManager


Venue

Service

Network

SDES RtcpPacket comes in from a RtpListener somewhere on the network

RtpSession.ctor()

RtpListener.RtpSessionAdded()

On RtpSessionAdded ()

Conference.ParticpantAdded ()

Participant Displayed in UI


9/23

Join Venue, ParticipantAdded

RtpListener.RtcpListeningThread(){

Receive Rtcp Packet

If ( RtcpPacket == SDES )

If ( ! RtpSessions.ContainsKey ( SDES.SSRC ) )

{

RtpListener.RtpSessionAdded

Conference.OnRtpSessionAdded{

Participant = new Participant( VenueService.GetParticipant ( SDES.CNAME ) )

Conference.ParticipantAdded ( Participant )

FMain.OnParticipantAdded ( Participant )

Add Participant to Form

}

}}

Rtcp Session Local

SD


10/23

Send Video

Conference API

Conference

Venue

Participant

Application


RtpSender


DirectShow API

Managed DShow

RtpListener

RtpSession

RtpStream

FilterGraphManager


FilterGraphManager


Venue

Serv

ice

Conference.AutoSend == true

After Venue.Join(), Conference calls

DefaultVideoCapabilitySender.Play()

FilterGraphManager.ctor()

Add Capture, Compress, and Network Render filters

NetworkRenderer.ctor() creates an instance

of RtpSender

Network

RtpSender.ctor() creates a MaintenanceThread

That starts sending SDES RtcpPackets announcing

the RtpStream and the RtpSession its related to

Video Capture Filters Capture

Thread calls RtpSender.Send ( )

15-30 times a second


11/23

Send Video

if (Conference.AutoPlay == true ), this is done automatically after entering Venue

Conference.DefaultVideoDevice.Play

VideoCapabilitySender.Play

MDShow.Device.Send

DShow.FilterGraphManager.ctor()

Add Capture, Compress, NetRenderer filters

Capture filter brings own thread

NetRenderer creates RtpSender which starts its own Maintenance thread

DShow.FGM.Play

Capture thread obtains video frame from device and

then calls NetRenderer.DoRenderSample which calls RtpSender.Send

RtpSender.MaintenenceThread

Get RtpSession information that was set as a static for the process and is shared

between RtpListener and all RtpSenders. This contains the CNAME that links theRtpListeners RtpSession to the RtpSenders RtpStreams.

Send SDES in the background which link the RtpStream created by RtpSender to

the RtpListeners RtpSession


12/23

VenueCapabilityViewer Added

Conference API

Conference

Venue

Participant

Application


RtpSender


DirectShow API

Managed DShow

RtpListener

RtpSession

RtpStream

FilterGraphManager


FilterGraphManager


Venue

Serv

ice

Network

SDES RtcpPacket comes in from a

RtpSender somewhere over the network RtpListener creates a new RtpStrea

And associates it with an RtpSessio

RtpListener.RtpStreamAdded ( )

Conference.OnRtpStreamAdded ( )

Conference.CapabilityViewerAdded ( )

CapabilityViewer icon is rendered over the Participants icon

Conference.RtpStreamToCapabilityViewer ( )

CapabilityViewer is associated to a Participant


13/23

VideoCapabilityViewer AddedRtpListener.RtcpListenerThread

Receives SDES from RtpSenderRtpListener.RtpStreamAdded

Associates RtpStream to RtpSession

Conference.RtpStreamToCapabilityViewer

Conference.CapabilityViewerAdded

FMain.RefreshImages

RtpListener.RtpListenerThreadForwards packets to RtpStream

Packets dropped because RtpStream.Subscribed == false


14/23

Play Video

Conference API

Conference

Venue

Participant

Application


RtpSender


DirectShow API

Managed DShow

RtpListener

RtpSession

RtpStream

FilterGraphManager


FilterGraphManager


Venue

Serv

ice

Conference.AutoPlay == true

Conference calls

CapabilityViewer.Play()

after CapabilityViewerAdded ()VideoCapabilityViewer.Play ()

calls MDShow.Play ()

which calls FilterGraphManager.ctor ()

then NetworkSource.ctor ()

NetworkSource.ctor() creates a background thread to

Process video, which calls RtpStream.Subscribe ()

to start RtpStream storing RtpPackets ina RingBuffer for later retrieval

NetworkSources DShow threNetworkSources thread calls

RtpStream.NextFrame in a loop and

passes the data to the other filters for

decompressing and rendering


15/23

Play Video

CapabilityViewer.Play

MDShowManager.Play

Fgm

NetSource

rtpStream.Subscribe

rtpStream.NextFrame

rtpPackets-> NetSource -> Decomp -> Render


16/23

Leave Venue

Conference API

Conference

Venue

Participant

Application


RtpSender


DirectShow API

Managed DShow

RtpListener

RtpSession

RtpStream

FilterGraphManager


FilterGraphManager


Venue

Serv

ice

User selects Leave Venue

Network

CapbilitySenders & CapabilityViewers

are stopped

Objects are Dispose () ed

BYE RtcpPackets are sent for all

originating RtpStreams & RtpSessions


17/23

Leave Venue

Rtcp BYE for RtpStream & RtpSession


18/23

Extras


19/23

RtpSender Details

RtpPacket

NetworkSenderSystem.Net.Sockets

RtcpPacket

RtpSender

FECDiags

new RtpSender ( ipEndPoint, name, role, mediaType )

rtpSender.Send( byte[] )


20/23

RtpStream

RingBuffer

RtpFrame

RtpSession

RtpListener

RtpStream

RingBuffer

RtpFrame

RtpSession

RtpListener

RtpPacket

NetworkListener

System.Net.Sockets

RtcpPacket

FECDiags

new RtpListener ( ipEndPoint)

event RtpListener.RtpSessionAddedevent RtpListener.RtpStreamAdded

byte[] = RtpStream.NextFrame()

Receive theProfessors video

streamProfessor hasjoined the

classroom

Received a videoimage

I join theclassroom


21/23

Multicast RTP

What is RTP? Peer to peer network transport

Handles stream & message data

Where latency is more important than guaranteed delivery

In use: Windows XP RTC, Java Media Framework, Real Networks

Why write an RTP stack? Multicast optimized threading & buffering

Forward Error Correction instead of buffering (low latency)

How?

Managed code w/ .NET object model DirectShow filters for easy conferencing development


22/23

Unicast Optimized

13 2 123 123

RTP Stream 1

1

1

1

2

2

RTP

Listene

r2

1

1

1

2

2

RTP

Listene

r3

1

1

1

2

2

RTP

Listener

1

1 2 3

RTP Stream 2

1 2 3

RTP Stream 3

1 2 3

RTP Packets


23/23

Multicast Optimized

13 2 123 123

RTP Stream 1

1

1

1

2R

TP

Listener

1 2 3

RTP Stream 2

1 2 3

RTP Stream 3

1 2 3

RTP Packets

Ring Buffer 1 Ring Buffer 2 Ring Buffer 3

Documents

Conference a Pi Workflow