Embed Size (px)
Citation preview
Media Services in Media Services in CINEMACINEMA
Presented by Kundan SinghJoint work with Henning Schulzrinne, Jonathan Lennox, Xiaotao Wu, Wenyu Jiang, Sankaran
Narayanan, Ajay Nambi and Ali Khwaja
Computer Science Department, Columbia University
April 25, 2003
2
PurposePurpose What are media services? What is CINEMA? What services are provided by CINEMA?
Conferencing Streaming media Unified messaging Interactive voice response
How can we use CINEMA?
The discussion on services and specific CINEMA implementation components are intermixed.
N
NNot relevant or not implementedOn-going workYes, Implemented
OY
3
IP telephony services (PSTN)IP telephony services (PSTN) Call routing services:
pre-call, one party speed dial call forwarding “follow me” call filtering/blocking
(in/out) do not disturb distinctive ringing
Call handling features
autoanswer
Multi-party features call waiting call transfer (blind,
consultative) conference call call park call pickup music on hold call monitoring
O
IP-telephony can’t win by just providing what PSTN has
4
IP telephony services IP telephony services (Internet)(Internet)
Presence-enabled calls place call only if callee is available
Presence-enabled conferencing call conference participants when all are
online and not busy IM conference alerts
receive IM when someone joins a conference
Unified messaging receive email, IM alert for new voicemails
O
5
Where the services reside?Where the services reside?
Common gateway interface (CGI) Call processing language (CPL) SIP servlet
Make call when boss is online …
Enter your authentication PIN for billing…
B2BUA
Double ringing sound when boss calls…
Endpoint
Forward to office phone during day, and home phone during evening…
Proxy/registrar
Endpoint
Language for End System Services (LESS)
Use finger for locating user…
O
6
ProgramCall
routing
““Bricks” in media Bricks” in media applicationsapplications
SIP SAP RSVP RTCP
RTP
MediaG.711MPEG
RTSP
SignalingQuality of service
Media transport
InternetTelephony
InternetRadio/TV Messaging
and Presence
Interactivevoice response
Unified messagingVideo
conferencing
Physical layer
Link layer
Network (IPv4, IPv6)
Transport (TCP, UDP)
Application layer
VoiceXML
DTMF MixingSpeech/
text
N
SDP
7
…
Audio Packet TransferAudio Packet Transfer
Digitization (e.g., sampling at 8kHz, 16 bits per sample, i.e, 128 kb/s or 320 bytes per 20 ms)
Real-time compression/encoding (e.g., G.729A at 8 kb/s) Transport to remote IP address and port number over
UDP (Why not TCP?) Processing on receiver side is the reverse
N
8
Unreliable UDP Unreliable UDP a) Packet lossb) Out-of-order (very rarely)c) Jitter (delay variation)
1 2 3 5 7 6
1 2 3 4 5 6 7
timeline
Sender
Receiver
(a)(b)
Problems with UDPProblems with UDP
N
9
1 2 3 5 7 6
1 2 3 4 5 6 7
Sender
Receiver
8 9 0 2 3 4
8 9 0 1 2 3 4
1 2 7
1 2 5 7 8 9 0 2 3
1 2 3 5 6 7 8 9 0 2
Receive BufferReceive Buffer Receive buffer: to absorb jitter Tradeoff in buffer size Adaptive delay adjustment Sequence number: to detect packet loss; Just ignore the loss!
Y
10
Encoded Audio
RTP Header
UDP header
IP header
msg
sendto(…, msg, …)recvfrom(…, msg, …)
Sequence number
Optional contributors’ list (CSrc)
Source identifier (SSrc)
Timestamp (proportional to sampling time)
Payload typeCC MV P X
Real-time Transport ProtocolReal-time Transport Protocol
Y
RTP: media transportRTCP: QoS feedback
11
Streaming mediaStreaming media
Messages RTSP vs HTTP
UDP, Stateful, Control, Timed Applications
Recording, Playback, Live media, Conference
Web browser
Mediaplayer
file
Web serverHTTP Web
browser
Mediaplayer
metafile
HTTP
RTSP
RTP
Media server
Y
12
Columbia Columbia rtspdrtspd
Existing Playback/recording of
-law (Sun .au files) and raw
Unicast address Multi-threaded; Portable (Unix/win32) Works with Apple
QuickTime client UDP + TCP IPv4 + IPv6
Planned Multicast MPEG video Mp3, wav audio Media over TCP Live media
Y
13
Mixer Transcoder-law
-law
G.729
G.729
-law
-law
RTP-based ConferenceRTP-based Conference
N
14
• What is the IP address of Alice’s host?• What audio/video encoding can it support?
bob’s hostAlice’s host128.59.19.194
REGISTER asalice=>128.59.19.194Call alice
128.59.19.194
Why do we need signaling?Why do we need signaling?
Session Initiation Protocol Email-like address: [email protected] DNS Proxy, redirect
Y
home.com
15
AliceBob
INVITE [email protected] can support -law and G.729Send me audio at 202.16.49.27:6780
OK; I can support -lawSend me audio at 128.59.19.194:8000
202.16.49.27 128.59.19.194
To port 6780
To port 8000RTP
RTP
Session descriptionSession description
Y
16
Columbia Columbia sipcsipc
Y
17
Emergency serviceEmergency service
Calling E911, location
Assistance Audio/video, remote-help
Notification Earthquake, fire
O
18
Presence/event notificationPresence/event notification
PA
registrar
Presence server
office.com
SUBSCRIBE
NOTIFY
PUA
PUA
PUA + PA
Y
19
Columbia Columbia IM and presenceIM and presence
Y
20
Endpoint call controlEndpoint call control Language for End System Services (LESS) for endpoint service
creation Direct user interaction, direct media control Handle converged information, e.g., call, presence, email Example: when buddy is online, make a call
<less name="online_call" require="generic presence ui"> <notification status="online" priority="0.5"> <address-switch field="origin"> <address is=“[email protected]"> <call /> <alert sound=“ring.au"
text="Calling …" /> </address> </address-switch> </notification></less>
O
21
Network call controlNetwork call control
SIP-CGI CPL SIP servlets
Priority.pl
SIP_FROMSIP_TOstdin
CGI-PROXY-REQUESTstdout
SIP proxy
Urgent
Low-priority
Voicemail
Phone
Y
if (defined $ENV{SIP_FROM} &&
$ENV{SIP_FROM} =~ /sip:[email protected]/)
{
foreach $reg (get_regs())
{
print "CGI-PROXY-REQUEST $reg SIP/2.0\n";
print "Priority: urgent\n\n";
}
}
22
Call transferCall transfer
REFER Blind/
consultation/ attended
active call
REFER CReferred-By: B
INVITE CReferred-By: B
BYE A
A B
C
active call
O
23
B2BUA and third-party call B2BUA and third-party call controlcontrol
Back-to-back UA Incoming call
triggers outgoing call
Services Calling card Anonymizer
INVITE
AB
CSIP
SIP
OK (SDP1)
ACKINVITE (SDP1)
OK (SDP2)
ACKINVITE (SDP2)
OK
ACK
O
24
VoicemailVoicemail
Problems in PSTN
Design alternatives
Issues
Redirect after 10s
vmail.pl
SIP_FROMSIP_TOstdin
CGI-PROXY-REQUESTstdout
If no response accept after 15s
Endpoint redirects
Voicemail acts like a phoneProxy controls
Y
25
Columbia Columbia sipumsipum
Goals Universal access Scalability Provider
independent Why SIP and
RTSP? Reuse existing
infrastructure and tools
(1) INVITE
INVITE
INVITEOK
CANCEL
(3) OK
(2) SETUP
(4) RTP
(5) BYE
rtspd
sipum
Y
26
Columbia Columbia sipumsipum Retrieval
Web interface rtsp://server/alice
/inbox/1677.au, sip:alice-1677-
retrieve@server press 1 to listen…
Configuration Folders Options Email
Y
27
VoiceXMLVoiceXML
Telephone
PSTNPSTN
Voice gateway
Web server
• Service logic (CGI, servlet, JSP)
• Voice and telephony functions• VoiceXML browser
Internet userVXMLVXML HTMLHTML
Internet
Internet
IVR platform• Voice and telephony functions (ASR, TTS, DTMF)• Service logic (application specific)
sipvxmlGateway
Y
28
VoiceXML contd.VoiceXML contd.
<form action=“url”> Enter your Id: <input name=‘id’> <input type=‘submit’> </form>
<form> <field name=‘id’> <prompt> Your ID, please. </prompt> </field> <block> <submit next=“url”/> </block></form>
Telephony, speech synthesis or audio output, user input and grammar, program flow, variable and properties, error handling, …
Y
29
DTMFDTMF
Telephone SIP/PSTNgateway
PSTNPSTN Internet
Internet
Internet user
D
D
Audio
SIP INFO
RTP/ RFC2833
Y
30
Columbia Columbia sipvxmlsipvxml
Unified messaging access
Email by phone Event notification
and scheduling Audio volume control
for conference Advanced
conference control
Telephone SIP/PSTNgateway
PSTNPSTN
sipvxmlsipvxml SIP phone
Web server
Internet
Internet
Email.tcl
Media serverrtspd
TTS, ASR, DTMF, XML, HTTP, RTSP
Y
31
Email + phoneEmail + phone
Login Email formatting Listen, reply,
delete, compose, forward
Navigation -next, previous, jump
Email formatting
SIP based Text-to-speech
VoiceXMLbrowser
Emailservlet
JSP
DB
procmailEmail
Email to IM
IM
Call
Internet
Internet
TTS
important mails
SIP
SIP
Internet
Internet
SIP
HTTP
Email by phone
Email to phone
Y
Inbox
Inbox
32
IM + voice callIM + voice call
Who can initiate? authentication,
billing, … Feedback
to voice user to IM user initial IM greeting
Talk-spurt detection Speech recognition
IM
Call
TTSASR
SIPMESSAGE
SIPINVITERTP
Y
33
Unified messagingUnified messaging
The gaps among different media (audio, video, text), devices (PC, phone) and means of communications (Email, SIP, IM) disappear for messaging
O
34
NotificationNotification Calendar Events Conference
s
Web server
Calendar.cgi “at 6:00pm”
Schedule from a browser
SIP call2129397063
IMkundan@
EmailKns10@…
O
35
Phone announcement serverPhone announcement server
Input Range:93970?? List: A, B, C
Example Announcement Emergency
Issues Voicemail Failure
detection
PASTTS
Destinations
SIP
Text or audio
. . .
Y
36
RTSP + TTS + ASRRTSP + TTS + ASR
Media server
TTSASR
SETUP rtsp://server/tts.cgi?text=How+are+you.
PLAY
RTP
SETUP rtsp://server/asr.cgi
RECORD
RTP
SET_PARAMETERText=I am fine, thank you.
O
37
Audio conference models Audio conference models (non-multicast)(non-multicast)
A
B C
D
B
A+C+D
A
B C
AB+C+D
A+B+D
C
D
D
A+B+C
A
B C
D
Topology star full-mesh ad-hoc
Advantages Heterogeneoussimple clients
No central point offailure
DisadvantagesExternal server with high bandwidth link
Complex endpointsComplex signaling
Typically only three party conferences
N
38
Centralized conferencingCentralized conferencing Conference as URL
[email protected] On the fly conferences
my.adhoc@server Basic task: join/leave
Dial in, Refer dial in Dial out, Refer dial out
REFERINVITE
REFER
INVITEINVITE
server
O
39
Columbia Columbia sipconfsipconf
G.711, GSM, DVI, G.722 mixing (decode-mix-encode) Video replication; IM; text; VNC screen sharing
D
D
D
E
E
E
A
B
C
G711
DVI
GSM
Linear
Linear
Linear
G711
DVI
G711
Playout delay Periodictimer
M=A+B+C
Mixed linear
M - A=B+C
M - B
M - C
Receive thread Send thread
Y
40
Columbia Columbia sipconfsipconf
Y
41
Conference controlConference control Access control Floor control Notification State
Participant: active, departed, booted, failed dialog state, floor state Media
sending: received-by-all, muted receiving: receiving-all, anchor-only
<conference-info version="0" state="full" entity=“[email protected]"> <user uri="sip:[email protected]" display-name=“Alice"> <status>active</status> <media-status> <media-stream media-type="audio"/> </media-status> </user> <user uri="sip:[email protected]" display-name=“Bob Wilson"> <status>departed</status> </user></conference-info>
O
42
Conference + VoiceXMLConference + VoiceXML
sipvxmlsipvxml
CallerCaller
sipconfsipconf
1. INVITE sipvxml2. Call accepted3. Enter your four digit PIN4. Entered 4-6-8-3
5. Authenticate user, 4683=>Alice6. Enter the conference identifier7. Entered 2-3-#
8. Permission to join, 23=>meet9. REFER [email protected] the old call
11.INVITE meet@conference
Call transfer vs bridged mode
Y
43
New conference applicationsNew conference applications
sipvxmlsipvxml
CallerCaller
1. INVITE sipvxml2. Menu 1. Vol Check 2. Mic Check3. User enters 2
4. User speaks out a voice sample5. Voice sample is analyzed
7. User adjusts the vol level.
The ease & flexibility of sipvxml enables us to build custom telephonic applications to suit our needs.
e.g., Volume Check Application
sipconfsipconf 8. User now joins conference.
6. SipVXML: Vol level too high/low/…
Y
44
ConferencingConferencing
Automatic volume adjustments
Automatic load balancing
Delay adjustments Conference
recording Local or RTSP
Y
sipcsipcSIP323SIP323
SIP/PSTNSIP/PSTNRecording in a media server
45
Ncast MPEG distributionNcast MPEG distribution
sip.cgi?sip:[email protected]
INVITE sip:[email protected]
Y
46
My owner’s SIP address is sip:[email protected]
Help!!! (invoke sipc to callsip:[email protected])
INVITE sip:[email protected]
Y
47
Summary of media servicesSummary of media services
Streaming media Playback, recording, live-media
Unified messaging Voicemail, video/IM integration, ...
Conferencing Mixing, load-balancing
Interactive voice response Tele-banking, VoiceXML
Y
48
CINEMACINEMAColumbia InterNet Extensible Multimedia Architecture Columbia InterNet Extensible Multimedia Architecture
CINEMA
Distributed component architecture
Libraries and APIs
Managementand control
Applicationand proxyservers
User-agent(client)applications
Y
49
Columbia SIP user agent: Columbia SIP user agent: sipcsipc
Media Audio, video, text, white board Screen sharing Shared web browsing
Advanced Presence, instant messaging Programmable call handling Conference control (on-going) Emergency notification and handling (on-
going) Device control Language for End System Services (on-going)
Y
50
Columbia SIP servers Columbia SIP servers (CINEMA)(CINEMA)
InternalTelephoneExtn: 7040
SIP/PSTN Gateway
Department PBX
Web based configuration
Web server
Telephoneswitch
SQLdatabase
sipd:Proxy, redirect, Registrar server
NetMeeting
H.323
rtspd: media server
sipum: Unified messaging
Quicktime
RTSP clients
RTSP
713x
Single machine
SNMP(Network Management)
sipconf: Conference server
siph323: SIP-H.323 translator
Local/long distance1-212-5551212
PSTN
Video encoder
Y
51
Libraries and Libraries and APIsAPIs
sipd sip323 sipconf sipum sipvxmlrtspd
CINEMA Libraries
libNT
Win32 stub
libcine
Utilities parsingIPv6
libsip
Basic SIP library
libsip++
SIP UA library
libmixer
RTP audio mixer
libdict
Hash table
libdb++
mySQL intf
RTSP mediaserver
SIP proxy server
SIP/H.323gateway
SIP/RTP conferencing
SIP/RTSP unified messaging
SIP/VoiceXMLbrowser
Xerces-COpenH323
MySQLPWLibResparse
librtsp
RTSPclient
librtp
RTP library
libsnmp
SIP MIB
ViaVoice/FliteXerces-C
Server Applications
Parsing, SIP, SDP, RTP, mySQL interface, SNMP interface, Portability stubs, etc.
CPL, CGI,Servlet
Y
52
Status of CINEMA servicesStatus of CINEMA servicesFeatures/services Status
SIP proxy, redirect, registration server Product
SIP Presence agent; programmable call routing Beta
Dial-in conference with audio, video, screen sharing, instant messaging, recording, load balancing, volume or delay adjustment, and IVR
Beta
Dial-out or refer dial-in conferencing, conference control On-going
Voice mail with retrieval, deletion and IVR Beta
Media streaming server for basic audio record and play Beta
SIP/VXML browser for VoiceXML 1.0 with basic functionality Alpha
Call transfer, back-to-back user agent On-going
Phone announcement server, IM/call converter, email-by-phone and email-to-phone, SIP anonymizer
Demo
Calendar event notification to IM, phone and email On-going
Web-based configuration and control Alpha
Media server with text-to-speech, speech-recognition, video, live media
On-going
53
What’s next?What’s next?from multimedia communications to from multimedia communications to collaborationcollaboration
Synchronous communications Conferencing, IM
Asynchronous communications Voicemails, message board, file sharing
Ubiquitous computing i-button, ID-card
Service creation User friendly, end-point/network
[email protected]://www.cs.columbia.edu/~kns10/http://www.cs.columbia.edu/IRT/cinema
Media Services in Media Services in CINEMACINEMA
BACKUP SLIDES
55
DO sip:[email protected] SIP/2.0…..<Control><Action>turn lamp on</Action></Control>
serial port
lamp
X10 device
SIP beyond telephonySIP beyond telephony
SUBSCRIBE [email protected] NOTIFY [email protected]
DO [email protected] INVITE [email protected]
56
PSTN interworkingPSTN interworking
Translating audio (PCMU/PCMA) Translating signaling (PRI/T1,ISUP)
Overlap signaling Advanced features in SIP are lost in PSTN
Translating identifiers (phone number) Determining transition points
Telephonenetwork
SIP/PSTN gateway
SIP server IP endpointTelephonesubscriber
+1 212 9397063 sip:[email protected]
57
PSTN to IPPSTN to IP
Gateway knows the SIP server <sip:[email protected];user=phone>
ENUM DNS +1 212 9397042 => 2.4.0.7.9.3.9.2.1.2.1.e164.arpa => sip:[email protected]
Suitable for relatively “static” contacts
58
IP to PSTNIP to PSTN Static mapping
1-212-939xxxx => @itgw1.cs.columbia.edu ITGW information is dynamic:
Overlapping networks Multiple providers Load balancing
TRIP Route advertisement Can be implemented in outbound proxy Suitable for current hierarchical network+1 @service.mci.com at 4¢/min+1212 @nyc.gw.com at 1¢/min+1212939 @itgw1.columbia.edu free
59
SIP-CGISIP-CGI
Programming language independent Maintains state via an opaque token For SIP proxies and endpoints:
call routing controlling forking call rejection call modification (Priority, Call-Info)
RFC 3050 Upload via web or REGISTER
60
SIP CGI contd.SIP CGI contd. Block *@anonymous.com:if (defined $ENV{SIP_FROM} && $ENV{SIP_FROM} =~ "sip:*@anonymous.com") { print "SIP/2.0 600 I can't talk right now\n\n";}
Make calls from boss as urgent:if (defined $ENV{SIP_FROM} && $ENV{SIP_FROM} =~ /sip:[email protected]/) { foreach $reg (get_regs()) { print "CGI-PROXY-REQUEST $reg SIP/2.0\n"; print "Priority: urgent\n\n"; }}
61
CPLCPL XML-based; interpreted Control
Restricted to branching and subroutines
no variables, no loops structured tree describing
actions performed on call setup event
Event incoming and outgoing
events Switch
address, language, time, priority
Action Proxy, redirect, reject
(busy) Graphical tools
Protocol-independent SIP, H.323
Extensions CPL for presence Logging Email
Callnot match
match
actions
62
CPL contd.CPL contd.<?xml version="1.0" ?><!DOCTYPE cpl PUBLIC "-//IETF//DTD RFC2824 CPL
1.0//EN" "cpl.dtd"> <cpl> <incoming> <time-switch> <time dtstart="20020830T140000Z" dtend="20020830T150000Z"> <reject status="reject" reason=“Busy" /> </time> </time-switch> </incoming></cpl>
63
SIP servletSIP servlet Java-based Runs in SIP server Receives SIP objects and processes them
public class RejectServlet extends SipServletAdapter { public boolean doInvite(SipRequest req) { SipResponse res = req.createResponse(); res.setStatus(603); res.send(); return true; }}
