TADS Developer Summit WebRTC Dan Burnett

Introduc)on to WebRTC

Dan Burne4 Chief Scien)st, Tropo Director of Standards, Voxeo

Alan Johnston Dis)nguished Engineer Avaya

WebRTC

WebRTC Tutorial Topics

•  What is WebRTC? •  How to Use WebRTC •  WebRTC Peer-‐to-‐Peer Media •  WebRTC Protocols and IETF Standards •  WebRTC W3C API Overview •  Pseudo Code Walkthrough •  PracFcal bits

TAD Summit Bangkok 2013 2

What is WebRTC?

WebRTC is “Voice & Video in the browser”

•  Access to camera and microphone without a plugin – No proprietary plugin required!

•  Audio/video direct from browser to browser •  Why does it maWer? – Media can stay local – Mobile devices eventually dropping voice channel anyway

– Games


The Browser RTC FuncFon •  WebRTC adds new Real-‐

Time CommunicaFon (RTC) FuncFon built-‐in to browsers –  No download –  No Flash or other plugins

•  Contains –  Audio and video codecs –  Ability to negoFate peer-‐to-‐

peer connecFons –  Echo cancellaFon, packet loss

concealement •  In Chrome & Firefox today,

Internet Explorer someFme and Safari eventually

HTTP or WebSockets

On-‐the-‐wire protocols

RTC APIs Other APIs

NaFve OS Services

Web Server

JavaScript/HTML/CSS

Web Browser

Browser RTC

FuncFon

(Signaling)

(Media or Data)

HTTP or WebSockets

Signaling Server


Benefits of WebRTC

For Developer •  Streamlined development –

one placorm •  Simple APIs – detailed

knowledge of RTC protocols not needed

•  NAT traversal only uses expensive relays when no other choice

•  Advanced voice and video codecs without licensing

For User •  No download or install –

easy to use •  All communicaton

encrypted – private •  Reliable session

establishment –  “just works”

•  Excellent voice and video quality

•  Many more choices for real-‐Fme communicaFon


WebRTC Support of MulFple Media

•  MulFple sources of audio and video are assumed and supported

•  All media, voice and video, and feedback messages are mulFplexed over the same transport address

Stereo Audio

Browser M on Mobile

Browser L on Laptop

Front Camera Video

Rear Camera Video

Microphone Audio

ApplicaFon Sharing Video

WebCam Video


WebRTC Triangle

•  Both browsers running the same web applicaFon from web server

•  Peer ConnecFon established between them with the help of the web server

Web Server (ApplicaFon)

Browser M (Running HTML5 ApplicaFon

from Web Server)

Browser L (Running HTML5 ApplicaFon

from Web Server)

Peer ConnecFon (Audio, Video, and/or Data)


WebRTC Trapezoid

•  Similar to SIP Trapezoid •  Web Servers communicate using SIP or Jingle or proprietary •  Could become important in the future.

Web Server A (ApplicaFon A)

Peer ConnecFon (Audio and/or Video)

Web Server B (ApplicaFon B)

SIP or Jingle

Browser M (Running HTML5 ApplicaFon

from Web Server A)

Browser T (Running HTML5 ApplicaFon

from Web Server B)


WebRTC and SIP

•  SIP (Session IniFaFon Protocol) is a signaling protocol used by service providers and enterprises for real-‐Fme communcaFon

•  Peer ConnecFon appears as a standard RTP session, described by SDP •  SIP Endpoint must support RTCWEB media extensions

Web Server

SIP Client


SIP Server SIP

SIP

Browser M


WebRTC and Jingle Web Server

Jingle Client


XMPP Server Jingle

Jingle

Browser M

•  Jingle is a signaling extension to XMPP (Extensible Messaging and Presence Protocol, aka Jabber)

•  Peer ConnecFon SDP can be mapped to Jingle •  Jingle Endpoint must support RTCWEB Media extensions


WebRTC and PSTN

•  Peer ConnecFon terminates on a PSTN Gateway •  Audio Only •  EncrypFon ends at Gateway

Web Server

Browser M PSTN Gateway Peer ConnecFon (Audio)

Phone


WebRTC with SIP

•  Browser runs a SIP User Agent by running JavaScript from Web Server •  SRTP media connecFon uses WebRTC APIs •  Details in [dram-‐iec-‐sipcore-‐websocket] that defines SIP transport over

WebSockets

13

Web Server

SRTP Media

SIP Proxy/Registrar Server

Browser M (running JavaScript SIP UA)

Browser T (running JavaScript SIP UA)

HTTP (HTML5/CSS/JavaScript)

WebSocket (SIP)

WebSocket (SIP)

HTTP (HTML5/CSS/JavaScript)


WebRTC Signaling Approaches

•  Signaling is required for exchange of candidate transport addresses, codec informaFon, media keying informaFon

•  Many opFons – choice is up to web developer

14


How to Use WebRTC

WebRTC usage in brief

Set Up Peer ConnecFon

Obtain Local Media

Exchange Offer/Answer

AWach Media or Data

Get more media

All media added

Peer ConnecFon established

AWach more media or data

Ready for call



•  getUserMedia() –  Audio and/or video –  Constraints –  User permissions

•  Browser must ask before allowing a page to access microphone or camera

•  MediaStream •  MediaStreamTrack

–  CapabiliFes –  States (sepngs)


Obtain Local Media


AWach Media or Data

Get more media

All media added



Ready for call



•  RTCPeerConnection –  Direct media –  Between two peers –  ICE processing –  SDP processing –  DTMF support –  Data channels –  IdenFty verificaFon –  StaFsFcs reporFng


Obtain Local Media


AWach Media or Data

Get more media

All media added



Ready for call



•  addStream() –  Doesn't change media state!

•  removeStream() –  DiWo!

•  createDataChannel() –  Depends on transport


Obtain Local Media


AWach Media or Data

Get more media

All media added



Ready for call



•  createOffer(), createAnswer()

•  setLocalDescription(), setRemoteDescription()

•  Applying SDP answer makes the magic happen


Obtain Local Media

Exchange Session

DescripFons

AWach Media or Data

Get more media

All media added



Ready for call


WebRTC usage – a bit more detail


Obtain Local Media

Exchange Session

DescripFons

AWach Media or Data

Get more media


Set Up Signaling Channel


SDP offer/answer •  Session DescripFons – Session DescripFon Protocol created for use by SIP in sepng up voice (and video) calls

– Describes real-‐Fme media at low level of detail •  Which IP addresses and ports to use •  Which codecs to use

•  Offer/answer model (JSEP) – One side sends an SDP offer lisFng what it wants to send and what it can receive

– Other side replies with an SDP answer lisFng what it will receive and send


WebRTC Peer-‐to-‐Peer Media

Media Flows in WebRTC

Web Server

Internet

Router

Browser L

Home WiFi Router

Browser T

Browser M

Coffee Shop WiFi Router

Browser D


Media without WebRTC

Web Server

Internet

Router

Browser L

Home WiFi Router

Browser T

Browser M


Browser D


Peer-‐to-‐Peer Media with WebRTC

Web Server

Internet

Router

Browser L

Home WiFi Router

Browser T

Browser M


Browser D


Web Server

Internet

Router with NAT

Browser L

Home WiFi with NAT

Browser T

Browser M

Most browsers are behind NATs on the Internet, which complicates the establishment of peer-‐to-‐peer media sessions.

Coffee Shop WiFi with

NAT

Browser D

NAT Complicates Peer-‐to-‐Peer Media


What is a NAT? •  Network Address Translator (NAT) •  Used to map an inside address (usually a private IP address) to outside address (usually a public IP address) at Layer 3

•  Network Address and Port TranslaFon (NAPT) also changes the transport port number (Layer 4) – These are omen just called NATs as well

•  One reason for NAT is the IP address shortage


Internet

Home WiFi with NAT

Browser T 192.168.0.6

Browser M 192.168.0.5

“Outside” Public IP Address 203.0.113.4

“Inside” Private IP Addresses 192.168.x.x

NAT Example


NATs and ApplicaFons •  NATs are compaFble with client/server protocols such as web, email, etc.

•  However, NATs generally block peer-‐to-‐peer communicaFon

•  Typical NAT traversal for VoIP and video services today use a media relay whenever the client is behind a NAT – Omen done with an SBC – Session Border Controller

–  This is a major expense and complicaFon in exisFng VoIP and video systems

•  WebRTC has a built-‐in NAT traversal strategy: InteracFve ConnecFvity Establishment (ICE)


Web Server

Internet

Router with NAT

Browser L

Home WiFi with NAT

Browser T

Browser M

ICE connecFvity checks can omen establish a direct peer-‐to-‐peer session between browsers behind different NATs


NAT

Browser D

Peer-‐to-‐Peer Media Through NAT


ICE ConnecFvity Checks •  ConnecFvity through NAT can be achieved using ICE connecFvity checks

•  Browsers exchange a list of candidates –  Local: read from network interfaces –  Reflexive: obtained using a STUN Server –  Relayed: obtained from a TURN Server (media relay)

•  Browsers aWempt to send STUN packets to the candidate list received from other browser

•  Checks performed by both sides at same Fme •  If one STUN packet gets through, a response is sent and this connecFon used for communicaFon –  TURN relay will be last resort (lowest priority)


Web Server

Internet

Router with NAT

Browser L

Home WiFi with NAT

Browser T

Browser M

If both browsers are behind the same NAT, connecFvity checks can omen establish a connecFon that never leaves the NAT.


NAT

Browser D

P2P Media Can Stay Local to NAT


Web Server

Internet

Router with NAT

Browser L

Home WiFi with NAT

203.0.113.4

Browser T

Browser M 192.168.0.5


NAT

Browser D

TURN Server 198.51.100.2

ICE uses STUN and TURN servers in the public Internet to help with NAT traversal.

STUN Server 198.51.100.9

ICE Servers


Web Server

Internet

Router with NAT

Browser L

Home WiFi with NAT

203.0.113.4

Browser T

Browser M 192.168.0.5


NAT

Browser D

TURN Server 198.51.100.2

Browser sends STUN test packet to STUN server to learn its public IP address (address of the NAT).

STUN Server 198.51.100.9

Browser Queries STUN Server


Web Server

Internet

Router with NAT

Browser L

Home WiFi with NAT

Browser T

Browser M


NAT

Browser D

TURN Server as a Media Relay

In some cases, connecFvity checks fail, and a TURN Media Relay on the public Internet must be used.

STUN Server

TURN Server Can Relay Media


WebRTC Protocols and IETF Standards

WebRTC: A Joint Standards Effort •  Internet Engineering Task Force (IETF) and World Wide Web ConsorFum (W3C) are working together on WebRTC

•  IETF –  Protocols – “bits on wire” –  Main protocols are already RFCs, but many extensions in progress

–  RTCWEB (Real-‐Time CommunicaFons on the Web) Working Group is the main focus, but other WGs involved as well

–  hWp://www.iec.org •  W3C –  APIs – used by JavaScript code in HTML5 –  hWp://www.w3c.org


WebRTC Protocols

IP

UDP SCTP

WebSocket SRTP SDP

TURN

STUN

ICE

TCP TLS Transport Layer

Network Layer

ApplicaFon Layer

DTLS

HTTP

SIP is not shown as it is opFonal TAD Summit Bangkok 2013 39

IETF RTCWEB Documents Document) Title) Ref)

Overview' “Overview:'Real'Time'Protocols'for'Browser6based'Applications”'

draft6ietf6rtcweb6overview'

Use'Cases'and'Requirements' “Web'Real6Time'Communication'Use6cases'and'Requirements”'

draft6ietf6rtcweb6use6cases6and6requirements'

RTP'Usage'“Web'Real6Time'Communication'(WebRTC):'Media'Transport'and'

Use'of'RTP”'

draft6ietf6rtcweb6rtp6usage'

Security'Architecture' “RTCWEB'Security'Architecture”' draft6ietf6rtcweb6security6arch'

Threat'Model' “Security'Considerations'for'RTC6Web”'

draft6ietf6rtcweb6security'

Data'Channel' “RTCWeb'Data'Channels”' draft6ietf6rtcweb6data6channel'

JSEP' “JavaScript'Session'Establishment'Protocol”'

draft6ietf6rtcweb6jsep'

Audio' “WebRTC'Audio'Codec'and'Processing'Requirements”'

draft6ietf6rtcweb6audio'

Quality'of'Service' “DSCP'and'other'packet'markings'for'RTCWeb'QoS”'

draft6ietf6rtcweb6qos'

'TAD Summit Bangkok 2013 40

Codecs

•  Mandatory to Implement (MTI) audio codecs are seWled on Opus and G.711 (finally!)

•  Video is not yet decided!

RFC 6716 .


WebRTC W3C API Overview

Two primary API secFons

•  Handling local media – Media Capture and Streams (getUserMedia) specificaFon

•  Transmipng media – WebRTC (Peer ConnecFon) specificaFon


Local Media Handling

•  In this example –  Captured 4 local media streams –  Created 3 media streams from them –  Sent streams over Peer ConnecFon

Sources

Presenter Video

DemonstraFon Video

Audio

PresentaFon Video

PresentaFon Stream

Presenter Stream

DemonstraFon Stream

Captured MediaStreams

Created MediaStreams

Tracks

“Audio” Track

“PresentaFon” Track

“Audio” Track

“Presenter” Track

“Audio” Track

“DemonstraFon” Track

Front Camera Video Rear Camera Video

Microphone Audio ApplicaFon Sharing Video

Browser M



•  Sources –  Encoded together –  Can't manipulate individually

Sources

Presenter Video

DemonstraFon Video

Audio

PresentaFon Video

PresentaFon Stream

Presenter Stream

DemonstraFon Stream



Tracks

“Audio” Track


“Audio” Track


“Audio” Track




Browser M



•  Tracks (MediaStreamTrack) –  Tied to a source –  Exist primarily as part of Streams; single media type –  Globally unique ids; opFonally browser-‐labeled

Sources

Presenter Video

DemonstraFon Video

Audio

PresentaFon Video

PresentaFon Stream

Presenter Stream

DemonstraFon Stream



Tracks

“Audio” Track


“Audio” Track


“Audio” Track




Browser M



•  Captured MediaStream –  Returned from getUserMedia() –  Permission check required to obtain

Sources

Presenter Video

DemonstraFon Video

Audio

PresentaFon Video

PresentaFon Stream

Presenter Stream

DemonstraFon Stream



Tracks

“Audio” Track


“Audio” Track


“Audio” Track




Browser M



•  MediaStream –  All contained tracks are synchronized –  Can be created, transmiWed, etc.

Sources

Presenter Video

DemonstraFon Video

Audio

PresentaFon Video

PresentaFon Stream

Presenter Stream

DemonstraFon Stream



Tracks

“Audio” Track


“Audio” Track


“Audio” Track




Browser M



•  Sepngs – Current values of source properFes (height, width, etc.)

–  Exposed on MediaStreamTrack •  CapabiliFes – Allowed values for source properFes –  Exposed on MediaStreamTrack

•  Constraints – Requested ranges for track properFes – Used in getUserMedia(), applyConstraints()


Transmipng media

•  Signaling channel – Non-‐standard – Must exist to set up Peer ConnecFon

•  Peer ConnecFon – Links together two peers – Add/Remove Media Streams

•  addStream(), removeStream() – Handlers for ICE or media change – Data Channel support


Peer ConnecFon

•  "Links" together two peers – Via new RTCPeerConnection() – Generates Session DescripFon offers/answers

•  createOffer(), createAnswer() –  From SDP answers, iniFates media

•  setLocalDescription(), setRemoteDescription()

– Offers/answers MUST be relayed by applicaFon code!

–  ICE candidates can also be relayed and added by app •  addIceCandidate()


Peer ConnecFon

•  Handlers for signaling, ICE or media change – onsignalingstatechange – onicecandidate, oniceconnectionstatechange

– onaddstream, onremovestream – onnegotiationneeded

– A few others


Peer ConnecFon

•  “Extra” APIs – Data – DTMF – StaFsFcs –  IdenFty

•  Grouped separately in WebRTC spec – but really part of RTCPeerConnection definiFon

– all are mandatory to implement


Data Channel API •  RTCDataChannel createDataChannel() •  Configurable with

–  ordered –  maxRetransmits, maxRetransmitTime –  negotiated –  id

•  Provides RTCDataChannel with –  send() –  onopen, onerror, onclose, onmessage*


DTMF API

•  RTCDTMFSender createDTMFSender() –  Associates track input parameter with this RTCPeerConnection

•  RTCDTMFSender provides –  boolean canInsertDTMF() –  insertDTMF() –  ontonechange –  (other stuff)


StaFsFcs API

•  getStats() – Callback returns staFsFcs for given track

•  StaFsFcs available (local/remote) are: – Bytes/packets xmiWed – Bytes/packets received

•  May be useful for congesFon-‐based adjustments


IdenFty API

•  setIdentityProvider(), getIdentityAssertion()

•  Used to verify idenFty via third party, e.g., Facebook Connect

•  Both methods are opFonal •  onidentity handler called amer any verificaFon aWempt

•  RTCPeerConnection.peerIdentity holds any verified idenFty asserFon


Pseudo Code Walkthrough

Pseudo Code

•  Close to real code, but . . . •  No HTML, no signaling channel, not asynchronous, and API is sFll in flux

•  Don't expect this to work anywhere


Back to first diagram

•  Mobile browser "calls" laptop browser •  Each sends media to the other

Stereo Audio

Browser M on Mobile

Browser L on Laptop

Front Camera Video

Rear Camera Video

Microphone Audio


WebCam Video


Mobile browser code outline

•  We will look next at each of these •  . . . except for creaFng the signaling channel

var pc; var configuration =

{"iceServers":[{"url":"stun:198.51.100.9"},

{"url":"turn:198.51.100.2",

"credential":"myPassword"}]}; var microphone, application, front, rear;

var presentation, presenter, demonstration;

var remote_av, stereo, mono;

var display, left, right;

function s(sdp) {} // stub success callback

function e(error) {} // stub error callback

var signalingChannel = createSignalingChannel();

getMedia();

createPC();

attachMedia();

call();

function getMedia() { // get local audio (microphone)

navigator.getUserMedia({"audio": true }, function (stream) {

microphone = stream;

}, e);

// get local video (application sharing)

///// This is outside the scope of this specification.

///// Assume that 'application' has been set to this stream. //

constraint =

{"video": {"mandatory": {"videoFacingModeEnum": "front"}}};

navigator.getUserMedia(constraint, function (stream) {

front = stream;

}, e);

constraint =

{"video": {"mandatory": {"videoFacingModeEnum": "rear"}}};


rear = stream;

}, e);

}

function createPC() {

pc = new RTCPeerConnection(configuration);

pc.onicecandidate = function (evt) {

signalingChannel.send(

JSON.stringify({ "candidate": evt.candidate }));

};

pc.onaddstream =

function (evt) {handleIncomingStream(evt.stream);}; }

function attachMedia() {

presentation =

new MediaStream(

[microphone.getAudioTracks()[0], // Audio

application.getVideoTracks()[0]]); // Presentation

presenter = new MediaStream(


front.getVideoTracks()[0]]); // Presenter

demonstration =

new MediaStream(


rear.getVideoTracks()[0]]); // Demonstration

pc.addStream(presentation);

pc.addStream(presenter);

pc.addStream(demonstration);


JSON.stringify({ "presentation": presentation.id,

"presenter": presenter.id,

"demonstration": demonstration.id

})); }

function call() {

pc.createOffer(gotDescription, e);

function gotDescription(desc) {

pc.setLocalDescription(desc, s, e);

signalingChannel.send(JSON.stringify({ "sdp": desc }));

}

}

function handleIncomingStream(st) {

if (st.getVideoTracks().length == 1) {

av_stream = st;

show_av(av_stream); } else if (st.getAudioTracks().length == 2) {

stereo = st;

} else {

mono = st;

}

}

function show_av(st) {

display.src = URL.createObjectURL( new MediaStream(st.getVideoTracks()[0]));

left.src = URL.createObjectURL(

new MediaStream(st.getAudioTracks()[0]));

right.src = URL.createObjectURL(


}

signalingChannel.onmessage = function (msg) { var signal = JSON.parse(msg.data);

if (signal.sdp) {

pc.setRemoteDescription(

new RTCSessionDescription(signal.sdp), s, e);

} else {

pc.addIceCandidate(

new RTCIceCandidate(signal.candidate)); }

};


getMedia();

createPC();

attachMedia();

call();


Mobile browser produces . . .

•  At least 3 calls to getUserMedia() •  Three calls to new MediaStream() •  App sends stream ids, then streams

Sources

Presenter Video

DemonstraFon Video

Audio

PresentaFon Video

PresentaFon Stream

Presenter Stream

DemonstraFon Stream

Captured MediaStreams Created MediaStreams Tracks

“Audio” Track


“Audio” Track


“Audio” Track


Front Camera Video

Rear Camera Video

Microphone Audio


Browser M


funcFon getMedia() [1]

•  Get audio •  (Get window video – out of scope)



{"url":"turn:198.51.100.2",








getMedia();

createPC();

attachMedia();

call();




}, e);




constraint =



front = stream;

}, e);

constraint =



rear = stream;

}, e);

}






};

pc.onaddstream =



presentation =

new MediaStream(






demonstration =

new MediaStream(










})); }

function call() {





}

}



av_stream = st;


stereo = st;

} else {

mono = st;

}

}







}


if (signal.sdp) {



} else {

pc.addIceCandidate(


};

navigator.getUserMedia({"audio": true }, function (stream) { microphone = stream;

}, e);



///// Assume that 'application' has been set to this stream.

//

. . .


funcFon getMedia() [2]

•  Get front-‐facing camera •  Get rear-‐facing camera



{"url":"turn:198.51.100.2",








getMedia();

createPC();

attachMedia();

call();




}, e);




constraint =



front = stream;

}, e);

constraint =



rear = stream;

}, e);

}






};

pc.onaddstream =



presentation =

new MediaStream(






demonstration =

new MediaStream(










})); }

function call() {





}

}



av_stream = st;


stereo = st;

} else {

mono = st;

}

}







}


if (signal.sdp) {



} else {

pc.addIceCandidate(


};

. . . constraint =



front = stream;

}, e);

constraint =

{"video": {"mandatory": {"videoFacingModeEnum": "rear"}}}; navigator.getUserMedia(constraint, function (stream) {

rear = stream;

}, e);


funcFon createPC()

•  Create RTCPeerConnection •  Set handlers



{"url":"turn:198.51.100.2",








getMedia();

createPC();

attachMedia();

call();




}, e);




constraint =



front = stream;

}, e);

constraint =



rear = stream;

}, e);

}






};

pc.onaddstream =



presentation =

new MediaStream(






demonstration =

new MediaStream(










})); }

function call() {





}

}



av_stream = st;


stereo = st;

} else {

mono = st;

}

}







}


if (signal.sdp) {



} else {

pc.addIceCandidate(


};

var configuration = {"iceServers":[{"url":"stun:198.51.100.9"},

{"url":"turn:198.51.100.2",

"credential":"myPassword"}]};




JSON.stringify({ "candidate": evt.candidate })); };

pc.onaddstream =

function (evt) {handleIncomingStream(evt.stream);};


Mobile browser consumes . . .

•  Receives three media streams •  Chooses one •  Sends tracks to output channels

Lem Headphone

Display

Right Headphone

Stereo Stream

Mono Stream

Audio & Video Stream

Sinks MediaStreams

“Right” Track

“Lem” Track

“Mono” track

“Video” Track

“Right” Track “Lem” Track

Tracks

(Audio & Video Stream selected) Browser M


FuncFon handleIncomingStream()

•  If incoming stream has video track, set to av_stream and display it

•  If it has two audio tracks, must be stereo •  Otherwise, must be the mono stream



{"url":"turn:198.51.100.2",








getMedia();

createPC();

attachMedia();

call();




}, e);




constraint =



front = stream;

}, e);

constraint =



rear = stream;

}, e);

}






};

pc.onaddstream =



presentation =

new MediaStream(






demonstration =

new MediaStream(










})); }

function call() {





}

}



av_stream = st;


stereo = st;

} else {

mono = st;

}

}







}


if (signal.sdp) {



} else {

pc.addIceCandidate(


};

if (st.getVideoTracks().length == 1) { av_stream = st;

show_av(av_stream);

} else if (st.getAudioTracks().length == 2) {

stereo = st;

} else {

mono = st;

}


FuncFon show_av(st)

•  Turn streams into URLs •  Set as source for media elements



{"url":"turn:198.51.100.2",








getMedia();

createPC();

attachMedia();

call();




}, e);




constraint =



front = stream;

}, e);

constraint =



rear = stream;

}, e);

}






};

pc.onaddstream =



presentation =

new MediaStream(






demonstration =

new MediaStream(










})); }

function call() {





}

}



av_stream = st;


stereo = st;

} else {

mono = st;

}

}







}


if (signal.sdp) {



} else {

pc.addIceCandidate(


};

display.srcObject = new MediaStream(st.getVideoTracks()[0]);

left.srcObject =

new MediaStream(st.getAudioTracks()[0]);

right.srcObject =







{"url":"turn:198.51.100.2",








getMedia();

createPC();

attachMedia();

call();




}, e);




constraint =



front = stream;

}, e);

constraint =



rear = stream;

}, e);

}






};

pc.onaddstream =



presentation =

new MediaStream(






demonstration =

new MediaStream(










})); }

function call() {





}

}



av_stream = st;


stereo = st;

} else {

mono = st;

}

}







}


if (signal.sdp) {



} else {

pc.addIceCandidate(


};


getMedia();

createPC();

attachMedia();

call();


funcFon aWachMedia() [1]

•  Create 3 new streams, all with same audio but different video



{"url":"turn:198.51.100.2",








getMedia();

createPC();

attachMedia();

call();




}, e);




constraint =



front = stream;

}, e);

constraint =



rear = stream;

}, e);

}






};

pc.onaddstream =



presentation =

new MediaStream(






demonstration =

new MediaStream(










})); }

function call() {





}

}



av_stream = st;


stereo = st;

} else {

mono = st;

}

}







}


if (signal.sdp) {



} else {

pc.addIceCandidate(


};

presentation = new MediaStream(



presenter =

new MediaStream(



demonstration = new MediaStream(



. . .


funcFon aWachMedia() [2]

•  AWach all 3 streams to Peer ConnecFon •  Send stream ids to peer (before streams!)



{"url":"turn:198.51.100.2",








getMedia();

createPC();

attachMedia();

call();




}, e);




constraint =



front = stream;

}, e);

constraint =



rear = stream;

}, e);

}






};

pc.onaddstream =



presentation =

new MediaStream(






demonstration =

new MediaStream(










})); }

function call() {





}

}



av_stream = st;


stereo = st;

} else {

mono = st;

}

}







}


if (signal.sdp) {



} else {

pc.addIceCandidate(


};

pc.addStream(presentation); pc.addStream(presenter);






}));






{"url":"turn:198.51.100.2",








getMedia();

createPC();

attachMedia();

call();




}, e);




constraint =



front = stream;

}, e);

constraint =



rear = stream;

}, e);

}






};

pc.onaddstream =



presentation =

new MediaStream(






demonstration =

new MediaStream(










})); }

function call() {





}

}



av_stream = st;


stereo = st;

} else {

mono = st;

}

}







}


if (signal.sdp) {



} else {

pc.addIceCandidate(


};


getMedia();

createPC();

attachMedia();

call();


funcFon call()

•  Ask browser to create SDP offer •  Set offer as local descripFon •  Send offer to peer



{"url":"turn:198.51.100.2",








getMedia();

createPC();

attachMedia();

call();




}, e);




constraint =



front = stream;

}, e);

constraint =



rear = stream;

}, e);

}






};

pc.onaddstream =



presentation =

new MediaStream(






demonstration =

new MediaStream(










})); }

function call() {





}

}



av_stream = st;


stereo = st;

} else {

mono = st;

}

}







}


if (signal.sdp) {



} else {

pc.addIceCandidate(


};





}


How do we get the SDP answer?

•  Signaling channel provides message •  If SDP, set as remote descripFon •  If ICE candidate, tell the browser



{"url":"turn:198.51.100.2",








getMedia();

createPC();

attachMedia();

call();




}, e);




constraint =



front = stream;

}, e);

constraint =



rear = stream;

}, e);

}






};

pc.onaddstream =



presentation =

new MediaStream(






demonstration =

new MediaStream(










})); }

function call() {





}

}



av_stream = st;


stereo = st;

} else {

mono = st;

}

}







}


if (signal.sdp) {



} else {

pc.addIceCandidate(


};


if (signal.sdp) {



} else {

pc.addIceCandidate(


};


And now the laptop browser . . .

•  Watch for the following – We set up media *amer* receiving the offer – but the signaling channel sFll must exist first! – Also, need to save incoming stream ids


Signaling channel message is trigger

•  Set up PC and media if not already done



{"url":"turn:198.51.100.2",

"credential":"myPassword"}]}; var webcam, left, right;

var av, stereo, mono;

var incoming;

var speaker, win1, win2, win3;




function prepareForIncomingCall() {

createPC();

getMedia();

attachMedia(); }






pc.onaddstream =


}

function getMedia() {

navigator.getUserMedia({"video": true }, function (stream) {

webcam = stream;

}, e);

constraint =

{"audio": {"mandatory": {"audioDirectionEnum": "left"}}};


left = stream;

}, e);

constraint =

{"audio": {"mandatory": {"audioDirectionEnum": "right"}}};


right = stream;

}, e);

}


av = new MediaStream(

[webcam.getVideoTracks()[0], // Video

left.getAudioTracks()[0], // Left audio

right.getAudioTracks()[0]]); // Right audio

stereo = new MediaStream(

[left.getAudioTracks()[0], // Left audio


mono = left; // Treat the left audio as the mono stream

pc.addStream(av);

pc.addStream(stereo);

pc.addStream(mono);

}

function answer() {

pc.createAnswer(gotDescription, e);




}

}


if (st.id === incoming.presentation) {

speaker.src = URL.createObjectURL(


win1.src = URL.createObjectURL(

new MediaStream(st.getVideoTracks()[0]));

} else if (st.id === incoming.presenter) { win2.src = URL.createObjectURL(


} else {



}

}

signalingChannel.onmessage = function (msg) {

if (!pc) {

prepareForIncomingCall();

}

var sgnl = JSON.parse(msg.data);

if (sgnl.sdp) {

pc.setRemoteDescription( new RTCSessionDescription(sgnl.sdp), s, e);

answer();

} else if (sgnl.candidate) {

pc.addIceCandidate(new RTCIceCandidate(sgnl.candidate));

} else {

incoming = sgnl;

}

};

signalingChannel.onmessage = function (msg) { if (!pc) { prepareForIncomingCall(); } var sgnl = JSON.parse(msg.data);

. . .

};


Signaling channel message is trigger

•  If SDP, *also* answer •  But if neither SDP nor ICE candidate, must be set of incoming stream ids, so save



{"url":"turn:198.51.100.2",



var incoming;






createPC();

getMedia();

attachMedia(); }






pc.onaddstream =


}



webcam = stream;

}, e);

constraint =



left = stream;

}, e);

constraint =



right = stream;

}, e);

}










pc.addStream(av);


pc.addStream(mono);

}

function answer() {





}

}









} else {



}

}


if (!pc) {


}


if (sgnl.sdp) {


answer();



} else {

incoming = sgnl;

}

};


. . .

if (sgnl.sdp) {


new RTCSessionDescription(sgnl.sdp), s, e);

answer(); } else if (sgnl.candidate) {

pc.addIceCandidate(new RTCIceCandidate(sgnl.candidate)); } else { incoming = sgnl; }

};


FuncFon prepareForIncomingCall()

•  No suprises here •  Media obtained is a liWle different •  But aWached the same way



{"url":"turn:198.51.100.2",



var incoming;






createPC();

getMedia();

attachMedia(); }






pc.onaddstream =


}



webcam = stream;

}, e);

constraint =



left = stream;

}, e);

constraint =



right = stream;

}, e);

}










pc.addStream(av);


pc.addStream(mono);

}

function answer() {





}

}









} else {



}

}


if (!pc) {


}


if (sgnl.sdp) {


answer();



} else {

incoming = sgnl;

}

};

createPC();

getMedia();

attachMedia();


FuncFon answer()

•  createAnswer() automaFcally uses value of remoteDescription when generaFng new SDP



{"url":"turn:198.51.100.2",



var incoming;






createPC();

getMedia();

attachMedia(); }






pc.onaddstream =


}



webcam = stream;

}, e);

constraint =



left = stream;

}, e);

constraint =



right = stream;

}, e);

}










pc.addStream(av);


pc.addStream(mono);

}

function answer() {





}

}









} else {



}

}


if (!pc) {


}


if (sgnl.sdp) {


answer();



} else {

incoming = sgnl;

}

};





}


Laptop browser consumes . . .

•  Three input streams •  All have same # of audio and video tracks •  Need stream ids to disFnguish

Tracks MediaStreams Sinks

Display

Display

Speaker

Display

“Audio” Track


“Audio” Track


“Audio” Track


PresentaFon Stream

Presenter Stream

DemonstraFon Stream

(All video streams selected) Browser L


FuncFon handleIncomingStream()

•  Use ids to disFnguish streams •  Extract one audio and all video tracks •  Assign to element sources



{"url":"turn:198.51.100.2",



var incoming;






createPC();

getMedia();

attachMedia(); }






pc.onaddstream =


}



webcam = stream;

}, e);

constraint =



left = stream;

}, e);

constraint =



right = stream;

}, e);

}










pc.addStream(av);


pc.addStream(mono);

}

function answer() {





}

}









} else {



}

}


if (!pc) {


}


if (sgnl.sdp) {


answer();



} else {

incoming = sgnl;

}

};

if (st.id === incoming.presentation) { speaker.srcObject =


win1.srcObject =

new MediaStream(st.getVideoTracks()[0]);

} else if (st.id === incoming.presenter) {

win2.srcObject =


} else { win3.srcObject =


}


Right Microphone

WebCam

Lem Microphone

Stereo Stream

Mono Stream

Audio & Video Stream

Sources Created MediaStreams

Right Audio

Video

Lem Audio


“Right” Track

“Lem” Track

“Mono” Track

“Video” Track

“Right” Track “Lem” Track

Tracks

Browser L

Laptop browser produces . . .

•  Three calls to getUserMedia() •  Three calls to new MediaStream() •  No stream ids needed


FuncFon getMedia() [1]

•  Request webcam video



{"url":"turn:198.51.100.2",



var incoming;






createPC();

getMedia();

attachMedia(); }






pc.onaddstream =


}



webcam = stream;

}, e);

constraint =



left = stream;

}, e);

constraint =



right = stream;

}, e);

}










pc.addStream(av);


pc.addStream(mono);

}

function answer() {





}

}









} else {



}

}


if (!pc) {


}


if (sgnl.sdp) {


answer();



} else {

incoming = sgnl;

}

};

navigator.getUserMedia({"video": true}, function (stream) { webcam = stream; }, e);

. . .


FuncFon getMedia() [2]

•  Request lem and right audio streams •  Save them as left and right variables



{"url":"turn:198.51.100.2",



var incoming;






createPC();

getMedia();

attachMedia(); }






pc.onaddstream =


}



webcam = stream;

}, e);

constraint =



left = stream;

}, e);

constraint =



right = stream;

}, e);

}










pc.addStream(av);


pc.addStream(mono);

}

function answer() {





}

}









} else {



}

}


if (!pc) {


}


if (sgnl.sdp) {


answer();



} else {

incoming = sgnl;

}

};

. . .

constraint =


navigator.getUserMedia(constraint,

function (stream) {left = stream;}, e);

constraint =

{"audio": {"mandatory": {"audioDirectionEnum": "right"}}}; navigator.getUserMedia(constraint,

function (stream) {right = stream;}, e);


FuncFon aWachMedia()

•  Create new streams •  Add them to PC for transmission



{"url":"turn:198.51.100.2",



var incoming;






createPC();

getMedia();

attachMedia(); }






pc.onaddstream =


}



webcam = stream;

}, e);

constraint =



left = stream;

}, e);

constraint =



right = stream;

}, e);

}










pc.addStream(av);


pc.addStream(mono);

}

function answer() {





}

}









} else {



}

}


if (!pc) {


}


if (sgnl.sdp) {


answer();



} else {

incoming = sgnl;

}

};

av = new MediaStream( [webcam.getVideoTracks()[0], // Video







pc.addStream(av);


pc.addStream(mono);


In real code . . .

•  The error callbacks must do something useful •  Methods with callbacks are asynchronous! – Need to wait for callbacks before conFnuing

•  Signaling channel is very important


PracFcal bits

•  Signaling •  Audio and video •  Data channel •  More info online


Signaling

•  Decision is most challenging part of WebRTC •  Server-‐side code provides –  LocaFng and idenFfying peer browser – Message (signaling) relaying –  Possibly media relaying/transcoding if a TURN server

•  OpFons – Write your own server code

•  Anything “real-‐Fme” will do – Use exisFng services

•  PubNub, FireBase, Google App Engine TAD Summit Bangkok 2013 88

Audio and Video

•  Very recent agreement on how to send and synchronize mulFple flows of same type

•  For interop, keep it one of each only •  SFll no agreement on codecs – Chrome: VP8 and H.264 – Firefox: VP8


Data Channel

•  Minimum safe chunk size sFll under debate – A few kb should be okay

•  Protocol field is for app use


Great online resources

•  InformaFonal sites – webrtc.org – html5rocks.com/en/tutorials/webrtc/basics – webrtchacks.com

•  Games/demos/apps – www.cubeslam.com – shinydemos.com/facekat – sharefest.me (github.com/Peer5/Sharefest)


The WebRTC School is the web’s premier locaFon for WebRTC Integrator and Developer educaFon. Associated with the training programs are the industry supported cerFficaFons, the WebRTC School Qualified Integrator (WSQI™) and WebRTC School Qualified Developer (WSQD™). For more informaFon and online demos, please visit

hWp://www.webrtcschool.com

Training and CerFficaFon


QuesFons?

hWp://webrtcbook.com


Technology

TADS Developer Summit WebRTC Dan Burnett