Upload
luis-lopez
View
2.174
Download
2
Embed Size (px)
Citation preview
Implementing a WebRTC endpoint in GStreamer:
challenges, problems and perspectives
8-9 October 2015Dublin, Ireland
Conference 2015
Miguel París
Jose A. [email protected]
2
Who we areJosé Antonio Santos Cadenas
● Software Engineer
● Telematic Systems Master's
● Developer at Naevatec (Madrid, Spain)
● Kurento Media Server (KMS) manager
● Integrating with client APIs
Miguel París
● Software Engineer
● Telematic Systems Master's
● Researcher at Universidad Rey Juan Carlos (Madrid, Spain)
● Kurento real-time manager
● Twitter: @mparisdiaz
What is WebRTC
3
● WebRTC is a new movement bringing RTC to the WWW
● It is based on standards
● Supported by Chrome & Firefox
● Windows Edge added partial support recently
● Safari is rumored to be on the way of providing it
● FOSS implementations for Smartphone platforms (iOS and Android)
● More tan 1B devices currently support WebRTC
DevelopingRTC applications
for the WWW
Before WebRTC After WebRTC
Begin End
• Unified APIs• Standards• FOSS• Multiplatform
WebRTC standards
4
● Standard APIs: Standardized by the WebRTC WG at W3C
Media Capture APIs● Standardize APIs for media capture
PeerConnection APIs (WebRTC 1.0 almost finished)● Standardize API for P2P media communications● Media = Audio + Video + Data● DataChannel API: low latency
WebRTC stats● RTP stats for inbound and outbound directions● encode/decode stats● DataChannel stats
Implementing WebRTC in GStreamer
5
● Deep interest in the GStreamer community
● Several implementations
OpenWebRTC (Ericsson)● Design for working as part of client applications
Kurento Media Server (Kurento.org)● Design for working as part of media server infrastructures
Main differences between them: OpenWebRTC captures media from camera and mic. KMS doesn't OpenWebRTC renders to display and speakers. KMS doesn't OpenWebRTC encodes/decodes audio/video. KMS only if needed.
6
<GstPipeline>pipeline0
[>]
KmsWebrtcEndpointkmswebrtcendpoint1
[>]
GstRTPRtxQueuertprtxqueue1
[>]
GstRtpVP8Payrtpvp8pay1
[>]
GstRtpOPUSPayrtpopuspay1
[>]
KmsWebrtcSessionkmswebrtcsession1
[>]
KmsRtcpDemuxkmsrtcpdemux1
[>] GstRtpSsrcDemuxrtpssrcdemux5
[>]
KmsWebrtcTransportSinkNicekmswebrtctransportsinknice1
[>]
GstNiceSinknicesink1
[>]
GstDtlsSrtpEncdtlssrtpenc1
[>]
GstFunnelfunnel
[>]
GstSrtpEncsrtp-encoder
[>]
GstDtlsEncdtls-encoder
[>]
KmsWebrtcTransportSrcNicekmswebrtctransportsrcnice1
[>]
GstDtlsSrtpDecdtlssrtpdec1
[>]
GstSrtpDecsrtp-decoder
[>]
GstDtlsDecdtls-decoder
[>]
GstDtlsSrtpDemuxdtls-srtp-demux
[>]
GstNiceSrcnicesrc1
[>]
GstRtpBinrtpbin1
[>]
GstRtpSsrcDemuxrtpssrcdemux4
[>]
GstRtpSessionrtpsession3
[>]
GstRtpSsrcDemuxrtpssrcdemux3
[>]
GstRtpSessionrtpsession2
[>]
KmsWebrtcEndpointkmswebrtcendpoint0
[>]
GstRtpVP8Depayrtpvp8depay0
[>]
KmsAgnosticBin2kmsagnosticbin2-1
[>]
GstQueuequeue3
[>]
KmsParseTreeBinkmsparsetreebin1
[>]
KmsVp8Parsekmsvp8parse0
[>]
GstFakeSinkfakesink3
[>]
GstTeetee3[>]
GstFakeSinkfakesink2
[>]
GstTeetee2[>]
GstRTPOpusDepayrtpopusdepay0
[>]
KmsAgnosticBin2kmsagnosticbin2-0
[>]
GstQueuequeue1
[>]
KmsParseTreeBinkmsparsetreebin0
[>]
GstOpusParseopusparse0
[>]
GstFakeSinkfakesink1
[>]
GstTeetee1[>]
GstFakeSinkfakesink0
[>]
GstTeetee0[>]
GstRTPRtxQueuertprtxqueue0
[>]GstRtpVP8Payrtpvp8pay0
[>]
GstRtpOPUSPayrtpopuspay0
[>]
KmsWebrtcSessionkmswebrtcsession0
[>]
KmsRtcpDemuxkmsrtcpdemux0
[>]
GstRtpSsrcDemuxrtpssrcdemux2
[>]
KmsWebrtcTransportSinkNicekmswebrtctransportsinknice0
[>]
GstNiceSinknicesink0
[>]
GstDtlsSrtpEncdtlssrtpenc0
[>]
GstFunnelfunnel
[>]
GstSrtpEncsrtp-encoder
[>]
GstDtlsEncdtls-encoder
[>]
KmsWebrtcTransportSrcNicekmswebrtctransportsrcnice0
[>]
GstDtlsSrtpDecdtlssrtpdec0
[>]
GstSrtpDecsrtp-decoder
[>]
GstDtlsDecdtls-decoder
[>]
GstDtlsSrtpDemuxdtls-srtp-demux
[>]
GstNiceSrcnicesrc0
[>]
GstRtpBinrtpbin0
[>]
GstRtpJitterBufferrtpjitterbuffer1
[>] GstRtpPtDemuxrtpptdemux1
[>]
GstRtpJitterBufferrtpjitterbuffer0
[>] GstRtpPtDemuxrtpptdemux0
[>]
GstRtpSsrcDemuxrtpssrcdemux1
[>]
GstRtpSessionrtpsession1
[>]
GstRtpSsrcDemuxrtpssrcdemux0
[>]
GstRtpSessionrtpsession0
[>]
LegendElement-States: [~] void-pending, [0] null, [-] ready, [=] paused, [>] playingPad-Activation: [-] none, [>] push, [<] pullPad-Flags: [b]locked, [f]lushing, [b]locking; upper-case is setPad-Task: [T] has started task, [t] has paused task
proxypad40[>][bfb]
sink[>][bfb]
sink_audio[>][bfb]
proxypad42[>][bfb]
sink[>][bfb]
sink_video[>][bfb]
sink[>][bfb]
src[>][bfb]
send_rtp_sink_1[>][bfb]
proxypad33[>][bfb]
src[>][bfb]
src[>][bfb]
send_rtp_sink_0[>][bfb]
proxypad31[>][bfb]
send_rtp_src_0[>][bfb]
sink[>][bfb]
rtp_src[>][bfb]
rtcp_src[>][bfb] rtcp_sink
[>][bfb]
sink[>][bfb]
src_1[>][bfb] recv_rtp_sink_1
[>][bfb]
rtcp_src_1[>][bfb] recv_rtcp_sink_1
[>][bfb]
src_421259003[>][bfb] recv_rtp_sink_0
[>][bfb]
rtcp_src_421259003[>][bfb] recv_rtcp_sink_0
[>][bfb]
proxypad44[>][bfb]
proxypad45[>][bfb]
proxypad46[>][bfb]
proxypad47[>][bfb]
sink[>][bfb]
proxypad34[>][bfb]
rtp_sink_0[>][bfb]
rtp_sink_0[>][bfb]
src[>][bfb]
proxypad36[>][bfb]
rtcp_sink_0[>][bfb]
rtcp_sink_0[>][bfb]
proxypad37[>][bfb]
rtp_sink_1[>][bfb]
rtp_sink_1[>][bfb]
proxypad39[>][bfb]
rtcp_sink_1[>][bfb]
rtcp_sink_1[>][bfb]
proxypad29[>][bfb]
funnelpad5[>][bfb]
src[>][bfb]
funnelpad6[>][bfb]
funnelpad7[>][bfb]
funnelpad8[>][bfb]
funnelpad9[>][bfb]
rtp_src_0[>][bfb]
rtcp_src_0[>][bfb]
rtp_src_1[>][bfb]
rtcp_src_1[>][bfb]
src[>][bfb][T]
proxypad28[>][bfb]
sink[>][bfb]
sink[>][bfb]
rtp_src[>][bfb]
proxypad26[>][bfb]
proxypad27[>][bfb]
rtcp_src[>][bfb]
rtp_sink[>][bfb]
rtp_src[>][bfb]
rtcp_sink[>][bfb]
rtcp_src[>][bfb]
sink[>][bfb]
rtp_src[>][bfb]
dtls_src[>][bfb]
src[>][bfb][T]
send_rtp_sink[>][bfb]
send_rtp_sink[>][bfb]
recv_rtp_sink[>][bfb]
recv_rtcp_sink[>][bfb]
recv_rtp_sink[>][bfb]
recv_rtcp_sink[>][bfb]
proxypad30[>][bfb]
proxypad32[>][bfb]
send_rtp_src_1[>][bfb]
proxypad35[>][bfb]
send_rtcp_src_0[>][bfb]
proxypad38[>][bfb]
send_rtcp_src_1[>][bfb]
sink[>][bfb]
rtcp_sink[>][bfb]
send_rtp_src[>][bfb]
send_rtcp_src[>][bfb]
recv_rtp_src[>][bfb]
sync_src[>][bfb]
sink[>][bfb]
rtcp_sink[>][bfb]
send_rtp_src[>][bfb]
send_rtcp_src[>][bfb]
recv_rtp_src[>][bfb]
sync_src[>][bfb]
proxypad14[>][bfb]
sink[>][bfb]
sink_audio[>][bfb]
audio_src_0[>][bfb]
proxypad15[>][bfb]
sink[>][bfb]
sink_video[>][bfb]
proxypad24[>][bfb]
proxypad25[>][bfb]
video_src_0[>][bfb]
sink[>][bfb]
src[>][bfb]
sink[>][bfb]
proxypad23[>][bfb]
src_0[>][bfb]
sink[>][bfb]
proxypad43[>][bfb]sink
[>][bfb]src
[>][bfb][T]
sink[>][bfb]
src[>][bfb]
sink[>][bfb]
src_0[>][bfb]
sink[>][bfb]
src_2[>][bfb]
sink[>][bfb]
src_0[>][bfb]
src_1[>][bfb]
sink[>][bfb]
src[>][bfb]
sink[>][bfb]
proxypad19[>][bfb]
src_0[>][bfb]
sink[>][bfb]
proxypad41[>][bfb]sink
[>][bfb]src
[>][bfb][T]
sink[>][bfb]
src[>][bfb]
sink[>][bfb]
src_0[>][bfb]
sink[>][bfb]
src_2[>][bfb]
sink[>][bfb]
src_0[>][bfb]
src_1[>][bfb]
sink[>][bfb]
src[>][bfb]
send_rtp_sink_1[>][bfb]
proxypad7[>][bfb]
src[>][bfb]
src[>][bfb]
send_rtp_sink_0[>][bfb]
proxypad5[>][bfb]
send_rtp_src_0[>][bfb]
sink[>][bfb]
rtp_src[>][bfb]
rtcp_src[>][bfb]
rtcp_sink[>][bfb]
sink[>][bfb]
src_1442068093[>][bfb]
recv_rtp_sink_0[>][bfb]
rtcp_src_1442068093[>][bfb]
recv_rtcp_sink_0[>][bfb]
src_836061664[>][bfb]
recv_rtp_sink_1[>][bfb]
rtcp_src_836061664[>][bfb]
recv_rtcp_sink_1[>][bfb]
proxypad16[>][bfb]
proxypad17[>][bfb]
proxypad20[>][bfb]
proxypad21[>][bfb]
sink[>][bfb]
proxypad8[>][bfb]
rtp_sink_0[>][bfb]
rtp_sink_0[>][bfb]
src[>][bfb]
proxypad10[>][bfb]
rtcp_sink_0[>][bfb]
rtcp_sink_0[>][bfb]
proxypad11[>][bfb]
rtp_sink_1[>][bfb]
rtp_sink_1[>][bfb]
proxypad13[>][bfb]
rtcp_sink_1[>][bfb]
rtcp_sink_1[>][bfb]
proxypad3[>][bfb]
funnelpad0[>][bfb]
src[>][bfb]
funnelpad1[>][bfb]
funnelpad2[>][bfb]
funnelpad3[>][bfb]
funnelpad4[>][bfb]
rtp_src_0[>][bfb]
rtcp_src_0[>][bfb]
rtp_src_1[>][bfb]
rtcp_src_1[>][bfb]
src[>][bfb][T]
proxypad2[>][bfb]
sink[>][bfb]
sink[>][bfb] rtp_src
[>][bfb]
proxypad0[>][bfb]
proxypad1[>][bfb]
rtcp_src[>][bfb]
rtp_sink[>][bfb]
rtp_src[>][bfb]
rtcp_sink[>][bfb]
rtcp_src[>][bfb]
sink[>][bfb]
rtp_src[>][bfb]
dtls_src[>][bfb]
src[>][bfb][T]
send_rtp_sink[>][bfb]
send_rtp_sink[>][bfb]
recv_rtp_sink[>][bfb]
recv_rtcp_sink[>][bfb]
recv_rtp_sink[>][bfb]
recv_rtcp_sink[>][bfb] proxypad4
[>][bfb]
proxypad6[>][bfb]
send_rtp_src_1[>][bfb]
proxypad9[>][bfb]
send_rtcp_src_0[>][bfb]
proxypad12[>][bfb]
send_rtcp_src_1[>][bfb]
proxypad18[>][bfb]
recv_rtp_src_0_1442068093_111[>][bfb]
proxypad22[>][bfb]
recv_rtp_src_1_836061664_100[>][bfb]
sink[>][bfb] src
[>][bfb][T]sink_rtcp[>][bfb]
sink[>][bfb]
src_100[>][bfb]
sink[>][bfb] src
[>][bfb][T]sink_rtcp[>][bfb]
sink[>][bfb]
src_111[>][bfb]
sink[>][bfb]
src_836061664[>][bfb]
rtcp_sink[>][bfb]
rtcp_src_836061664[>][bfb]
send_rtp_src[>][bfb]
send_rtcp_src[>][bfb]
recv_rtp_src[>][bfb]
sync_src[>][bfb]
sink[>][bfb]
src_1442068093[>][bfb]
rtcp_sink[>][bfb]
rtcp_src_1442068093[>][bfb]
send_rtp_src[>][bfb]
send_rtcp_src[>][bfb]
recv_rtp_src[>][bfb]
sync_src[>][bfb]
Kurento WebRtcEpThe pipeline
Interactive Connectivity EstablishmentICE I
7
● Using libnice (NiceSrc and NiceSink)
Involved RFCs:
RFC 5245: Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols
RFC 5389: Session Traversal Utilities for NAT (STUN)
RFC 5766: Traversal Using Relays around NAT (TURN): Relay Extensions to Session Traversal Utilities for NAT (STUN)
Interactive Connectivity EstablishmentICE II
8
● Our experience with libnice
0.1.7 version seem to be stable, but lack many required features
0.1.10 version incorporate relevant features but seem not to provide the appropriate stability yet
● Leaks of TCP sockets● Connection states● Connectivity problems (Fails 1/1000)
Fixed in 0.1.13 ? (we have to check it)
ICE III
9
● What next needed in ICE for WebRTC?
IPv6 shall be mandatory in WebRTC. We need to check IPv6 support in libnice draft-ietf-rtcweb-transports-09 (mandates IPv6 support)
HTTP CONNECT protocol for TURN might be important in corporate networks (connectivity)
draft-hutton-httpbis-connect-protocol-00 STUN consent freshness may become mandatory for
WebRTC. This may require some adjustments on libnice for supporting consent negotiation (security)
Draft-ietf-rtcweb-stun-consent-freshness-16 ICE stated published through events
ICE state should be reported as specified in the WebRTC 1.0 WG draft
DTLS / SRTP I
10
● Using dtlssrtp(dec|enc) bins
dtls(dec|enc) manages DTLS handshake
srtp(dec|enc) decrypts/encrypts RTP packets● Involved RFCs:
RFC 5764: Datagram Transport Security (DTLS) Extension to Establish Keys for the Secure Real-time Protocol (SRTP)
RFC 3711: The Secure Real-time Transport Protocol (SRTP)
DTLS / SRTP II
11
● What next for WebRTC?
New cryptographic framework for group communications
This is under definition at the Privacy Enhanced RTP Conferencing (PERC) IETF WG
Only early drafts here draft-jones-perc-private-media-framework-00 draft-jones-perc-private-media-reqts-00 draft-mattsson-perc-srtp-cloud-00 draft-westerlund-perc-webrtc-use-case-00
PERC shall provide in the next few months (improve efficiency) Enabling unicast group communications without requiring infrastructure to
decipher New mechanisms enabling unicast group communications without
requiring infrastructure to cipher once per RTC session. These may require re-factoring of current DTLS stack
DTLS state published through events.
Full connectivity requires ICE connectivity + successful DTLS handshake. DTLS is being incorporated by the W3C in the WebRTC 1.0 spec as event
generator dealing with connection health
Congestion control I
12
● This is an essential feature for providing satisfactory QoE for WebRTC users
How does it work?
Bandwidth estimator Based on packet delay (useful RTP ExtHdr: abs-send-time) Based on packet loss
REMB (Receiver Estimated Maximum Bitrate) GStreamer does not provide any kind of features for this off the self
How Kurento estimates bandwidth?
on-sending-rtcp
Get packet-loss, bytes-sent, etc. from RtpSession stats
Advantages: simple
Drawbacks Reactive and not predictive Requires packet loss to detect BW constraints Getting RtpSession stats quite inefficient
Congestion control II
13
● For having satisfactory congestion control we should look to the following drafts
draft-alvestrand-rmcat-remb-03: REMB mechanism implemented by Chrome
draft-ietf-rmcat-cc-requirements-09: Congestion Control Requirements for Interactive Real-Time Media
draft-ietf-avtcore-rtp-circuit-breakers-10: Multimedia Congestion Control: Circuit Breakers for Unicast RTP Sessions
● In summary, congestion control should:
be predictive
be efficient
be fair
"break the circuit" (drop the BW) in case of evidence of huge congestion This is to avoid breaking other types of traffic.
DataChannels
14
● Kurento has DataChannels support
● Protocols: DTLS-SCTP
● Drafts
draft-ietf-rtcweb-data-channel
draft-ietf-rtcweb-data-protocol
draft-ietf-mmusic-data-channel-sdpneg Impl. based on sctp(dec|enc) provided by Ericsson
What have Kurento done?
Fixed problems when using Gstreamer 1.4
Fixed a lot of bugs
Implementation of the DataChannel protocol AppSrc, AppSink
Thank you
Suggestions, comments and complains:[email protected]@gmail.com
http://www.kurento.orghttp://www.github.com/[email protected]: @kurentoms
http://www.nubomedia.eu
http://www.fi-ware.org
http://ec.europa.eu