View
143
Download
0
Category
Preview:
Citation preview
. © Copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Achieving Best Quality Video on Mobile with ScalabilityGene GershanokVidyoKranky Geek, 2016
. © Copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
. © Copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
• Multiparty Video API
• Cross Platform Support
• Customizable layout compositing
Vidyo.io
Video API Platform as a Service
https://vidyo.io@Vidyo_io
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
• Video Routing Platform• Video Codec• Audio AEC• Device Management • Rendering
Full picture
Components
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
• Fluctuating Network Bandwidth• Packet Loss & Jitter• Varying Device Capabilities• Bitrate vs. Battery Life• Maintaining Low Latency
Experience with building a platform
Mobile Video Challenges
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
A set of video frames
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Let’s keep it simple
F1
F2
F3
F4
F5
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Traditional single-layer coding
F1
F2
F3
F4
F5
----
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
E
A
Transcoding MCU
B
C
D
MCU
- High Delay- Transcoding Loss- Complexity & Cost- Limited Layouts
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
a
c
B
Simulcast Router
- Better error resiliency- Custom layouts- No server-side decode
A
B
C
a
b
c
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Simulcasting
For 2:1 resolution ratios:- ~50% overhead vs. single
layerHigh Resolution (A)
Low Resolution (a)
Simulcast
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
ac
Bb
Scalable Router
A
B
C
VR
a
b
c
- Error concealment on downlink- Less bandwidth- Synchronized streams
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Scalable Video Coding - SVC
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
A set of video frames
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Let’s keep it simple
F1
F2
F3
F4
F5
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Traditional single-layer coding
F1
F2
F3
F4
F5
----
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Temporally Scalable Coding
F1
F2
F3
F4
F5
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Temporally Scalable Coding
F1
F2
F3
F4
F5
TO TO
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Layer 0 (7.5 fps)
F1
F5
TO TO ----
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Layers 0 and 1 (15 fps)
F1
F5
TO
TO
T1
F3
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Layers 0, 1, and 2 (30 fps)
T1
TO
TO
T2 T2
F3
F1
F2
F4
F5
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Adaptation Example: 2 to 1 to 0
T1
TO
TO
T2 T2
F3
F1
F2
F4
F5
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Error Resilience
XT1
TO
TO
T2 T2
F3
F1
F2
F4
F5
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Error Resilience (cont’d)
XT1
TO
TO
T2 T2
F3
F1
F2
F4
F5
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Error Resilience (cont’d)
X
XT1
TO
TO
T2 T2
F3
F1
F2
F4
F5
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Error Resilience (cont’d)
XT1
TO
TO
T2 T2
TO
F3
F1
F2
F4
F5
F1
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Spatially Scalable Coding
TO
SO
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Scalable Video Coding
TO
SO
TO
SOT1
S1T2
S2
T2
S2
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Scalable Video Coding
TO
SO
TO
SOT1
S1T2
S2
T2
S2
360p@ 30 fps
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Scalable Video Coding
TO
SO
TO
SOT1
S1T2
S2
T2
S2
720p @ 15 fps
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Fluctuating Network Bandwidth
Mobile Video Challenges Addressed
Bit stream will scale as packets are dropped without needing IFrames
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Fluctuating Network Bandwidth Packet Loss and Jitter
Mobile Video Challenges Addressed
Bit stream will scale as packets are dropped without needing IFrames
Only need to FEC and NACK T0 packets
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Fluctuating Network Bandwidth Packet Loss and Jitter Varying Device
Capabilities
Mobile Video Challenges Addressed
Bit stream will scale as packets are dropped without needing IFrames
Only need to FEC and NACK T0 packets
Send and Receive based on screen real estate
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Fluctuating Network Bandwidth Packet Loss and Jitter
Bitrate vs Battery Life
Varying Device Capabilities
Mobile Video Challenges Addressed
Bit stream will scale as packets are dropped without needing IFrames
Only need to FEC and NACK T0 packets
Send and Receive based on screen real estate
Only ask for what you can decode
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Fluctuating Network Bandwidth Packet Loss and Jitter
Bitrate vs Battery Life
Varying Device Capabilities
Mobile Video Challenges Addressed
Maintaining Low Latency
Bit stream will scale as packets are dropped without needing IFrames
Only need to FEC and NACK T0 packets
Send and Receive based on screen real estate
Only ask for what you can decode
No transcoding, only forwarding packets
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
What did we gain?
~20%Fewer bits than simulcast
can tolerate more than 20% packet loss rate(using both temporal and spatial scalability)
ADAPTABILITY & SUPERB ERROR RESILIENCE
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Scalability in Video Codecs
• Already in:– H.264 (Annex G, SVC – temporal, spatial/quality)– VP8 (temporal only)– HEVC (H.265) v.1 (temporal) and v.2 (SHVC – spatial/quality)– VP9 (Vidyo & Google)
• Work in progress:– Alliance for Open Media - AV1 (temporal mandatory, spatial
under review)
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Error Resilience Support in Standards
• Error resilience features supported in the RTP payload formats of all modern video codecs:– VP8 (RFC 7441)– H.264 SVC (RFC 6190)– H.265 HEVC (RFC 7798)– VP9 (draft-ietf-payload-vp9)
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Vidyo WebRTC
Vidyo is active in the evolution of WebRTC:• Google / Vidyo are co-developing VP9, the next generation WebRTC
codec• Contributing software for both the WebM and WebRTC open source
projects• Co-Chairing IETF Group on AVT Extensions• Co-authored numerous RFCs related to signaling and transport
Vidyo participates in many standardization bodies:
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Scalable VP9 Status
• Chrome has full VP9 scalability in decoder– Encoder configure through URL flags
• Ongoing work with WebRTC Editors to provide ways to configure the video encoder for different use cases
© copyright 2016 Vidyo Inc., confidential, proprietary and patent pending information
Questions?
Gene Gershanok
@GeneGershanok@Vidyo_io
Recommended