Streaming Media & Entertainment

© Copyright 2019 Xilinx赛灵思技术日XILINX TECHNOLOGY DAY

赛灵思技术日XILINX TECHNOLOGY DAY

Bob FengDirector, Consumer Business SegmentPrincipal System Architect, ProAV & Broadcast Business Segment03/19/2019

Streaming Media & Entertainment


Professional Streaming Endpoints

˃ DistributionHigh quality & Low latency

‒ Pleasant user experience

Low bitrate‒ Network bandwidth efficiency

Low power‒ Cost efficiency

˃ ArchiveLossless quality‒ Post editing & production

Low storage size‒ Cost efficiency

Simultaneously satisfy distribution & archiving need


RTP

/RTS

P –

UD

P C

ompr

esse

d Fl

ows

Unc

ompr

esse

d Fl

ows

Unc

ompr

esse

d Fl

ows

Professional Streaming Network

Sense + Acquire

Distribute Watch + Interact

Deliver (Stream)

Compress + Package

Capture + Process

Content Delivery Network

Scal

e / C

onve

rt / M

ix

Manageable End-to-End Latency

LightCompression

HLS

>> 3


Shared Media Content at Home

˃ One set up box per broadband connectionInadequate bandwidth for more anyway

˃ Leverage in-house home network WiFi or Gigabit Ethernet

˃ Transcoded stream and multicastMultiple watchers in multiple rooms

Set Up Box

Broadband Internet

Shared Watch: No Compromised Immersion


Shared Game Play at Home

Shared Play: Equalized Smoothness

Geforce GTX 1080tiCoffee Lake i9-9900K64GB DDR4


Broadband Modem

Office

RemoteContent

Game Console

STB

1 GbE / WIFI Switcher/Router1 GbE

HDMI HDMIHDMI HDMI

HDMI HDMI

HDMI

1 GbE 1 GbE

1 GbE

1 GbE1 GbE

Broadband

STB STB

STB

A lot of setup boxes: Broadband Bandwidth ???

Conventional Content Watching Network at Home


Retaining the best quality & Minimizing the end-to-end Latency

Latency

Conventional Game Play Network at Home


Challenge #1 - Picture Quality: A Bandwidth Trade-off

˃ Original ˃ 55Mbps

˃ Average PSNR: 33.12 dB

˃ 6.65Mbps

˃ Average PSNR: 23.98 dB


Video vs. Home Broadband Bandwidth

High resolution video: The root cause of Internet Speed Demand

Deeply H.264/265 compressed; If lower, picture quality suffers

Raw Bandwidth 0.4 Gbps 3 Gbps 12 Gbps 48 Gbps


Challenge #2: Latency - Game Play Crucial Factor

Smooth game play: Motion (joystick/mouse move) to Photon (Display) < 20 ms


Challenge #3: Codec Mystery - Lossless vs. Lossy

Type Codec Choices Compression Ratio

Picture Quality (PQ) 4K Bandwidth

Latency Network

Lossless(Visually or Mathematically)

TICO/JPEG-XSVC-2JPEG2000DNxHDProRes

4:1 to 10:1 Best PQ (close to original) and low-latency (<< 1 frame *)4:2:2/4:4:4 - suitable for both video and graphics content

1 ~ 3 Gbps 1 ~ 10 ms Wired:Gigabit EthernetLAN / Home

Lossy H.264/AVCH.265/HEVC

100:1 to 300:1 Decent PQ and latency (>= 1 frame *)4:2:0 typical –suitable for video content

25 ~ 50 Mbps >= 16ms Wireless:802.11 A/CInternet / WAN

Proper choose between codec type and network

* Per 60-Hz frame = 16ms


Stream Throughput > Network BandwidthToo many setup boxes

Too many concurrent streams

Timing critical AV packet Late arrivalsIrrelevant competing packets

Challenge #4: Packet Losses & Out of order

Appropriate traffic shaping and prioritization desired


Unbearable Experience #1: Crappy Image Quality

Streaming Master: Game PC Streaming Slave: Mac Air

Likely Cause: Improper codec + packet loss


Unbearable Experience #2: Obese Latency

Streaming Master: Game PC Streaming Slave: Mac Air – delayed 100ms

Likely Cause: Improper codec + Packet out of order


Req

uire

men

tsTr

ends

Other Challenges

Multi-Stream & Multi-Codec

Several formats, codecs, and bit rates

Performance

Portable, low power, flexibility

Need to reduce end-to-end latency

Low LatencyLive IP Streaming

YouTube, Facebook, othersOTT streaming

Synchronization

Simultaneous capture & playback

HD and UltraHD

1080p and 2160p growth

Connectivity

Higher data rates and different standards

Storage

Several types

>> 15


Solution Offers from Xilinx


Zynq UltraScale+ MPSoC Single-Chip Solution

Real-Time Processors32-bit Dual-CoreUp to 600MHz

Platform & Power ManagementGranular Power ControlFunctional Safety

Configuration & Security UnitAnti-Tamper & TrustIndustry Standards

FPGA Acceleration16nm UltraScale+™ LogicCustomizable Engines

Video Codec8K4K (15fps)4K2K (60fps)

High-speedPeripheralsPCIe® Gen2, USB 3.0, DisplayPort

Graphics ProcessorARM Mali-400MP22D/3D Visualization

MemorySubsystemDDR3/4LPDDR3/4

Application Processors64-bit ARM®v8Up to 1.5GHz

* ZU11EG, ZU15EG, ZU19EG Only

CG DevicesDual-Core A53s

EG DevicesQuad-Core A53sGPU

EV DevicesQuad-Core A53sGPU and Video Codec

>> 17


Xilinx VCU FeaturesEncoder Decoder

Feature H.265 (HEVC) H.264 (AVC) H.265 (HEVC) H.264 (AVC)General • Embedded Processors

• Separate Encoder/Decoder Engines -> Simultaneous Encode and Decode• Up to 8 streams of encode and 8 streams of decode simultaneously; 1 stream at max. resolution & rate

Profiles MainMain IntraMain10Main10 IntraMain 4:2:2 10Main 4:2:2 10 Intra

BaselineMainHighHigh10High 4:2:2

MainMain IntraMain10Main10 IntraMain 4:2:2 10Main 4:2:2 10 Intra

Baseline MainHighHigh10High 4:2:2

Level Up to 5.1 High Tier Up to 5.2 Up to 5.1 High Tier Up to 5.2Maximum Resolution/rate 3840x2160p60 (4KUHD), 7680x4320p15 (8KUHD), 4096x2160p30 (DCI4K)Sample Bit Depth 8 or 10 bpcChroma Format YCbCr 4:2:0, 4:2:2, 4:0:0Slice Type I, P, BVideo Format ProgressivePixel Format Semi Planar (YCbCr)Entropy Encoding CABAC CABAC, CAVLC CABAC CABAC, CAVLCSlice Support Supported Supported Supported Supported Tile Support Supported Not Applicable Supported Not ApplicableIn-loop Deblocking Filter Supported Supported Supported SupportedConfigurable Bit Rate Up to 533Mb/s Up to 960Mb/s Up to 533Mb/s Up to 960Mb/s

Rate Control CBR, VBR, Constant QP

>> 18


Xilinx VCU Latency˃ Xilinx has control over stages 2, 3, 5, and 6

Stages 1, 4, 7 are user dependentAdditional latencies in jitter buffers (few ms) and input/output sync (up to 1 frame)*

˃ The calculations assume 60 frames per second

˃ Low latency assumes 8 slices per frame

˃ Low latency will have restrictions on GOP structure and bit rates

Date Use caseCapture Latency

(ms)

Encode Latency

(ms)

Decode Latency

(ms)

DisplayLatency

(ms)

Total(ms)

2017.1 Normal latency 16.6 16.6 83.33 16.6 133

2017.3 Reduced latency 16.6 16.6 50 16.6 99.8

2018.1 Low latency VCU Only 16.6 4 22.6 16.6 59.8

2019.1* Low Latency Full Pipeline* 5 5 12.5 12.5 35

Video Processing Stages Adding Latency

*Numbers and plans subject to change

PS DDR Memory (Encoder, Capture,

and Display)

PL DDR Memory (Decoder)

Live Video Source

HDMI / SDI / MIPI /

Ethernet / DP

Processing Pipeline (VPSS)

Frame Buffer DMA

Encode

Ethernet

Video Sink

PS DDR Memory (Encoder, Capture,

and Display)

PL DDR Memory (Decoder)

HDMI / SDI / MIPI /

Ethernet / DP

Video Mixer

Frame Buffer DMA

Decode

Ethernet

1 Camera

2 Capture

6 Display

3 Encode

5 Decode4

IP Network

7 Monitor

2 3 5 6


MPSoC Development Tools & Target Reference Designs

˃ Demonstrates capabilities of EV device’s VCU integrated block

˃ HDMI Live video capture to evaluate the compression quality

˃ Video encode and decode

˃ Encoder parameter configuration

Vivado® HLx– Targets the FPGA fabric– Primarily used by FPGA designers– Supports novice to expert level flows– Used to design hardware and

generate hardware definitions

SDSoC™– Targets the APU and FPGA logic– Builds on world-class Vivado tool and

Eclipse– Primarily used by software designers

and system architects– Allows rapid SoC-level

hardware/software iterations

PetaLinux– Yocto-based embedded software

development– Integrates several other tools/scripts

to provide unified development flow– Primarily used by software

developers

VCU TRD (Target Reference Design)

MPSoC Base TRD


Lossless Codec IP from Xilinx PartnersJPEG 2000 JPEG-XS VC2 ProRes

Standard Body ISO 15444-1 ISO 21122 SMPTE 2042-1 AppleMaximum Resolution/rate SD, HD, 3840x2160p60 (4KUHD), 4096x2160p30 (DCI4K), 7680x4320p60 (8KUHD)Sample Bit Depth 8, 10, 12, 14, 16 bpc 10, 12 bpcChroma Format 4:2:2, 4:4:4, 4:2:2:4, 4:4:4:4 4:2:2, 4:4:4, 4:4:4:4Color Space RGB, YUV, XYZ, YCbCr RGB, YCbCrVideo Format Interlaced or ProgressiveLatency Encode: 1~2 frames

Decode: 0.5~1 frames Near zero: a few lines Near zero: a few lines Encode: 1+ framesDecode: 1+ frames

External Memory DDR3/4 N/A N/A DDR3/4Configurable Bit Rate or Compression Ratio CBR: 0.25 ~ 8 Gbps

Lossless: variable & unlimitedNear lossless: 2:1Visually lossless: 10:1 Visually lossless: 4:1

1080p50: 245 Mbps3840x2160p60: 1 Gbps7680x4320p60: 4 Gbps

Application Streaming & Archiving Archiving & NLEFPGA Fitting 2K HD: 7K70T

4K UHD: 7K325T8K UHD: KU060

2K HD: Aritx/Spartan-74K UHD: Aritx/Spartan-78K UHD: K7 / KU / KUP

2K HD: Aritx/Spartan-74K UHD: Aritx/Spartan-78K UHD: K7 / KU / KUP

2K HD: Kintex-74K UHD: KU8K UHD: KUP

Vendor


SMPTE standards & protocolsIS06/NMOS: media network discovery and enumeration

SMPTE 2110: packetization & traffic shaping

STMPTE 2022-5: SDI packetization & FEC

SMPTE 2022-7: Dual stream redundancy

SMPTE 2059: Multi-endpoint synchronization

IEEE 1588: Precision time stamps

……

Improved Network Protocols

A pile of broadcast standards developed over 10+ years


Partner Video over IP PortfolioTr

ansp

ort SMPTE 2022-1/5

Forward Error Correction (FEC) for 1G and 10G networks respectivelySMPTE 2022-2/6Transport protocol for SDI video on IP networks (1G and 10G)SMPTE 2022-7Seamless protection switching to allow network redundancySMPTE 2110Transport protocol for separate audio, video and ANC dataSony NMI Sony’s proprietary transport protocol

Tim

ing SMPTE 2059-1/2

Precision Time Protocol to allow nano-second level synchronization of media streams

Soft

war

e NMOS/IS-04, IS-05, IS-06 Network Media Open Specifications Control SW used to configure and manage media networksSony Live System Manager (LSM) Sony’s control SW

Com

pres

sion

Integrated Broadcast CodecsintoPIX TICO JPEG XS (pending)intoPIX JPEG2000intoPIX JPEG2000 ULL Sony LLVCAll lightweight compression codecs for transporting 4K over 10GbE

http://www.macnicatech.com/

http://www.macnicatech.com/


Conceptual FPGA design for Professional Streaming

Processing System

Programmable Logic

DDR Controller

PlatformManagement

UnitSecurity System

Control

Application Processing UnitQuad-Core

ARM® Cortex™-A53

Graphics Processing UnitARM Mali-400 MP2

High-Speed Connectivity

General ConnectivityReal-Time Processing UnitDual-Core

ARM Cortex-R5

Video Processing

Multi-Channel VCUH.264/H.265

“Lossless ” Video CodecProRes, DNxHD, TICO, JPEG-2000, JPEG-XS

DDR Memory

DIsplayPortUHD-SDIHDMIDIsplayPortUHD-SDI

DDR Memory

Cable DriversCable EqualizersRetimers

Ethernet PHYsEthernet MACEthernet MAC

USB3.0SATA3.1

GbESD/eMMC

AppsMedia Software Stack

Audio ProcessingSystem Management

OS

Capture Process Compress Deliver

Package(Record)


Ultra low latency, lossless & synchronous transport

HDMI RX

TICO Encoder

TICO Packetizer

AES67 Packetizer

Audio Extract

ST2110Scheduler

ST2059 Master L2+L3

FramerEthernet

MACEthernet

PHY

IEEE1588Master

L2+L3Framer

EthernetMAC

EthernetPHY

EthernetMAC

EthernetPHY

EthernetMAC

EthernetPHY

L2+L3Deframer

L2+L3Deframer

Packet Filter + Host Queue

Packet Filter + Host Queue

ST2022-7PacketBuffer

TICO Depacketizer

AES67 Depacketizer

ST2059Slave

TICO Decoder

VideoTiming

HDMI TX

Conceptual drawing – A 7z030 based reference design coming soon from Macnica

1/10 GbE

1/10 GbE

1/10 GbE

1/10 GbE

Leveraging a pile of SMPTE standards developed over 10+ years

Ultra low latency codec

Ultra low latency codec

Dual channel redundancy

Fully synchronous timing

Fully synchronous timing

Dual channel redundancy


Flexible Capture with Any-to-Any Connectivity

Sense + Acquire


Deliver (Stream)

Compress + Package

Capture + Process

˃ Xilinx supports UHD-4K interfaces with transceivers and IP cores NOWHDMI 1.4/2.0/2.0a, DisplayPort 1.2a, 12/6/3G-SDI, 10/25/100GbEST2110 support with partner IP

˃ Short Xilinx roadmap to UHD-8K supportDisplayPort 1.3/1.4HDMI 2.1

˃ V4L2 device driver support

˃ Flexibility to use only the I/O you need

https://www.xilinx.com/products/intellectual-property/hdmi.html

https://www.xilinx.com/products/intellectual-property/ef-di-displayport.html

https://www.xilinx.com/products/intellectual-property/uhd-serial-digital-interface.html

https://www.xilinx.com/products/technology/ethernet.html


Sense + Acquire


Deliver (Stream)

Compress + Package

Capture + Process

Custom Video Processing Engines for Your Application

˃ Video processing involves transforming the uncompressed data in some way

Examples: Color Space Conversion, De-interlacing, Scaling, MixingVideo processing ASSP roadmaps are weak for UHD-4K and beyond

˃ FPGA logic-based processing allows highly customized processing pipelines before the compression stage

˃ Real-time video processing IPXilinx VPSS (Video Processing Subsystem) and VMIX (Video Mixer) ‒ Supports UHD4k60p

https://www.xilinx.com/products/intellectual-property/video-processing-subsystem.html

https://www.xilinx.com/products/intellectual-property/ef-di-vid-mix.html


Sense + Acquire


Deliver (Stream)

Compress + Package

Capture + Process

Integrated VCU for Low-Power Multi-Channel Compression

˃ VCU: Integrated Multi-Channel Encode/Decode

˃ EncodeH.264 and/or H.265, up to UHD-4kp60Configurable GOP, Rate Control (CBR, VBR, Constant QP)

˃ DecodeH.264 and/or H.265, up to UHD-4kp60

˃ Power EstimateMany, many variables involved; exact figure depends on your use caseApproximate power for encode core: ‒ HEVC UHD4k 4:2:2 10-bit: ~0.9W (30 fps) and ~1.8W (60 fps)

Please use XPE (Xilinx Power Estimate) tool for detailed estimate

• ProRes• DNxHD• TICO• VC-2

• JPEG2000• And more…

˃ Visually lossless

˃ Fabric-based solutions include:

https://www.xilinx.com/products/technology/power/xpe.html


Sense + Acquire


Deliver (Stream)

Compress + Package

Capture + Process

Flexible Encapsulation, Delivery, and Storage Options

˃ HardwareEthernet: Embedded quad-gigabit or soft IP options (10GbE)Integrated interfaces: SATA 3.1, USB 3.0, SD/eMMCLogic can be used for additional storage interfaces if needed

˃ Ethernet: Streaming OutputVideo bitstream is encapsulated in many formats using the APUHTTP (HLS, HDS, MSS, DASH), UDP (RTP, RTSP, RTCP)

˃ Package and Record: File OutputVideo bitstream is encapsulated in any file format requiredMP4, MOV, AVI, MXF, etc…


Improved Home Content and Game Sharing

HDMI IP

1/10/25 GbE / WIFI Switcher/Router1 GbE

Office

HDMI IP

HDM

I

HDM

I

HDMI IP

Game Console

STB

HDMIHDM

I HDMI

HDMI

HDMI IP

HDMI

HDMI IPHDMI

HDMI

1 GbE

1 GbE1 GbE

1 GbE

1 GbE

Broadband Modem

RemoteContent

Broadband

1 GbE


The Best Offer for Capture, Compress and Deliver

˃ Capture, Compress, and DeliverZynq Ultra Scale+ MPSoC’s EV device provides a solution for multiple capture interfaces, compression options, and reliable deliveryFPGA logic integrated with SoC provides high degree of flexibility and differentiation

˃ Solves the Key Requirements for Streaming & RecordingCapture: HDMI, DisplayPort, SDI, and moreProcess: Logic-based programmable video processing pipelineCompress: Embedded multi-channel multi-codec compression and decompression built-inPackage: Software-based packaging of streams in any transport or file formatDeliver: Ethernet streaming (soft or hard IP)Record: Multiple built-in interfaces for storage media; augment with additional interfaces if needed

˃ Open-Source Media Framework

˃ Industry-leading Tools (Vivado, PetaLinux, SDSoC) & TRD


For More Information:

Pro AV & Broadcast Solutions: xilinx.com/broadcast Zynq UltraScale+ MPSoC: xilinx.com/products/silicon-devices/soc/zynq-ultrascale-mpsoc.htmlMPSoC Reference Designs: wiki.xilinx.com/Technical+Articles#TRDXilinx Software Zone: xilinx.com/products/design-tools/software-zone.html

Contact your local Xilinx sales representative

https://www.xilinx.com/applications/broadcast.html

http://www.xilinx.com/products/silicon-devices/soc/zynq-ultrascale-mpsoc.html

http://www.wiki.xilinx.com/Technical+Articles#TRD

https://www.xilinx.com/products/design-tools/software-zone.html


Adaptable.Intelligent.

赛灵思技术日XILINX TECHNOLOGY DAY

Documents

Streaming Media & Entertainment