UltraGrid: Low-LatencyHigh-QualityVideo ...€¦ · 1/53 Introduction UltraGridFeatures UltraGridCommunity What’sNew? Tutorial/Demo Roadmap UltraGrid: Low-LatencyHigh-QualityVideo

1/53

Introduction UltraGrid Features UltraGrid Community What’s New? Tutorial/Demo Roadmap

UltraGrid: Low-Latency High-Quality VideoTransmissions on Commodity Hardware

Petr HolubCESNET z.s.p.o., Prague/Brno, Czech Republic

<[email protected]>

SITOLA

Internet2 Global SummitDenver, 2014–04–07

http://www.sitola.cz/

http://www.ces.net/

mailto:[email protected]


http://www.ces.net/

2/53


Overview

IntroductionExamples of Applications

UltraGrid Features

UltraGrid Community

What’s New?

Tutorial/Demo

Current & Future Development


http://www.ces.net/

3/53


What is UltraGrid?

● A�ordable platform for high-quality interactive videotransmissions using commodity hardware◾ high-resolution video: HD, 4K, 8K◾ as low latency as possible on commodity hardware◾ commodity equipment equipment

◆ Linux, Mac, Windows◆ common GPUs and video capture cards

◾ open-source software, dual BSD/GPL license◆ UltraGrid itself is BSD-licensed, may become “GPL infected” (e.g.,

x264)◾ picks up where common videoconferencing ends

◆ 10Mbps – 10Gbps (or more)


http://www.ces.net/

4/53


Who Is UltraGrid For?

● “Power users”◾ scienti�c visualizations◾ medicine: X-ray imagery, cardiology, pathology◾ education: remote lecturing◾ arts, cinematography, broadcasting◾ collaborative environments: multi-point operation◾ many others


http://www.ces.net/

5/53


Who Is UltraGrid For?

● Researchers◾ experimental validation of your research results◾ Holub et al.: real-time image processing algorithms◾ Perkins et al.: network protocols, congestion control,. . .◾ Holub & Rudová et al.: data distribution models & scheduling◾ Renambot et al.: image compression & large scale visualizationsystems

◾ Hutanu et al.: distributed visualization systems


http://www.ces.net/

6/53


Applications of UltraGridRemote education

CCT LSU CSC7600 – Introduction to HPC, T. Sterling


http://www.ces.net/

7/53


Applications of UltraGridPostproduction in cinematography

Detached BaseLight consoles at CinePost (Barrandov, CZ)


http://www.ces.net/

8/53


History of UltraGrid

● History of Development◾ 2002–2004: ISI EAST (720p)◾ 2005–2012: CESNET (→ 1080i)◾ 2006–2008: forks by KISTI (GUI, AJA KONA) and i2cat (SAGE)◾ 2013: i2cat starts contributes to common codebase maintainedby CESNET

● Somemilestones◾ 2002: 720p◾ 2005: 1080i, multipoint◾ 2007: CPU compressions, self-organization, optical multicast◾ 2008: 2K/4K◾ 2011: GPU compressions◾ 2012: 8K


http://www.ces.net/

9/53


World Firsts. . .

● 2005 – Multi-Point Uncompressed HD

● iGrid 2005 – two independentn-way demos◾ UltraGrid: using packet re�ectors◾ iHDTV: multicast


http://www.ces.net/

10/53


World Firsts. . .

● 2007 – Self-Organizing Multi-PointUncompressed/Compressed HD◾ with CoUniverse◾ self-organizing multi-pointdistribution setup withuncompressed/DXT1compression switchingbased on availablebandwidth


http://www.ces.net/

http://couniverse.sitola.cz/

11/53


World Firsts. . .

● 2011 – GPU-JPEG Transatlantic 4K

◾ CineGrid Workshop, December 2011◾ real-time movie post-production review/approval process◾ playback on a machine worth $1.000 ($500 PC + $500 NVIDIA580GTX)


http://www.ces.net/

12/53


World Firsts. . .

● 2012 – GPU-JPEG Transatlantic Multi-Point 8K

◾ from pre-rendered sources◾ JPEG→ DXT5-YCoCg ona single machine

◾ useful also as 16× HD(multi-camera setups)


http://www.ces.net/

13/53


World Firsts. . .● 2013 – UltraGrid + CoUniverse + inter-domain bandwidth ondemand◾ USA + Japan + Czech Rep using AutoGOLE with NSI◾ multipoint using self-organizing re�ectors


http://www.ces.net/

14/53


Features of UltraGrid

● Supported video formats◾ HD, 2K◾ 4K – tiled or native◾ 8K – experimental, problems with available sources◾ (almost) arbitrary video resolution◾ multichannel video (e.g., 3D HD, 4K)

● Uncompressed vs. compressed◾ only interested in low-latency compression◾ GLSL-accelerated DXT1, DXT5-YCoCg◾ CUDA-accelerated JPEG, DXT5-YCoCg◾ CPU-based low-latency H.264 via x264 & libavcodec

◆ that’s when UltraGrid becomes GPL


http://www.ces.net/

15/53


Features of UltraGrid● Video input

◾ capture cards: HD/3G/6G-SDI, SDI, HDMI (incl. stereoscopicHDMI 1.4a), analog HD and SD

◆ vendor SDKs: Blackmagic, DeltaCast, BlueFish444◆ vendor-independent: Video4Linux2, QuickTime, DirectShow

◾ screen capture input◆ beware of cursor – may be lost, depending on the API

◾ testcard◾ �le input

● Video output◾ playback cards: HD/3G/6G-SDI, SDI, HDMI (incl. stereoscopicHDMI 1.4a), analog HD and SD

◆ best for displaying interlaced video

◾ computer screen output (OpenGL, SDL)◾ SAGE output◾ �le output◾ specialized display �lters


http://www.ces.net/

16/53



● Video processing◾ deinterlacer◾ software video mixer with overlay support

◆ probably the only a�ordable real-time mixer with scalingsupport for 4K right now

◾ logo overlay


http://www.ces.net/

17/53



● Supported audio formats◾ various sample rates◾ multi-channel

● Uncompressed vs. compressed◾ uncompressed by default◾ OPUS codec

● Synchronization of audio with video◾ depends on the source, if it is synchronized◾ synchronized when using HD-SDI and HDMI as audio source◾ audio is transmitted with as low latency as possible when usingstandalone soundcard


http://www.ces.net/

18/53



● I/O◾ capture/playback cards: HD-SDI, SDI, HDMI, analog HD and SD

◆ manufacturers’ SDKs, Video4Linux2, QuickTime

◾ screen capture input◾ computer screen output (OpenGL, SDL)◾ SAGE output◾ specialized display �lters◾ stereoscopic HDMI 1.4a◾ testcard

● Audio◾ balanced, unbalanced, HD-SDI, HDMI◾ various system interfaces including JACK

● Full-duplex operation

● Simple GUI

Line-interlaced stereoscopic video


http://www.ces.net/

19/53



● Network transmission format – extended RTP◾ IP/UDP/RTP is used as the basic format

◆ backward compatible with RTP tools, e.g., for monitoring◾ extended application headers for additional �exibility

◆ http://www.cesnet.cz/wp-content/uploads/2013/01/

ultragrid-4k.pdf

◆ extended packet numbering◆ �exible speci�cation of media parameters, format, compression,

etc.◆ notion of “bu�er position” – does not depend on uncompressed

structure as RFC 4175


http://www.ces.net/

http://www.cesnet.cz/wp-content/uploads/2013/01/ultragrid-4k.pdf

http://www.cesnet.cz/wp-content/uploads/2013/01/ultragrid-4k.pdf

20/53


Range of ApplicabilityBandwidth

● HD (1080p @ 30p)◾ H.264: 10–50Mbps◾ M-JPEG: 50–200Mbps◾ uncompressed: 1–2.2 Gbps

● 4K (2160p @ 30p)◾ H.264: 30–100Mbps◾ M-JPEG: 150–600Mbps◾ uncompressed: 4–9Gbps

● 8K (4320p)◾ M-JPEG: 500Mbps–2Gbps◾ (uncompressed: >10Gbps)

● 1–3× for Forward Error Correction (FEC)


http://www.ces.net/

21/53


Range of ApplicabilityLantecy

● End-to-end latency (capture to playback) in local network◾ <150ms for interactivity: ITU-T rec G.114

● Video◾ depends on capture/playback hardware: 1.75–5.5 frames(58–183ms)

◾ e.g., ≈ 2 frames:BlackMagic Decklink HD Extreme→ OpenGL on MacOS x

● Audio◾ video embedded: synchronous◾ independent source/playback: 20ms or less


http://www.ces.net/

22/53


GPU-Accelerated Compression

● Available compression schemes◾ DXT1: CPU-based (FastDXT library from EVL)◾ DXT1, DXT5: OpenGL Shader Language (GLSL) based◾ JPEG: NVidia CUDA based◾ DXT5: NVidia CUDA based (for 8K)

SAGE display with various compressions


http://www.ces.net/

23/53



● Fine-grained parallelization of JPEG◾ per-row/column DCT/IDCT◾ per pixel RLE◾ per pixel Hu�man◾ parallel stream compacting◾ parallel decompression using restart intervals◾ use of auxiliary indexes for more e�cient parsing

● Available also as BSD-licensed open-source library:http://gpujpeg.sf.net/


http://www.ces.net/

http://gpujpeg.sf.net/

24/53



● Performance numbers (including transfer to/from GPU)◾ DXT1 GLSL: 798Mpix/s (NVidia 580GTX), 593Mpix/s (ATI 6990)◾ DXT5 GLSL: 349Mpix/s (NVidia 580GTX), 305Mpix/s (ATI 6990)◾ JPEG CUDA: up to 1.580Mpix/s = 4.740MB/s (NVidia 580GTX,4:4:4, Q=60)

◾ DXT5 CUDA: ≥1.580Mpix/s (NVidia 580GTX)


http://www.ces.net/

25/53



● JPEG performance

20 40 60 80 1000

3

6

9

12

Quality

Dura

tion

[ms]

(a) Encoder performance (GPU only)

20 40 60 80 1000

3

6

9

12

Quality

Dura

tion

[ms]

1080p1080p 4:2:02160p2160p 4:2:0

(b) Decoder performance (GPU only)

20 40 60 80 1000

5

10

15

20

Quality

Dura

tion

[ms]

(c) Encoder performance (both CPU and GPU)

20 40 60 80 1000

5

10

15

20

Quality

Dura

tion

[ms]

(d) Decoder performance (both CPU and GPU)

20 40 60 80 100

25

30

35

40

45

50

Quality

PSN

R[d

B]

(e) Quality of reconstruction

20 40 60 80 100

100

200

400

800

1,600

3,200

6,400

12,800

Quality

Imag

esi

ze[k

B]

(f) Compressed image size

Figure 6: JPEG coder performance and quality loss in dependence on quality and mode settings. Average over all image typeshas been used for the evaluation.

13


http://www.ces.net/

26/53


CPU-Based H.264

● X264 library provides low-latency H.264 mode◾ no B-frames are used◾ distributed I-frames to avoid bandwidth spikes

● Practical usability◾ HD @ 30p: 4 core Core i7 system◾ 4K @ 30p: 12 core Intel Xeon system

● X264 library is GPL or commercially licensed◾ UltraGrid becomes GPL when linked with it◾ commercial license can be obtained for commercial projects


http://www.ces.net/

27/53


Latency Impact of Compression

● Uncompressed for DeckLink HD→ DeltaCast 3G◾ 2.5 frames (83ms)

● Impact of compressions◾ 2.5 frames (+<16.7ms) for CUDA JPEG◾ 3.5 frames (+33.3ms) for GLSL DXT1/5◾ ≈ 5 frames (+83.3ms) for H.264


http://www.ces.net/

28/53


Forward Error Correction (FEC)

● LDGM◾ CPU and GPU implementations◾ CPU (SSE optimized) is used because of CPU↔GPUtransmissions overhead

◾ packet loss up to 10% can be mitigated with reasonableoverhead

◾ can make JPEG survive up to 25% packet loss● Simple method: shifted multiplication

◾ used for audio by default


http://www.ces.net/

29/53


User-Empowered Multi-Point Distribution

● UltraGrid supports multicast, but. . .◾ how available/dependable it is?

● UDP packet re�ectors◾ controlled by the user◾ lower e�ciency◾ possible per-user processing: transcoding, security,. . .

● Transcoding re�ectors◾ per user processing, individual quality adjustments

● Self-organization of the network◾ research problem: scheduling streams with bitratescomparable to capacity of links

◾ CoUniverse framework (http://couniverse.sitola.cz)


http://www.ces.net/

http://couniverse.sitola.cz

30/53


Users Worldwide

SourceForge stats

● Distribution:◾ source, binaries (http://www.ultragrid.cz/, SourceForge)◾ embedded in SAGE (http://www.sagecommons.org/)

● Installations around the world:◾ Czech Republic (universities and university hospitals), USA(UCSD, UMich, UIC, Internet2, NLM/NIH, NorthwesternU, . . . ),Spain (i2cat, UPM), Portugal (FCCN), Netherlands (SARA), Poland(PSNC), Korea (KISTI), Russia, Germany (H-BRS), Japan (AIST),. . .


http://www.ces.net/

http://www.ultragrid.cz/

http://www.sagecommons.org/

31/53


Open-Source Development Model

● Why?◾ because it allows to implement your research

● Contributors◾ current: CESNET, Masaryk University, i2cat, EVL UIC◾ past: ISI EAST, CCT LSU, SARA

● Benevolent dictators for life◾ we review before accepting/committing◾ to maintain and improve code quality◾ we welcome your contributions!


http://www.ces.net/

32/53


What’s New? – Release 1.2

● MSWindows support● New hardware support

◾ capture cards: DELTACAST, BlueFish444

● File-based I/O

● Transcoding re�ector● Integration of 2-camera GColl

◾ group-to-group communication with partial gaze awareness● Software video mixer

◾ up to 4K video in real-time

● Image anonymization


http://www.ces.net/

33/53


Overview


UltraGrid Features

UltraGrid Community

What’s New?

Tutorial/Demo



http://www.ces.net/

34/53


Practical Notes on 4K

● Interface diversity◾ single link: HDMI 1.4(a) (limited to 30 fps), DisplaPort 1.2 (nocapture cards), HDMI 2.0 (approved but no products), 6G-SDI(only BlackMagic)

◾ multi link: 4× HD-SDI, 4× 3G-SDI, 8× HD-SDI, 4× HDMI◆ synchronization issues when using multiple cards (beware:

Decklink Quad)

● Need for higher frame rates (≥ 50p)◾ strobing artifacts are even more visible than in HD


http://www.ces.net/

35/53


Recommended Hardware

● Audio (!)◾ echo canceling mic◾ great experiences with ClearOne Chat 150◾ good experiences with ClearOne Chat 50/60

◆ we have once found USB-level con�ict with keyboard/mouse

◾ problems with various older pieces (e.g., Polycom C100S)● Video

◾ DeckLink cards for basic applications: HDMI & HD/3G/6G-SDIcapture & playback

◾ DELTACAST for more advance applications◆ capture of raw SDI streams◆ HDMI capture of computer resolutions, up to 4K◆ multi-SDI capture/playback


http://www.ces.net/

36/53


Tutorial: Simple Use● Getting help$ ./bin/uv −t helpAvailable capture devices:

aggregateimportswmixscreendecklinkdeltacastdeltacast−dvitestcardv4l2none

$ ./bin/uv −t decklink:helproot@hd5:~/ultragrid# bin/uv −t decklink:help...

Display initialized−none

Decklink options:−t decklink[:<device_index(indices)>[:<mode>:<colorspace >[:3D][:timecode][:connection=<input >]]

(You can omit device index, mode and color space provided that your cards supports format autodetection.)

device: 0.) Intensity Pro

display modes:0.) NTSC 720 x 486 29.97 FPS1.) NTSC 23.98 720 x 486 23.98 FPS2.) PAL 720 x 576 25.00 FPS...


http://www.ces.net/

37/53


Tutorial: Simple Use

● Linux transmit/receive (duplex)$ ./bin/uv −t decklink:2:24 −d gl −c libavcodec \−s alsa:dmix:CARD=C150 −r alsa:plughw:CARD=C150 129.11.42.100

● Linux transmit/receive (duplex)./bin/uv −t decklink:1 −c libavcodec:codec=H.264:bitrate=10M \−s embedded −−audio−codec OPUS −P 5010 −l 60M 147.251.240.74

● Windows transmituv.exe −t dshow:1 −f ldgm −c libavcodec:codec=H.264 \−s portaudio:0 −r portaudio:4 147.251.54.247


http://www.ces.net/

38/53


Tutorial: Compression & FEC

● video: JPEG−c JPEG:90

● video: H.264−c libavcodec:codec=H.264:bitrate=10M

● video: DXT1 & DXT5−c RTDXT:DXT1−c RTDXT:DXT5

● audio: OPUS−−audio−codec OPUS

● LDGM-based FEC−f LDGM


http://www.ces.net/

39/53


Tutorial: Transcoding Multipoint

● Simple multipoint$ ./bin/hd−rum 8M 5004 receiver1 receiver2

● Transcoding re�ector is part of UltraGrid$ ./bin/hd−rum−transcode [global_opts] buffer_size port [host1_options]host1 [[host2_options] host2] ...

where global_opts may be:−−control−port <port_number > − control port to connect to

and hostX_options may be:−c <compression > − compression−m <mtu> − MTU size. Will be used only with compression.−l <limiting_bitrate > − bitrate to be shaped to−f <fec> − FEC that will be used for transmission.

$ ./bin/hd−rum−transcode 8M 5004 −c libavcodec:codec=H.264 \−m 9000 receiver1 receiver2 −c jpeg receiver3


http://www.ces.net/

40/53


Tutorial: Software Videomix

● Content of ~/.ug−swmix.rc3840:2160:25:UYVY

2 import:gopro−record 0 390 2490 1400deltacast−dvi:board=0:channel=0:codec=UYVY 2490 0 1350 1080

4 deltacast−dvi:board=0:channel=1:codec=UYVY 2490 1080 1350 1080

● Running UltraGrid$ ./bin/uv −t swmix:file −t import:gopro−record −t \deltacast−dvi:board=0:channel=0:codec=UYVY −t \deltacast−dvi:board=0:channel=0:codec=UYVY <receiver >


http://www.ces.net/

41/53


Tutorial: Software Videomix

● Aliases for device names

● Content of ~/.ug.rcalias ekg deltacast−dvi:board=0:channel=1

2 capture−filter ekg blank:0:1000:300:24

● Simple use:$ ./bin/ug −t ekg 192.168.1.10


http://www.ces.net/

42/53


Tutorial: File-Based I/O

● Storing to �le(s)$ ./bin/uv −t deltacast −−record target_dir

● Reading from �le(s)$ ./bin/uv −−capture−filter logo \−−playback /mnt/tchor/skladacka/export.mjpg/:loop \−l 600M 162.244.108.176


http://www.ces.net/

43/53


Docs and Support

● Other tips for running UltraGrid can be found @https:

//www.sitola.cz/igrid/index.php/Running_UltraGrid

● More information @http://www.ultragrid.cz/

● Ask us @<[email protected]>


http://www.ces.net/

https://www.sitola.cz/igrid/index.php/Running_UltraGrid

https://www.sitola.cz/igrid/index.php/Running_UltraGrid

http://www.ultragrid.cz/


44/53


Multi-Point I2GS 2014 DemoDemo requirements

● Show 4K angiology imagery

● Have EVL discuss their experiences with UltraGrid in SAGEenvironments

● Components◾ permanent audio:

◆ Denver↔ EVL◆ Denver↔ Sitola

◾ permanent video: Denver↔ EVL, Denver→ Sitola◆ Denver↔ EVL◆ Denver→ Sitola

◾ switched video: Sitola→ Denver – HD, 4K◆ Sitola→ Denver – HD, 4K


http://www.ces.net/

45/53


Multi-Point I2GS 2014 DemoDemo scheme


http://www.ces.net/

46/53


Multi-Point I2GS 2014 Demo

● Denver◾ UltraGrid instances

◆ bin/uv −t v4l2:/dev/video0:YUYV:640:480:1/30 −c JPEG:80 −d \

gl −s alsa:default:CARD=C150 −r alsa:dmix:CARD=C150,DEV=0 \

−P 8004:5004:8006:5006 localhost

◆ bin/uv −P 7004 −d gl hd5.fi.muni.cz

◆ bin/uv −P 6004:6004:6006:6006 −r alsa:dmix:CARD=C150,DEV=0 \

hd5.fi.muni.cz

◾ Re�ector instances◆ hd−rum−multi/hd−rum 8M 5004 mesaverde.evl.uic.edu hd5.fi.\

muni.cz

◆ hd−rum−multi/hd−rum 8M 5006 mesaverde.evl.uic.edu hd5.fi.\

muni.cz


http://www.ces.net/

47/53


Multi-Point I2GS 2014 Demo

● Sitola◾ UltraGrid audio sender/receiver

◆ bin/uv −s alsa:plughw:CARD=C150 −r alsa:plughw:CARD=C150 −d\

gl −P 5004:5004:5006:6006 162.244.108.176

◾ UltraGrid HD sender◆ bin/uv −t decklink:0:10 −c JPEG −f ldgm −l 600M −P 7004 \

162.244.108.176

◾ UltraGrid 4K sender◆ bin/uv −−playback /mnt/tchor/skladacka/export.mjpg −f ldgm\

:1000:1000 −l 600M −P 7004 162.244.108.176


http://www.ces.net/

48/53


Overview


UltraGrid Features

UltraGrid Community

What’s New?

Tutorial/Demo



http://www.ces.net/

49/53


Ongoing Development

● GPU-based LDGM FEC◾ CUDA-based◾ works up to uncompressed 4K◾ extended performance and con�guration pro�ling◾ to be submitted into Future Generation Computer Systemsjournal

● New implementation of video mixer – with i2cat◾ to avoid OpenGL/GLSL readback performance problems◾ based on OpenCV with OpenCL acceleration


http://www.ces.net/

50/53


Ongoing Development

● CoUniverse◾ self-organization of complex multi-point scenarios incl.transcoding

◾ research work that slowly moves into production● Reimplementation of the UltraGrid core

◾ increased modularity and �exibility◾ better maintainability – C++


http://www.ces.net/

51/53


Future Development Plans

● Real-time image analysis◾ (semi)automated anonymization◾ domain-speci�c composition of various images

● OpenCL implementation of CUDA-based algorithms◾ we will start with LDGM◾ complex problem for complex algorithms

● Port of UltraGrid to mobile devices◾ by i2cat◾ starting with Android

● Robust software-based echo cancellation◾ problems with clock drift on sound cards


http://www.ces.net/

52/53


Selected Papers

[1] HOLUB, Petr - MATYSKA, Luděk - LIŠKA, Miloš - HEJTMÁNEK, Lukáš - DENEMARK, Jiří -REBOK, Tomáš - HUTANU, Andrei - PARUCHURI, Ravi - RADIL, Jan - HLADKÁ, Eva.High-de�nition multimedia for multiparty low-latency interactive communication.Future Generation Computer Systems, Amsterdam, The Netherlands, Elsevier Science, TheNederlands. ISSN 0167-739X, 2006, vol. 22, no. 8, pp. 856–861.

[2] HOLUB, Petr - ŠROM, Martin - PULEC, Martin - MATELA, Jiří - JIRMAN, Martin.GPU-accelerated DXT and JPEG compression schemes for low-latency networktransmissions of HD, 2K, and 4K video. Future Generation Computer Systems,Amsterdam, The Netherlands, Elsevier Science, The Nederlands. ISSN 0167-739X, 2013,vol. 29, no. 8, 1991–2006-16 pp.


http://www.ces.net/

53/53


Thank you for your attention!

<[email protected]>

<[email protected]>

This work is supported by LM2010005 project.


http://www.ces.net/



Documents

UltraGrid: Low-LatencyHigh-QualityVideo ...€¦ · 1/53 Introduction UltraGridFeatures UltraGridCommunity What’sNew? Tutorial/Demo Roadmap UltraGrid: Low-LatencyHigh-QualityVideo