43
1 © 2015 Gachon University. All rights reserved. Next Generation Video Streaming System using Scalable High Efficiency Video Coding (SHVC) Aug. 24. 2015 KIISE SWCC 2015 Eun-Seok Ryu [email protected] Department of Computer Engineering, School of Information Technology Gachon University

Next Generation Video Streaming System using Scalable High

  • Upload
    others

  • View
    2

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Next Generation Video Streaming System using Scalable High

1 © 2015 Gachon University. All rights reserved.

Next Generation Video Streaming System using

Scalable High Efficiency Video Coding (SHVC)

Aug. 24. 2015

KIISE SWCC 2015

Eun-Seok Ryu [email protected]

Department of Computer Engineering,

School of Information Technology

Gachon University

Page 2: Next Generation Video Streaming System using Scalable High

2 © 2015 Gachon University. All rights reserved.

Contents

+ Short Bio / 과거에 상상하던 멀티미디어 기술

1. Next Generation Video Coding Technologies• High Efficiency Video Coding (HEVC)

• Scalable HEVC (SHVC)

• Multi-view Video Coding (MVC) / 3D

2. Error Resilient Video Streaming• Unequal Error Protection using Raptor Codes

• Error Concealment using EC modes

• Selective Video Streaming

3. Video Streaming Applications• Video Streaming over Multiple Network Interfaces

• Power Aware HEVC Streaming with DASH

• Three-Screen TV using SVC Platform

• Telehealth and Haptic Video Streaming for the Individuals with Visual Impairments

Page 3: Next Generation Video Streaming System using Scalable High

3 © 2015 Gachon University. All rights reserved.

가천대학교 IT대학 컴퓨터공학과 조교수 류은석

• 주요경력 (연구분야: 멀티미디어 통신 시스템)

• 2015.03-현재 : 가천대학교 IT대학 컴퓨터공학과 (성남) 조교수

• 2014.03-2015.02: 삼성전자 (수원) 수석연구원 / 파트장•임베디드 비디오 처리 및 통신 연구/특허개발

• 2011.01-2014.02: InterDigital Labs (San Diego), Staff Engineer•JCT-VC HEVC/SHVC 비디오 표준화 연구

• 2008.09-2010.12: Georgia Inst. Tech. (Atlanta), Postdoctoral Research Fellow•Three-Screen TV를 위한 홈게이트웨이 연구

• 2008.03-2008.08: 고려대학교 정보통신기술연구소 (서울), 연구교수•Scalable Video Coding 기술을 이용한 다중망 비디오 전송기술 연구

• 학력

• 고려대학교 컴퓨터학과 이학사(1999) / 이학석사(2001) / 이학박사(2008) (지도교수유혁)

• 연구실적

• 논문: 19 Journal / 17 International Conf. / 22Korean Domestic Conf.

• 국제표준: 14 Video Standard Contribution

• 국제/국내 특허: 12+ US Patent Applications / 17 Korea Patent Applications

Page 4: Next Generation Video Streaming System using Scalable High

4 © 2015 Gachon University. All rights reserved.

과거에 상상하던 미래의 멀티미디어 기술

1912, Hugo Gernsback's 124C 41 A Romance of the Year 2660

1925, Telemedicine

1954, Teledoctor

1963, TV Glasses

Radio Doctor (1924)

Page 5: Next Generation Video Streaming System using Scalable High

5 © 2015 Gachon University. All rights reserved.

Part 1Next Generation Video Coding Technologies- High Efficiency Video Coding (HEVC)

- Scalable HEVC (SHVC)

- Multi-view Video Coding (MVC) / 3D

Page 6: Next Generation Video Streaming System using Scalable High

6 © 2015 Gachon University. All rights reserved.

The History of Video Coding Standards (1)

201420121988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2016

H.261

MPEG-1

H.263

MPEG-4Part 2

MPEG-2/H.262

JOINT

H.264/MPEG-4 Part 10 (AVC)

HEVCH.264/MPEG-4 AVC

SVC & MVC Ext.

HEVC SVC & MVC Ext.

MPEG-4 Version 2

H.263 + H.263 ++VideoConference

PMP

UHD,Mobile FHD

VC-2VC-1

(from WMV) (from Dirac)

AdobeFlash Video

(8K)

*김찬열, HEVC Overview, 삼성전자 DMC 연구소, Dec. 2014.

Page 7: Next Generation Video Streaming System using Scalable High

7 © 2015 Gachon University. All rights reserved.

The History of Video Coding Standards (2)

201420121988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2016

VC-1

VP6 VP7 VP8VP3

VC-2

WebMProject

Theora

WMV7

WMV8

WMV9

Dirac

Google acquired On2

Open Source

(Open Source)

(Open Source)

(Open Source)

AdobeFlash Video

VP9

*김찬열, HEVC Overview, 삼성전자 DMC 연구소, Dec. 2014.

Page 8: Next Generation Video Streaming System using Scalable High

8 © 2015 Gachon University. All rights reserved.

High Efficiency Video Coding (HEVC)

• Joint Standard (JCT-VC)

• ISO/IEC JTC1: ISO/IEC 23008-2 (MPEG-H Part 2)

• ITU-T: ITU-T Recommandation H.265

• Higher resolution / frame rate / color space

• UHD / 4K – 8K video

AVC HEVC

Picture > Slices > MB (Max 16x16)Macroblock / Block- Transform 4x4 / 8x8

Picture > Slices > CTUs (Max 64x64)CU / PU / TU (Coding/Prediction/Transform Unit)- Transform 4x4 ~ 32/32 (DCT+DST for Intra)

Intrapicture prediction- Up to 9 directional modes

Intrapicture prediction- Angular prediction 33 directional+2 modes

Variable Block Size Asymmetric Motion Partitioning (AMP)

Motion copy mode- Direct mode

Motion copy mode- Merge Mode /Advanced Motion Vector Prediction(AMVP)- Transmit MVD

Deblocking filter only Deblocking + Sample Adaptive Offset (SAO)

CAVLC / CABAC CABAC

Slices (FMO) Tiles / Wavefront

CTU CTU CTU CTU CTU CTU CTU CTU

CTU CTU CTU CTU CTU CTU CTU CTU

CTU CTU CTU CTU CTU CTU CTU CTU

CTU CTU CTU CTU CTU CTU CTU CTU

CTU CTU CTU CTU CTU CTU

CTU CTU CTU CTU CTU CTU

CTU CTU CTU CTU CTU CTU

Thread1

Thread2

Thread3

Page 9: Next Generation Video Streaming System using Scalable High

9 © 2015 Gachon University. All rights reserved.

High Efficiency Video Coding (HEVC)

Tools for Parallel Processing and Standard Extensions:

• Performance Comparison

• HEVC Extensions• SHVC / MV-HEVC / 3D-HEVC

• Transmission • MPEG DASH / MMT

• Industrial Broadcasting Standard• ATSC 3.0 (North America)

•Contents: UHD / Smart interactivity / Multiple contents on the screen•Network: High speed broadcasting network / Emerging broadband network / Hybrid delivery•Devices: Larger screen devices / Multimedia devices everywhere / Single service on the multiple screens

Page 10: Next Generation Video Streaming System using Scalable High

10 © 2015 Gachon University. All rights reserved.

Scalable Video Coding (SVC/SHVC)

Base Layer

(Temporal & SNR Scalability)

Spatial Enhancement Layer 1

(Temporal Scalability)

I frame

B frame

B frame

B frame

P frame

Page 11: Next Generation Video Streaming System using Scalable High

11 © 2015 Gachon University. All rights reserved.

Scalable Video Coding (SVC/SHVC) (2)

• Supports various device capabilities with single

bitstream

• Reduces total BW

SVC < AVC (around 20%, for 3 Spatial layers)

• No transcoding, No duplicated contents for every

target bitrates

Temporal

Decomposition

Transform/

Entropy coding

Motion

coding

2D spatial

upsampling

Intra prediction

for intra block

Temporal

Decomposition

Transform/

Entropy coding

Motion

coding

Intra prediction

for intra block

2D spatial

Down-

sampling

QP1

QP2 Mu

ltiplex

motion

info.

Video

input

Enhancement Layer Encoder

Base Layer Encoder

SVC

Bitstream

Base Layer

Enhancement Layer n

Enhancement Layer n’

HDTV

Laptop

Mobile

High quality,

Resolution, and

temporal video

Medium quality,

Resolution, and

temporal video

Low quality,

Resolution, and

temporal video

Enhancement Layer Encoder

2D spatial

Down-

sampling

Page 12: Next Generation Video Streaming System using Scalable High

12 © 2015 Gachon University. All rights reserved.

SVC versus SHVC

• Higher-layer decoding picture buffer (DPB) uses lower-layer’s

reconstructed picture

• Simple / easy to deploy

*US20140010294 A1 (InterDigital Patent Application /2014)

Page 13: Next Generation Video Streaming System using Scalable High

13 © 2015 Gachon University. All rights reserved.

Multi-view Video Coding (MVC) / 3D

• Close correlation between multiple views.

CC

TV

#2

Close

Correlation

Camera Server

Data

Processing /

Streaming

Server

WAN

LAN

MVC encoding

Picture #1 Picture #2 Picture #3

City Park

View 1

Time 1

View 2

Time 1

View 3

Time 1

View 4

Time 1

View 5

Time 1

View 1

Time 3

View 2

Time 3

View 3

Time 3

View 4

Time 3

View 5

Time 3

View 1

Time 2

View 2

Time 2

View 3

Time 2

View 4

Time 2

View 5

Time 2

View 1

Time 4

View 2

Time 4

View 3

Time 4

View 4

Time 4

View 5

Time 4

View 1

Time 5

View 2

Time 5

View 3

Time 5

View 4

Time 5

View 5

Time 5

View

Tim

e

Page 14: Next Generation Video Streaming System using Scalable High

14 © 2015 Gachon University. All rights reserved.

Part 2Error Resilient Video Streaming- Unequal Error Protection using Raptor Codes

- Error Concealment using EC modes

- Selective Video Streaming

Page 15: Next Generation Video Streaming System using Scalable High

15 © 2015 Gachon University. All rights reserved.

General Video Streaming System

• Video server and client

Video

Encoder

Error

Protection

Selective

Scheduler

Channel Prediction

Network

MANE

(smart router)

Edge server

Home

gateway

QoS Controller(EC mode selection)

signal

data

Video Client

(Decoder with EC)

data &

signal

Error Protection

- Forward Error Correction (FEC)

- Unequal Error Protection (UEP)

NetworkVideo Client

with decoder

Video Server

with encoder

Error Recovery

Error Concealment

- Interpolation / Picture copy / ...

Error Resilient Video Streaming

Channel monitoring / feedback

Page 16: Next Generation Video Streaming System using Scalable High

16 © 2015 Gachon University. All rights reserved.

2D Parity FEC / Interleaving

P1 P2 P3

P4 P5 P6

P7 P8 P9

XOR

XOR

C1 C2 C3

R1

R2

R3

Rep

air

Pack

ets

Repair Packets

Video Packet Interleaving

P1 P2 P3 P4 FEC0

P5 P6 P7 P8 FEC1

P9 P10 P11 P12 FEC2

P13 P14 P15 P16 FEC3

FEC4 FEC5 FEC6 FEC7

k 2t

Data Parity

n

Page 17: Next Generation Video Streaming System using Scalable High

17 © 2015 Gachon University. All rights reserved.

Raptor (FEC) Codes

• Need low computational complexity

• Show strong packet loss recovery feature

• No packet order coordination is needed

• Need low memory consumption

• Support unlimited encoding length

Source Symbols

(LDPC & Half )

Pre-coded Symbols

Raptor-encoded Symbols

( LT-coding )

Redundant symbols

Video

Encoder

QoS

Controller

Bitstream &

Priority info.

Damaged

bitstream

Available bandwidth

Raptor

Encoder

Bitstream &

FEC (n, k) Raptor

Decoder

Error pattern file

Bitstream with

repair symbols Video

Decoder

Statistic

logs

Page 18: Next Generation Video Streaming System using Scalable High

18 © 2015 Gachon University. All rights reserved.

Unequal Error Protection (UEP)

• Picture-level UEP

• In single layers•I / P / B pictures

•# of picture referencing

• In different layers•Base Layer / Enhancement layer

• Region of Interest (ROI) UEP

• Flexible Macroblock Ordering

(FMO) with Slice Group (SG)

Slice Group 1 Slice Group 0 Slice Group 2

Type 0

Type 3

Type 1

Type 4

Type 2

Type 5

SG 0

SG 1

SG 2

SG 3

SG 4

SG 0 SG 1

SG 0

SG 1

SG 2

SG 0

SG 1 SG 0

SG 1

SG 0

SG 1

Page 19: Next Generation Video Streaming System using Scalable High

19 © 2015 Gachon University. All rights reserved.

Picture-level UEP

• I / P / B pictures

• Amount of picture referencing (both picture-level and MB/PU level)

• Hierarchical B pictures located in the same temporal level

1. For the video frame priority, use the temporal level (or TID) of hierarchical predictionstructure.

2. However, pictures within the same temporal level can still have different priorities.Causes different levels of temporal error propagations and showes different video quality degradations.

3. frame_priority_idc field distinguishes priority of the frames within the same temporal level.

* Eun-Seok Ryu, “Prediction-Based Picture Prioritisation Method for Hierarchical B-structure of High Efficiency Video Coding”, IET Electronics Letters, Vol. 49, No. 20, pp. 1268-

1270, Sep. 26, 2013.

Page 20: Next Generation Video Streaming System using Scalable High

20 © 2015 Gachon University. All rights reserved.

Picture-level UEP: Experimental Results

Test settings

• RA in common test condition

• Drop 1 picture per intra period

• HM6.1

Observation

• Clear difference in performance

between packet loss at Position A

and at Position B

• upper: all TID = 0, lower: 4 level

TIDs

• Error Propagation Delta (Blue v.s.

Red): 2.8dB in Y-PSNR

• Results: gain 2.2dB to 7.5dB

*Results are from JCTVC-K0262.

Page 21: Next Generation Video Streaming System using Scalable High

21 © 2015 Gachon University. All rights reserved.

ROI UEP with SVC

EL2EL1BL

Encoded LayeredSVC Stream

H.264 SVC Encoder

Module forROI Scalability

EL2EL1BL

High Quality(ROI)

Medium Quality

EL1BL

ROI

Control # of Layers after Encoding

* Eun-Seok Ryu, Sung Won Han, “The Slice Group-Based SVC Rate Adaptation Using Channel Prediction Model”, In IEEE ComSoc Multimedia Communications Technical Committee

E-Letter (IEEE MMTC), Vol. 6, No. 5, pp. 39-41, May 2011.

Page 22: Next Generation Video Streaming System using Scalable High

22 © 2015 Gachon University. All rights reserved.

Error Concealment (EC)

• Picture Copy (PC)

• Temporal Direct (TD)

• Motion Copy (MC)

• Base Layer Skip (BLSkip; Motion & Residual upsampling)

• Reconstructed BL upsampling (RU)

• EC-mode Signaling (ECMS)

0

4

2

31

6

75

8

picture copy

picture copy

hierarchical B pictures

Picture Copy.

MV0

MV1

collocated block

MVc

direct mode block

RPL 1RPL 0 current B picture

Temporal Direct for B Pictures

Page 23: Next Generation Video Streaming System using Scalable High

23 © 2015 Gachon University. All rights reserved.

EC: ECMS Motivation

• Traditionally, Error Concealment (EC) methods are applied at the decoder side to

recover from packet loss and to reduce error propagation.

• There are various EC methods that decoder can perform.

• However, the decoder alone does not know which EC method is optimal because it

does not have the original picture.

• Thus, the optimal EC method is simulated in the encoder and is signaled.

Page 24: Next Generation Video Streaming System using Scalable High

24 © 2015 Gachon University. All rights reserved.

EC: ECMS Method

1) Encoder simulates various EC methods on a damaged picture and its error

propagation effects.

2) Determines the best EC method.

3) Signals the determined optimal EC mode to the video decoder.

* Eun-Seok Ryu, Joongheon Kim, “Error Concealment Mode Signaling Method for Robust Mobile Video Transmission”, International Journal of Electronics and Communication,

vol. 69, no. 7, pp. 1070-1073, July 2015.

Page 25: Next Generation Video Streaming System using Scalable High

25 © 2015 Gachon University. All rights reserved.

EC: ECMS Method: Experimental Results

• EC0: Picture copy from the first picture in RPL0.

• EC1: Picture copy from the first picture in RPL1.

• EC2: Picture copy from the ILP (upsampled base layer picture)

• EC3: Proposed EC mode signaling

• Condition: JCTVC SHVC 4.0, BL QP: 22, EL QP 20. Random Access with 1sec intraperiod. (common test condition)

• Related MMT CE proposal: M32346 (San Jose Meeting, Jan. 2014)

Average PSNR gain between EC modes and proposed method (2x spatial scalability).

Average PSNR gain between EC modes and proposed method (single layer coding).

Average Gain: 3.27dB for 2x

Coding Sequence Resolution frame rate frame lengthSingle layer coding (HEVC) Gain (Y-PSNR, dB)

EC0 EC1 Proposed Prop.-EC0 Prop.-EC1

Single Layer Coding (HEVC)

ParkScene 1920x1080 24 240 28.86 29.05 29.14 0.28 0.10

Han 1920x1080 30 300 32.34 32.19 32.48 0.14 0.29

KristenAndSara 1280x720 60 600 37.14 37.19 37.31 0.18 0.12

BQMall 832x480 60 600 22.13 22.14 22.17 0.04 0.03

Coding Sequence Resolution frame rate frame lengthResults (Y-PSNR, dB) Gain (Y-PSNR, dB)

EC0 EC1 EC2 Proposed Prop.-EC0 Prop.-EC1 Prop.-EC2

2x Spatial Scalability (SHVC)

ParkScene 1920x1080 24 240 28.87 29.07 34.18 34.38 5.50 5.31 0.20

Han 1920x1080 30 300 32.27 32.13 38.84 38.84 6.57 6.72 0.00

KristenAndSara 1280x720 60 600 37.17 37.23 24.67 37.35 0.18 0.12 12.68

BQMall 832x480 60 600 22.12 22.12 22.23 22.82 0.70 0.69 0.58

Page 26: Next Generation Video Streaming System using Scalable High

26 © 2015 Gachon University. All rights reserved.

Selective Video Streaming

• Rate Adaptation

• Quantization Parameter (QP) Control

• Transcoding

• Selective Video Streaming•Packet (Bitstream) Drop (Temporal / Spatial Subsampling)

•SVC Layer-switching for Rate Adaptation

Page 27: Next Generation Video Streaming System using Scalable High

27 © 2015 Gachon University. All rights reserved.

Selective Video Streaming (2)

• Layer-switching for SHVC-based DASH• We cannot adapt current bitrate during a segment (e.g. 10 sec.)

• Important factor for precise rate-adaptation

1.Number of layers (e.g. 256kbps, 512kbps, 1mbps)

2.Length of segment (e.g. 5-10 sec.)

• Frequent feedback from clients is a big burden to a VSP server

Page 28: Next Generation Video Streaming System using Scalable High

28 © 2015 Gachon University. All rights reserved.

Part 3Video Streaming Applications- Video Streaming over Multiple Network Interfaces

- Power Aware HEVC Streaming with DASH

- Three-Screen TV using SVC Platform

- Telehealth and Haptic Video Streaming for the Individuals with Visual

Impairments

Page 29: Next Generation Video Streaming System using Scalable High

29 © 2015 Gachon University. All rights reserved.

Video Streaming over Multiple Network Interfaces

• Application-level• Gain: 0.7dB ~ 1.6dB in PSNR

• Transport layer-level

• Link-level

Network

Media Server Client

Smart

Channel

Allocation

• Eun-Seok Ryu, Sung Won Han, “Priority-Based Selective H.264 SVC Streaming Over Erroneous Converged Networks”, Multimedia Tools and Applications, Vol. 68, No. 2, pp 337-353,Jan. 2014.

• Eun Seok Ryu, Hye-Soo Kim, Sungjun Park, Chuck Yoo, “Priority-Based Selective H.264 SVC Video Streaming Over Erroneous Multiple Networks”, Proceedings of The 2011 IEEEInternational Conference on Consumer Electronics (ICCE), pp. 337-338, Jan 2011.

• Eun-Seok Ryu, Jung-Hwan Lee, Hyuck Yoo, "Extracting and Transmitting Video Streams based on H.264 SVC in a Multi-Path Network", Journal of the Korean Institute of InformationScientists and Engineers, Vol. 35, No 6, pp. 510-520, Dec. 2008.

• Eun-Seok Ryu, Nikil Jayant, “Video Streaming over Multiple Network Interfaces by using H.264 SVC”, Poster Session of Georgia Tech Broadband Institute Advisory Board Meeting, Oct.2008.

• Eun-Seok Ryu, Jung-Hwan Lee, Chuck Yoo, “Content-aware and Network-adaptive Video Streaming over Multiple Network Interfaces”, Proceedings of the 2008 InternationalConference on Ubiquitous City Technology (ICUCT 2008), pp. 43-49, Feb. 2008.

• Eun-Seok Ryu, Jung-Hwan Lee, and Chuck Yoo. “Multi-Channel based Scalable Video Streaming for Evacuation Guidance System” , International Conference On ConvergenceInformation Technology, November. 2007.

Page 30: Next Generation Video Streaming System using Scalable High

30 © 2015 Gachon University. All rights reserved.

Power Aware HEVC Streaming with DASH

• Complexity-based content variables (different tools using computational complexity statistics)

• Power (battery) measurement

• Content selection using feedback message

MF

Fragment

Complexity

aware

encoder

Complexity

aware

MPD

Media file

generation

File

Packager

PREPARATION

Media

File

Origin

Storage CDN

DISTRIBUTION

MPD

File

HTTP

Cache

HTTP

Server

Decoder

Power

sensing

Receiver

CONSUMPTION

Power aware client

Power

aware

adaptation

control

Bandwidth

sensing

Request

Application

Complexity

statistics

and control

Quality

Bitrate

Low complexity mode

Medium complexity mode

High complexity mode

Low resolution Medium resolution High resolution

*Yuwen He, Markus Künstner, Srinivas Gudumasu, Eun-Seok Ryu, Yan Ye, Xiaoyu Xiu, “Power Aware HEVC Streaming for Mobile”, Visual Communications and Image Processing

(VCIP) 2013, Nov. 17. 2013.

Page 31: Next Generation Video Streaming System using Scalable High

31 © 2015 Gachon University. All rights reserved.

Three-Screen TV using SVC Platform (1/9)

Introduction:

• SW-based gateway for 3STV• Low storage in server (20% lower)• Real-time SVC decoding and displaying (3-layered 1080p SVC @PC)• Error Resilient

Raptor FEC over

SVC Bitstream HDTV

Laptop

Mobile

Settop Box

Wi-Fi AP

Three Screen-TV using SVC Platform

VSP Server

Wi-Fi network

Home Gateway

(SVC Server)

VSP network

1 Spatial Layer : 640 x 360

2 Spatial Layer : 1280 x 720

3 Spatial Layer : 1920x1080Ethernet or

Wi-Fi network

SVC decodingRaptor FEC en/decodingVideo Content Multicast to Home

Page 32: Next Generation Video Streaming System using Scalable High

32 © 2015 Gachon University. All rights reserved.

Three-Screen TV using SVC Platform (2/9)

Proposed Key Technologies:

• Adaptive overhead control of Raptor FEC

• Real-time BW adaptation using SVC layer-switching

Feedback-based SVC Layer Switching

MSGMSG

• Switches SVC layers according to client’s ABW

• Uses RSS I (Received Signal Strength Indicator)-based channel measurement model

• Pros : Light-weight rate control• Post processing in home gateway

• Eun-Seok Ryu, Nikil Jayant, “Home Gateway for Three-Screen TV Using H.264 SVC and Raptor FEC”, IEEE Transactions on Consumer Electronics (TCE), Vol. 57, No. 4, pp. 1652-1660, Dec. 27. 2011.

• Eun-Seok Ryu, Sung Won Han, “Two-Stage EWMA-based H.264 SVC Bandwidth Adaptation”, IET Electronics Letters, Vol. 48, No. 20, pp. 1271-1272, Sep. 27. 2012.

Page 33: Next Generation Video Streaming System using Scalable High

33 © 2015 Gachon University. All rights reserved.

Three-Screen TV using SVC Platform (3/9)

Raptor FEC Overhead Adaptation:

Meaning : (Example)

Video Seq. : H.264 SVC (Layer 3) for HDTV

(25 frame/sec., 41.48dB)

- Orig. Bitrate : 3.03Mbps

- Rapt. Bitrate : 4.24Mbps (40% OH)

➢Avg. PSNR = 41.48 dB in 25% PLR

Control

OH

Network Condition (PLR)

Feedback

50

60

70

80

90

100

5% 10% 15% 20% 25% 30% 35% 40%

Err

or Reco

very

Packet Loss Rate (%)

0%

10%

20%

30%

40%

50%

Overhead

PLR (%)Overhead (%)

0% 10% 20% 30% 40% 50%

5% 97.18 99.9 100 100 100 100

10% 90.65 96.82 99.81 99.97 100 100

15% 84.84 89.48 99.64 99.90 99.87 100

20% 80.16 80.91 91.33 98.21 100 100

25% 75.91 76.75 78.70 93.64 99.84 100

30% 68.67 69.74 69.97 75.23 92.89 99.22

35% 64.12 65.45 64.29 65.39 69.61 94.35

40% 58.96 60.68 60.88 59.61 62.27 66.10

Experimental Results of Raptor Error Recovery

Refer the Table

Complexity : (Linear decoding complexity)➢Avg. En/decoding Time = 3.0 ms / frame

Page 34: Next Generation Video Streaming System using Scalable High

34 © 2015 Gachon University. All rights reserved.

Three-Screen TV using SVC Platform (4/9)

Packetizing SVC NALs:

34

Page 35: Next Generation Video Streaming System using Scalable High

35 © 2015 Gachon University. All rights reserved.

Three-Screen TV using SVC Platform (5/9)

System Architecture and Implementation:

Server Client

Page 36: Next Generation Video Streaming System using Scalable High

36 © 2015 Gachon University. All rights reserved.

Three-Screen TV using SVC Platform (6/9)

Experimental Results 1:

Packet loss rate (PLR)

(a) 3% (b) 5% (c) 10% (d) 15%

SVC without FEC (forward error correction)

Proposed Method: SVC with Raptor FEC (with 20% overhead)

Packet loss rate (PLR)

(a) 3% (b) 5% (c) 10% (d) 15%

Page 37: Next Generation Video Streaming System using Scalable High

37 © 2015 Gachon University. All rights reserved.

Three-Screen TV using SVC Platform (7/9)

Experimental Results 2:

Moving traces with mobile TV on building map.

Gain: Enhanced the video qualityfrom 2dB to 5dB in PSNR.

Home gateway in office

Page 38: Next Generation Video Streaming System using Scalable High

38 © 2015 Gachon University. All rights reserved.

Three-Screen TV using SVC Platform (8/9)

Online Video Demo:

• H.264 SVC streaming for three-screen TV :

http://www.youtube.com/watch?v=qKdO01Nf-14

• H.264 SVC with Raptor FEC (Fountain code) :

http://www.youtube.com/watch?v=LIBoSjOlpuY

• H.264 SVC layer switching :

http://www.youtube.com/watch?v=gkCL4mG5VYA

Page 39: Next Generation Video Streaming System using Scalable High

39 © 2015 Gachon University. All rights reserved.

Three-Screen TV using SVC Platform (9/9)

Conclusion:

• Home Gateway for 3-screen TV• SW-based RT home gateway for 3STV

• Low storage in server (28% - 36% reduced bitrates )

• Low computational complexity (3-layered 1080p SVC @PC)

• Proposed method showed distinguished performance (2dB-5dB in PSNR)

• Home Gateway for 3DTV (on-going research)• Using JMVC reference tool

• Two factors for efficient coding•Interview redundancy : correlations between views•Psycho-visual redundancy : The suppression theory of human visual perception of 3D stereoscopic video allows sub-sampling of one of the views

39

Page 40: Next Generation Video Streaming System using Scalable High

40 © 2015 Gachon University. All rights reserved.

Telehealth: Robust Medical Video Transmission

• Introduction

• Telesurgery: enable the surgical operation with the

collaboration of local and remote surgeons by using

medical video streaming•On cruise ship in the ocean

•On ambulance in the battlefield for wounded soldiers

• Isolated static site

Telesurgery with 3D-medical image

Da-Vinci surgical robot system

Page 41: Next Generation Video Streaming System using Scalable High

41 © 2015 Gachon University. All rights reserved.

Haptic Video Streaming for the Individuals with Visual Impairments (with GT & GWU)

• Over 39 million individuals are legally blind.

• No video conferencing system for visually impaired person exists today.

• We propose a novel video conferencing system (VCS) for visually impaired.

• In addition to being able to hearing, the blind can now feel consecutive 3D images by

using a combination of Kinect (or 2D camera with depth-map generation) and Haptic

(or 3D Braille) device.

• Research results could be extended to the smart guidance system for the

blind and telesurgical system.

*Chung Hyuk Park, Eun-Seok Ryu, Ayanna M. Howard, “Telerobotic Haptic Exploration in Art Galleries and Museums for Individuals with Visual Impairments”, IEEE Transactions

on Haptics (accepted), 2015

Page 42: Next Generation Video Streaming System using Scalable High

42 © 2015 Gachon University. All rights reserved.

Conclusion

• Next Generation Video Streaming System Using Scalable High Efficient

Video Coding (SHVC)

• HEVC / Scalable HEVC (SHVC)

• Error Resilient Video Streaming•Error Protection with Raptor Codes

•Error Concealment with ECMS

•Selective Streaming / UEP

• Three-screen TV using Scalable Video Coding

Video

Encoder

Error

Protection

Selective

Scheduler

Channel Prediction

Network

MANE

(smart router)

Edge server

Home

gateway

QoS Controller(EC mode selection)

signal

data

Video Client

(Decoder with EC)

data &

signal

Page 43: Next Generation Video Streaming System using Scalable High

43 © 2015 Gachon University. All rights reserved.

Thank You !

Personal webpage: https://sites.google.com/site/hoperyu/

Research blog: http://r2d2n3po.tistory.com/

Questions > [email protected]