F.LIVEShannon Chen (cchen116@Illinois.edu),

Zhenhuan Gao, and Klara NahrstedtUniversity of Illinois at Urbana-Champaign

Towards Interactive Live Broadcast Free-viewpoint TV Experience

This material is based in part upon work supported by the National Science Foundation (NeTS-0520182)

FREE-VIEWPOINT TV (FTV)

Multi-Lens Capturing

Free-Viewpoint Viewing

DESIRED FTV FEATURES• Interactive• Live• Broadcasting

DESIRED FTV FEATURES• Interactive

• Live• Broadcasting

*beep*

DESIRED FTV FEATURES• Interactive

• Live• Broadcasting

DESIRED FTV FEATURES• Interactive• Live

• Broadcasting

DESIRED FTV FEATURES• Interactive• Live

• Broadcasting

DESIRED FTV FEATURES• Interactive• Live• Broadcasting

1000-scale audience group

EXISTING FTV DELIVERY

FRAMEWORKS• Type-1: View chosen by content provider• EyeVision System [Kanade ‘01] used in Super Bowl

XXXV

Editing DecodeAggregateCapture

Capture

Capture

Encode

Director

Viewpoint decision

Audience

• Interactive• Live

EXISTING FTV DELIVERY

FRAMEWORKS• Type-2: Aggregated stream bundle• Nagoya System [Tanimoto ‘12]

• Interactive• Live

DecodeAggregateCapture

Audience

CaptureViewpoint decision

Capture

Encode

OUR SOLUTION: F.LIVE• View-based delivery framework

Audience 1

Viewpoint decisions

Renderer

Audience 2

Renderer

Decode

Decode

Capture

Capture

Capture

Encode Transmit

Transmit

TransmitEncode

Encode

Session manager

Transmission assignment

• Interactive• Live • Broadcasting ?

OUR SOLUTION: F.LIVE• No aggregation → distributed entities• P2P sharing among audience• Pub/Sub Model

Session manager

OUR SOLUTION: F.LIVE• No aggregation → distributed entities• P2P sharing among audience• Pub/Sub Model

Session manager

Publishers

OUR SOLUTION: F.LIVE• No aggregation → distributed entities• P2P sharing among audience• Pub/Sub Model

Session manager

Broker

OUR SOLUTION: F.LIVE• No aggregation → distributed entities• P2P sharing among audience• Pub/Sub Model

Session manager

Subscribers

OUR SOLUTION: F.LIVE

Session manager

View decisions

OUR SOLUTION: F.LIVE

Session manager

Transmission assignments

OUR SOLUTION: F.LIVE

Session manager

OUR SOLUTION: F.LIVE• No aggregation → distributed entities• P2P sharing among audience• Pub/Sub Model

S

SS

Physical proximity

• Broadcasting

CHALLENGES• Interactive ← Synchronization delay• Live ← Content freshness• Broadcast ← Producer bandwidth

CHALLENGES

R

CHALLENGES• Synchronization delay• Content freshness

Buffer for stream A

Buffer for stream BCam

B

Cam A

Audience site

1

1

CHALLENGES• Synchronization delay• Content freshness

Buffer for stream A

Buffer for stream BCam

B

Cam A

Audience site

1

2

2

1

CHALLENGES• Synchronization delay• Content freshness

Buffer for stream A

Buffer for stream BCam

B

Cam A

Audience site

1

12

3

3

2

Wait

CHALLENGES• Synchronization delay• Content freshness

Buffer for stream A

Buffer for stream BCam

B

Cam A

Audience site

1

12

2

3

4

4

3

CHALLENGES• Synchronization delay• Content freshness

Buffer for stream A

Buffer for stream BCam

B

Cam A

Audience site

1

12

2

3

4

4

3

Rendering

CHALLENGES• Synchronization delay• Content freshness

Buffer for stream A

Buffer for stream BCam

B

Cam A

Audience site

1

12

2

3

4

4

3

Rendering

Propagation delay (i.e., frame elapse = freshness)

Synchronization delay

CHALLENGES• Synchronization delay• Content freshness• Producer Bandwidth

CHALLENGES• Synchronization delay• Content freshness• Producer Bandwidth

CHALLENGES• Synchronization delay• Content freshness• Producer Bandwidth

Gbps

FOREST PLANNING CHALLENGES

• If we have all the subscription information at the beginning of forest planning , an optimal planning is not a hard problem

• However, subscriptions are dynamic• P2P churn• View change “Forest

adaptation”Initial forest construction

Cam B

Cam A

Cam C

FOREST PLANNING CHALLENGES

• If we have all the subscription information at the beginning of forest planning , an optimal planning is not a hard problem

• However, subscriptions are dynamic• P2P churn• View change “Forest

adaptation”Initial forest construction

Cam B

Cam A

Cam C

Audience join

Bandwidth?Freshness?

FOREST PLANNING CHALLENGES

• If we have all the subscription information at the beginning of forest planning , an optimal planning is not a hard problem

• However, subscriptions are dynamic• P2P churn• View change “Forest

adaptation”Initial forest construction

Cam B

Cam A

Cam C

Audience leave

FOREST PLANNING CHALLENGES

• If we have all the subscription information at the beginning of forest planning , an optimal planning is not a hard problem

• However, subscriptions are dynamic• P2P churn• View change “Forest

adaptation”Initial forest construction

Cam B

Cam A

Cam C

Audience leave

3 4

FOREST PLANNING CHALLENGES

• If we have all the subscription information at the beginning of forest planning , an optimal planning is not a hard problem

• However, subscriptions are dynamic• P2P churn• View change “Forest

adaptation”Initial forest construction

Cam B

Cam A

Cam C

View change

FOREST PLANNING CHALLENGES

• If we have all the subscription information at the beginning of forest planning , an optimal planning is not a hard problem

• However, subscriptions are dynamic• P2P churn• View change “Forest

adaptation”Initial forest construction

Cam B

Cam A

Cam C

View change 3 3

FOREST PLANNING CHALLENGES

• If we have all the subscription information at the beginning of forest planning , an optimal planning is not a hard problem

• However, subscriptions are dynamic• P2P churn• View change

We do not aim for optimization (no tearing down trees) Heuristics that deal with new batch of subscription

requests (audience join/leave/view change) Details on initial construction and adaptation

algorithms (w/ pseudocodes) can be found in the paper [Shannon Chen et al. INFOCOM’16]

“Forest adaptation”

Initial forest construction

EVALUATION• Simulation settings

• Single TV-studio performer site

• Network capability of audience sites: [Netmap]• In/out-bound bandwidth, site-to-site propagation delay

• Simulate new subscription requests when there are 0 to 100,000 audiences in the session

High-resolution cameras

Moderate 100-camera array

# of cameras 16 30 100Camera framerate 30 FPS 30 FPS 30 FPSCamera bitrate 12 Mbps (HDTV) 6 Mbps 2 Mbps (SDTV)

EVALUATION METRICS• Synchronization delay (Interactive)• Content freshness (Live)• Producer bandwidth (Broadcast)

SYNCHRONIZATION DELAY

• Unstable at first: not many candidates for newly joined/rejoined audience to find a group of sources with similar propagation delays

• Delay for handling new coming subscriptions during application session is in 100ms-scale in stable state

020

040

060

080

010

000

200400600800

1000High-Res Setting

Audience group size0

200

400

600

800

1000

0200400600800

1000Moderate Setting

Audience group size0 5000 100000

200400600800

1000100-Camera Setting

Audience group size

Sync

del

ay

(ms)

Sync

del

ay

(ms)

Sync

del

ay

(ms)

PRODUCER BANDWIDTH

CONSUMPTION

• P2P sharing restricts the growth of bandwidth consumption

• Outbound bandwidth requirement is well-manageable by Gbps infrastructure

0 200 400 600 80010000

250

500High Bitrate Setting

Audience group size

Tota

l pro

duce

r bi

trate

(Mbp

s)

0 200 400 600 80010000

250

500Moderate Setting

Audience group size

Tota

l pro

duce

r bi

trate

(Mbp

s)0 2500 50000

250

500100-Camera Setting

Audience group size

Tota

l pro

duce

r bi

trate

(Mbp

s)

CONTENT FRESHNESS

• > 50% audience have higher-than-average elapses

• But the tree structure makes the elapse grows sub-linearly• Max elapse < 4.5 seconds

(compare: CBS TV network’s time elapse is 5 sec)

COMPARE TO OTHER FTV FRAMEWORKS

Editing DecodeAggregateCap

Cap

Cap

Encode

Audience

DecodeAggregateCap

Audience

CapViewpoint decision

Cap

Encode

Type-1: customized content

Type-2: aggregated content

Viewpoint decision

MVC

COMPARE TO OTHER FTV FRAMEWORKS

020406080

100120140160

Band

widt

h co

nsum

ptio

n of

aud

ienc

e sit

e (M

pbs)

High-Res Moderate 100-CamerasType-1 (dash)View-based (solid)

High-Res Moderate 100-CamerasType-2 (stripe); View-based (solid)

Producer site bandwidth consumption

Audience site bandwidth consumption

CONCLUSION• We propose a new FTV content delivery framework

which aims at co-existence of three desired features• Interactive• Live• Broadcasting

• Result of large-scale simulation shows the proposed F.Live framework with view-based delivery chain achieves• Interactive response time in 100ms-scale• Acceptable content freshness by TV industry standard• Feasible bandwidth consumption while sustaining

1000-scale audience group

THANK YOU