Distributed Coordination and Cross-Layer Optimization in

École nationale supérieure des télécommunications 1 Z. Li, 2007

Distributed Coordination and Cross-Layer Optimization in

Multi-Access Wireless Video Streaming System

Zhu Li, PhD

Principal Staff Research Engineer

Multimedia Research Lab

Motorola Labs, USA


Outline

• Overview of my Multimedia Research at Motorola Labs

• Motivation

• Problem Formulation– Interference limited multiple access channel,

– Typically operate at VLBR for mutli-media traffic,

– Multi-user diversity in channel states and content rate-distortion characteristics

– How to achieve a optimal received quality among users that also best utilizes radio

resource ?

• Solution– Problem parallelization and co-ordination via dual decomposition

– Cross-Layer optimization of video adaptation with resource pricing

• Simulation Results

• Conclusion & Future Work


An overview of my

Multimedia Computing & Communication (MC2)

research at Motorola Labs


2007


Devices

• Explosive growth of devices:

– Billions of cell phones/PDAs

– Billions of computers

– Billions of TVs

– Billions of Media Players

• Different Multimedia

Capabilities in:

– display,

– capture,

– storage,

– computing,

– communication


Networks

• Better technology from

equipment makers

– Better wireless spectrum efficiency,

WiMAX/LTE

– High speed DLS/Cable

– Fiber optical solutions

• More capacity from service

providers

– More bandwidth, better coverage,

– Convergence of data, voice and

media service from service

providers

– Vertical integration of application

and services


Content

• Explosive growth of digital

media

– Web, Email, Audio, Video, Game

– News, Music, Movie, Talk show,

Game, 2nd Life.

• Rapid changes in the way

contents are produced and

consumed

– Personal vs Commercial

– Passive (TV) vs Interactive (Blog,

Game)

– Centralized vs P2P


People and Technology

• People’s need:– Good Access, be able to get what you want, a storage and communication

problem

– Mobility across devices and access points: anywhere, on any device, not tied to TV only, get what they want, with good media quality (coding) and availability (communication/networking).

– Intelligence and Personalization: be able to find what they are interested in and locate what they want, browsing with (implicit and explicit) personal preference.

– Self-expression, Interaction and Social Networking, P2P video,video blog, live events streaming, social group based video sharing. Immersive video interaction.

• Technology Gap ?– Distribution: multimedia coding, streaming and networking

– Search & Mining, multimedia analysis, indexing and retrieval, search and mining.

– Interaction, visual/audio/motion sensor data processing, pattern recognition, tracking.


It is a good time for MC2 research….

• Networked multimedia experience is still in its infancy, like

web browsing before broadband access and search

engines, there are,

• Real challenges and exciting research opportunities for

MC2 applications

– multimedia distribution (coding/communication) and,

– multimedia searching (computing) problems,

– multimedia based interaction (computing) problems

• Opportunities to advance the state-of-art in MC2 techniques:

– Systems: novel multimedia computing & communication systems

– Algorithms: visual signal processing, analysis, computer vision and

pattern recognition

– Tools: optimization, statistics, and machine learning


MC2 problems under investigation

• Multimedia Computing Problems:

– Video Search: LUminance Field Trajectory (LUFT) Based Video Indexing

and Retrieval (With A. Katsaggelos at IVPL/Northwestern)

– Large Subject Set Visual Pattern Recognition: Localized subspace

learning for large label set (head pose and motion, suspect face database)

visual pattern recognition problems, (with Y. Fu and T. Huang at

IFP/UIUC)

– Spatio-Temporal Visual Pattern Recognition: Human behavior

recognition, accelerometer sensorial data based human motion/behavior

recognition, spatio-temporal volume tensorial modeling, (with Y. Fu, S.

Yan at IFP/UIUC)


MC2 problems under investigation

• Multimedia Communication Problems:

– Video Coding and Adaptation (Motorola Lab):

» Video Summarization and Coding for VLBR (12~48kbps) Streaming

» H.265 research: joint scalability and error-resilience coding, motion field

scalable coding, new visual signal decomposition schemes.

» Multi-View Video Coding and Networking,

– Video Networking (with J. Huang and M. Chiang at Princeton):

» Video over Wireless Multi-Access Channel: Adaptation and Resource

Pricing for Multiple Access Wireless Video Communication

» Video over P2P networks: self-organizing multicasting, distributed resource

pricing.

» Video over Wireless Broadcast Channel: Joint source-channel coding for

wireless video broadcasting (mobile TV) with limited feedback, relay with

network coding, optimization.


In this talk

• Purpose:

– To lay a landscape of my current MC2 research and collaborations

– To show some in-depth techniques and results in Video over Mutli-Access

Network problems,

– To share some of my views and opinions on MC2 research and

applications,

– To have potential collaborations in the future with interested faculties


Mixed Voice/Video over CDMA Up Link

Radio tower

• Mixed QoS requirements for

Video/Voice traffics

• Limited resource, video has to

operate at VLBR

• Shared radio resource, and

interference limited capacity,

Ri=f(Pi;P-i),

• Diversity of channel gains and

source rate-distortion

characteristics among users.

• How to optimize video adaptation

and transmission to achieve better

QoS and radio resource efficiency

?


A General Formulation

• Total utility maximization subject to a shared resource

constraint,

– Where utility function Ui() is a concave differentiable function reflecting the

quality-bit rate/resource trade-offs. (true for most video source’s PSNR-R

function)

– Difficult to solve the primal problem by allocating {xi} directly, because of

coupling of {xi} in constraint.

– Transform the problem for a distributed solution, utilizing computing

capability at mobiles

max,,,

..,)(max21

xxtsxUi

i

i

iixxx n


Distributed Solution of the Dual Problem

•Lagrangian relaxation:

•The dual problem:

•Decomposed into n separable video adaptation problems at

mobiles :

•And a base station resource pricing problem:

]}))(([max{min max,,,0 21

xxxUi

iiixxx n

))((maxarg*

))((max

))((max

,,,

max,,,

21

21

iiix

i

i

iiixxx

i

iiixxx

xxUx

xxU

xxxU

i

n

n

i

ig )(min0

)()()( max

** xxxUg iiii

)()(),,,,( max21 xxxUxxxLi

i

i

iin


BTS Mobile i

Announce resource price in iteration kk

Mobile optimization:

Protocol for Distributed Optimization

))((maxarg* i

k

iix

i xxUxi

Report back resource used xi* in iteration k

Increase price, if

Otherwise decrease price i

i xx max

k

kk 1 Mobile optimization:

))((maxarg* 1

i

k

iix

i xxUxi


Distributed Optimization for Multiple Access Video Network

• Geometrical

Interpretation on price:

– From the Karush-Kuhn-

Tucker (KKT) condition:

– Allocations {xi*} will have

the same marginal utility

(slope) as -price.

– Optimal price must also be

tight on all available

resource.

*

**,,0

i

i

i x

U

x

L

U1(x1)

U2(x2)

U3(x3)

x1* x2

* x3*

U1(x1)

U2(x2)

U3(x3)

x1* x2

* x3*


Video Over Multiple Access Channel

• In solving real world problems with this distributed pricing

scheme:

– Source coding: scalability, adaptability issues

– Diversity in Channel state

– Diversity in content

– Collaboration in resource allocation, scheduling

– Uplink problem: interference limited

– Downlink problem: power limited.

– Computational complexity


CDMA Uplink with Mixed Voice/Video Traffic

• Consider a single cell CDMA uplink:

– Pvoice – received power for a voice user

– M – total voice users

– Pvideo – total received power for all video users

– Gvoice - modulation scheme related constant, BPSK = 1, QPSK = 2

– W - bandwidth (Hz)

– voice - voice QoS minimum SINR

• Received Power Constraints:

– QoS for voice users:

– Max allowable total received power for video users

,)1(0

voice

voicevideo

voice

voice

voice

PMPWn

P

R

WG

.1 0max WnPMR

WGP voice

voicevoice

voice


Problem Formulation

• Control video mobiles’ transmitting power to achieve social

optimality in total received utility (video quality) :

– Optimization is over a sliding window of size T

– Utility (PSNR, e.g) is a function of total rate in T

– Total N video users.

• How to solve ?

– Spend resource that can give maximum return in quality

» Account for content diversity, each has different R-D curves

» Account for channel state diversity,

– Distributed solution

],0[,..,max max

11 00

1

TtPtPtsdttRUN

j

j

N

j

T

jjtP

Njj

P


Multiple Access for Dominanting Received Power Users

• Time-Division Multiplexing (TDM) is needed among video

users

– Video users’ received power too strong for spectrum efficiency

– Example: 4 video users’ achieve able total rates plot:

– Therefore, we choose TDM among video users.


Problem Formulation with TDM Among Video Users

• Allocate transmission slots among video users to achieve

social optimality in total received utility (video quality) :

– Total time slots {tj} length is T.

– RTDM is the rate achieved using Pmax for a single video user, with current

voice traffic load.

,..,~

max11

01

N

j

j

N

j

jjt

TttstUNjj

.~

jTDMjjj tRUtU


Dual Decomposition : Pricing Solution

• The primal problem is difficult to solve.

– The problem is convex, since we assume utility functions are convex.

– Constraints are also convex

– Strong duality exists.

• Dual Decomposition through Lagrangian Relaxation:

– Lagrangian:

– Mobile source adaptation surplus problem:

– Base station resource pricing problem:

,~

,max11

0

N

j

j

N

j

jj TttUJ tt

.~

maxarg jjjlj ttUtj

,,max 0 tJ

.~

jTDMjjj tRUtU


Video Source Adaptation with Resource Pricing

•The source surplus problem is to maximize pay off as utility minus

cost in resource

–Distributed to each video source, interact with other video users thru the price.

–If scalable coded source, optimal bit extraction subject to a price on resource. Utility

could be the PSNR quality of the video

–For VLBR (e.g. 24~120kpbs), code video frames at very low PSNR is not

preferable. Use video summarization scheme instead.

.~

maxarg jjjlj ttUtj

jjjS

j StSDSj

minarg*


Video Summary

•What is video summary ?

–A shorter version of the original video that preserves most information.

•Definitions:

– n-frame video sequence:

– m-frame video summary:

– reconstruction by repeating last summary frame:

– distortion:

– rate:

},,{110 mlll fffS

}',','{' 110 nS fffV

)',()(1

0

jj

n

j

ffdSD

coding-inter,

coding-intra,

)()(1

1

1

01

0

1

0

m

t

l

l

l

m

t

l

m

t

l

t

t

t

t

rr

r

fbSR

},,{ 110 nfffV


Video Summary Examples

n=10, S={f0, f3, f5, f8 } , m=4, D(S)=0.6

1 2 3 4 5 6 7 8 9 100

0.5

1

1.5

2

d(f

k, f

k')

summary distortion

1 2 3 4 5 6 7 8 9 100

0.2

0.4

0.6

0.8

1

1.2

1.4

d(f

k, f

k-1)

summary frames

f0 f0 f0 f3 f3 f5 f5 f8f8f5

f0 f1 f2 f3 f4 f5 f6 f9f8f7d(f0, f1)

d(f0, f2)

V=

VS’=


Frame Distortion for Summarization: What is a good d(fj, fk) ?

“foreman” seq in 2-d (1st and 2nd component) PCA space

scale PCA

.

352x240 video frame 11x8 image icon d-dimensional point

1

101 201

301

X1

X2400

.d(fj,fk)


Surplus problems at mobiles

•The adaptation problem:

– compute summary at mobile j, s.t. the following surplus function is maximized,

– for the given voice traffic load, RTDM is known, R(Sj) is the bit rate for the resulting

video summary.

– exhaustive search is exponential in complexity,

– the problem has some structure for which we will exploit for a Dynamic

Programming solution.

TDM

jj

S

jjjS

j

R

SRSD

StSDS

j

j

minarg

minarg*


Distortion State and Cost

• Summary Segment Distortion:

• Distortion State Dtk, for summaries with t frames ending with fk,

• Bit cost for Dtk,

• The surplus problem:

11

1 ),(t

t

t

t

t

l

lj

jl

l

l ffdG

}{min2

2

1

1

221

0,,

n

k

k

l

l

l

l

lll

k

t GGGGDt

t

1

0

)(t

j

l

k

t jfbR

}{min221 ,,

,

TDM

ktk

tlll

kt

R

RDJ

t


The surplus recursion at mobile

– To simplify notation, let a new price on bit be,

–The recursion:

codinginter if},{min

codingintra if},{min

)}(])([

)]()([{min

)}()]()()([

{min

)]}()(

)()([{min

}{min

1

11

1

11

1

,1

11

11

1

121

1

111

1

2

1

121

1

1

1

121

121

,,

,,

00,,

10

0,,

100,,

11,,

,1

k

l

kllt

l

kkllt

l

k

e

n

k

k

l

n

l

l

n

l

l

lll

kl

n

k

k

l

n

l

n

l

l

l

l

lll

kl

n

k

k

l

l

lll

k

t

k

tlll

kt

t

tt

t

tt

t

ktl

tt

ttt

t

ttt

t

tt

t

tt

t

reJ

reJ

fbGGG

fbfbGG

fbfbfbfb

GGGGGG

fbfb

fbfbGGG

RDJ

TDMR


Trellis Representation

1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 60

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

summarization: = 1.50e-004, Kmax

=5

J10=9.99

J21=10.09

J22=9.99

J23=9.89

J24=10.11

J25=9.99

J32=10.05

J33=10.00

J34=10.04

J35=9.89

J43=10.09

J44=10.15

J45=10.00

J54=10.24

J55=10.09 J

65=10.24

fra

me k

epoch t

– DP solution for surplus

maximization under a given

price on resource

– Start with first frame

– Compute the max surplus

incoming edge at each

node

– Backtracking for optimal

solution.


Summarization Results

10 20 30 40 50 60 70 80 90 100 110 1200

20

40

60

80

d(f

k, f k)

summary distortion

= 16.0e-4, D(S)=24.6, R(S)=80.8kb

0 20 40 60 80 100 1200

10

20

30

40

50

60

d(f

k, f k-

1)

summary frames

10 20 30 40 50 60 70 80 90 100 110 1200

20

40

60

80

d(f

k, f k)

summary distortion

=12.0e-4 D(S)=16.8, R(S)=107.2kb

0 20 40 60 80 100 1200

10

20

30

40

50

60

d(f

k, f k-

1)

summary frames

=1.6e-5,

PSNR=30dB, D(S)=24.6, R(S)=80.8kb=1.2e-5,

PSNR=30dB, D(S)=16.8, R(S)=107.2kb


Video Summarization Scheme for VLBR Channels

• Optimally select a subset of frames to code at a higher

PSNR quality

– “Foreman”

sequence

– Bit rate range:

11.2kpbs ~46.5kbps

– PSNR: 29dB ~

34.3dB

– R(S)


Base Station Price Control Problem

• Base station solves for a price that maximizes total utility

– Achieved through a sub-gradient method, checking for constraint violation at each price iteration:

– The sub-gradient search converges if the step sizes:

– In practice, price iteration stops when total utility improvement ratio is below certain threshold.

– Also the time slot allocation need to be schedulable.

,,max0

tJ

.,0max1

1 TStN

j

i

jj

iii

0lim i

i i

i


Joint Packet Scheduling for video summary transmission

• Video packets are delay sensitive.

– In TDM scheme, we have a GREEDY solution: sort packets by their

deadlines, transmit the nearest deadline ones.

– Pricing iteration is actually on schedulability (deadline violations)

}0,max{,0max1 iGREEDYii


Simulation Results

• Simulation set up:

– Channel (IS-95 alike):

Video Users:

» 4 segments (90 frames each) from “foreman” and “mother-daughter”

sequences

» Fixed PSNR: 27.8dB (foreman), and 31.0dB (mother-daughter)

Entity Symbol Value

Bandwidth W 1.228MHz

Noise density n0 8.3*10-7

mW/Hz

Voice target SINR voice 6dB

Voice modulation BPSK

Voice received power Pvoice 1mW

Voice spreading gain Gvoice 128

Voice rate Rvoice 9.6kbps

Video target SINR video 6dB

Video modulation QPSK


Price and Distortion Convergence

• Pricing iteration convergence at base station (left Fig), and

summarization distortions at mobiles (right Fig):


Simulation Results

• Resulting video summaries with pricing co-ordination

– D(S1)=3.09, D(S2)=6.42

– D(S3)=0.76, D(S4)=0.81


Simulation Results – Compare with SIMCAST

• Resulting video summaries without pricing co-ordination

– D(S1)=2.85, D(S2)=31.43

– D(S3)=0.059, D(S4)=0.068


Summary for Uplink Solution

• The performance

– In this work we proposed an efficient solution to support mixed voice and VLBR

video traffic that can help seamless migration from 2.5G to 3G and B3G systems

– The solution is distributed, with minimum communication overhead (prices,

summary frames) between base station and mobiles

– The computational complexity for source adaptation is distributed among mobiles

– The solution seems to work well in convergence

• In the future

– Extend dual decomposition to handle upload bandwidth allocation in P2P

streaming

– Handle more complex constraints in wireless ad hoc network scenario.

• Thanks to my research collaborators in this topic:

– Prof. Aggelos Katsaggelos, Northwestern

– Prof. Mung Chiang, Princeton

– Prof. Jianwei Huang, CUHK

– Ying Li, Princeton, visiting PhD Student at Multimedia Lab - Motorola Labs,


Related Publications

– Z. Li, J. Huang, and A. K. Katsaggelos, “Pricing Based Collaborative Mutli-User Video Streaming Over Power Constrained Wireless Down Link”, oral paper, IEEE Int’l Conference on Acoustics, Speech and Signal Processing (ICASSP), Toulouse, France, 2006.

– Z. Li, J. Huang, M. Chiang, and A. K. Katsaggelos, “Intelligent Wireless Video Communication: Source Adaptation and Multi-User Collaboration”, invited paper, special issue on Multimedia Communication, Ed. Changwen Chen, China Journal of Communication, December, 2006.

– Z. Li, J. Huang, and A. K. Katsaggelos, “Utility Driven Video Segment Scheduling for Peer-to-Peer Live Video Streaming System”, 45th Allerton Conference on Communication, Control and Computing, Monticello, IL, USA, 2007.

– J. Huang, Z. Li, M. Chiang, and A. K. Katsaggelos, “Pricing Based Efficient Multi-User Wireless Video Communication over a CDMA Downlink”, accepted to IEEE Trans. on Circuits & System for Video Tech.

– Y. Yang, Z. Li, W. Shi, Y. Chen, and H. Xu, “Network-Aware Mobile Gaming Traffic Shaping and Scheduling”, submitted to IEEE Trans. on Multimedia.


Questions

?

…Thanks!


Backup: Network Device Icons

Radio tower


Power Constrained CDMA Down Link Video

• Code Division

• Total Transmitting Power Constrained

• Content R-D diversity

• Maximize total video quality

Video source 1

Video source 2

Video source n

Radio tower


Power Constrained CDMA Down Link Video

• Problem Formulation:

– Considering a segment of video of duration T for all users

– Allocate a power function for each user, subject to a total power constraint

• Similar solution to the uplink problem:

– Goal: achieve max total quality among users

– Two stage solution:

» Power Level Allocation

» Joint Packet Scheduling

» Base station iterates on power price, until total utility converges.

0)(,)(..),)((max1

max

01)(,),(),( 21

tPPtPtsdttPU j

n

j

j

T

t

j

n

j

jtPtPtP n


Down Link Problem Dual Decomposition

• Dual Decomposition:

– Lagrangian:

– Dual problem

– Where source problem becomes separable, for given price :

– Base station problem: pricing control:

)()(max)(1

max1,,, 21

n

jjj

n

jj

PPPPPPUJ

n

)}()(max{min1

max

1,,,0 21

n

j

jj

n

j

jPPP

PPPUn

j

i

jjP

j PPUPj

)(max*

i

)}(,0max{ max

*1

j

j

ii PP


Down Link Source Problem

• Source Problem:

– Maximize a surplus function:

– For each user, find an optimal power level Pji that maximizes surplus

– Power price is given by base station,

– Utility depends on video coding and available adaptation scheme

ji

jjP

ij PPUP

j

)(max


Down Link Source Problem

• Source Problem:

– Video Summarization for VLBR case, very similar to the uplink case:

D(S) is the summarization distortion, P(S, W, h) is the power level needed to

transmit all summary frames with bandwidth W and channel gain h.

– Scalable video stream extraction for medium rate range:

),,()(minarg)(*jj

ij

S

ij hWSPSDS

j

),,()(maxarg)(*

jj

i

jL

i

j hWLPLPSNRLj


Source Problem

• Solutions for both Summarization and Bit Extraction are

similar in structure

– A “Convex Hull” solution similar to the bit constrained summarization.

– FGS scalable stream is quantized into packets, an optimal extraction for

given price on resource is a path thru the all possible extraction routes.

– Has the following recursive relation:

– Which gives us a polynomial complexity Viterbi algorithm like solution.

}{max

}][max{max

max

*

1

1

1,,,

1,,,

*

121

21

nnn

nnkknn

kkn

llnl

ll

n

k

llllll

n

k

lllll

n

RUV

RURU

RUV


FGS Packets extraction

• Example of DP trellis and optimal path

• GoP size n=8, data from “foreman” sequence

• Optimal extraction paths for prices 0.002 and 0.008


Joint Packet Scheduling

• Now each video source come up with a set of packets

(frames) with different size and deliver deadline, is it

actually schedulable ?

• For scalable case, no need to schedule, because of receiver

buffering .

• For video summary case, need to guarantee each frame’s

data arrive on time.

• Solution: A greedy water filling scheduling algorithm.


Greedy Water Filling Solution

• Packets are identified by triplets: {Bki, tk

j, Tkj} sort by their

deliver deadlines

• For video summary case, need to guarantee each frame’s

data arrive on time.

• Solution: A greedy water filling scheduling algorithm.

– For a given packet and its deadline, find the min power level that will be

able to send it on time.


Greedy Water Filling Solution

• Power function for user j in transmitting packet k

else

TtttPLLtP

kkjk

,0

],[),();(

• Determine the level L

thru water-filling

• B(L*) = Bkj

dtWN

tPh

WN

LtPhWLB

j

jj

T

t

j

j

kjk

k

))(

1log(

));(

1log()(

0

0


Simulation

• For Pmax=2.4, resulting optimal video summaries for 4 users:

– Optimal price =101.45

– Average bit rates: 20.1 43.3 8.1 9.4 kbps

– Channel: H=[0.75 1.0 0.8 0.65]

*

Pmax=2.4/4=0.6


Simulation

• Joint packet scheduling vs. single user greedy scheduling

– Pmax = 2.4

– Left: Joint Scheduling, Right: Single user based, not schedulable for

Pmax=2.4


Summary for Down Link work

• Solution based on Dual Decomposition

• Coordination thru pricing on power

• Collaboration thru joint packet scheduling

• Computational Complexity can be distributed

• Works for a variety of adaptation scheme like

summarization, scalable stream extraction.


Summary for pricing scheme

• Dual decomposition is a powerful framework in distributed

optimization

– Source granularity and utility modeling is essential

– Similarities to a set of economics problems, methodologies like pricing

and auction can be applied

• Future work

– Investigate pricing scheme for P2P, convergence issues, stability issues

– Source-Channel coding and optimization scheme for video broadcasting

– Auction schemes for multi-user video over wireless mesh network.

• Thanks to my research collaborators

– Prof. Mung Chiang, Princeton

– Prof. Jianwei Huang, CUHK

– Prof. Aggelos Katsaggelos, Northwestern

– Ms. Ying Li, PhD Student, Princeton


Q & A

……

Thanks!

Documents

Distributed Coordination and Cross-Layer Optimization in