Royalty Cost Based Royalty Cost Based Optimization for Video Optimization for Video

Compression Compression

Emrah Akyol, Onur G. Guleryuz, and M. Reha CivanlarEmrah Akyol, Onur G. Guleryuz, and M. Reha Civanlar

DoCoMo USA Labs,DoCoMo USA Labs,

Palo Alto, CAPalo Alto, CA

USAUSA

22

OutlineOutline

Setup and motivationSetup and motivation Problem definitionProblem definition Our solution with some interesting simulation Our solution with some interesting simulation

results.results. ConclusionConclusion

33

Setup-1: Diverse set of terminals in Setup-1: Diverse set of terminals in media deliverymedia delivery

Cell phones

PDAs

…

HDTVs

Media

Encoding 1

Encoding 2

Encoding K

…

Required quality/ effective bandwidth

…

Increasing media quality/effective bandwidth

Example terminals decoding the media

content licensing cost 1

content licensing cost 2

content licensing cost K

44

Setup-2: Diverse set of toolsSetup-2: Diverse set of tools

Tool 1

Tool 2Tool 3

Tool 4

…

Tool T

Media data

Decoded Media

Tool j = video motion compensation

Integer-pixel accuracy motion compensation

1/2-pixel accuracy motion compensation

1/4-pixel accuracy motion compensation

1/8-pixel accuracy motion compensation

Increasing quality

Compression tools, error correction tools, transport

tools, …

Tools have different

royalty/licensing costs.

(Media Consumer)

55

Thought ExperimentThought Experiment

• Many media delivery technologies available.

• One can transport media through a variety of networks, using a vast range of tools

that correspond to vast ranges in efficiency in end to end delivery.

• Rather than restricting to rigid toolsets, standard profiles, etc., can one be flexible

and allow all tools to contribute?

• Standardization process mostly allows a coarse set of options. It mostly caters to the

average good. Many good tools and technologies get cut out because they are not

general enough. Selection almost always involves compromises.• Standardization royalties may force simple tools and sophisticated tools equal share

of the revenue. Impetus for alternative avenues for tool deployment.• There is movement in this direction (software decoders, MPEG RVC, …). • MPEG RVC (reconfigurable video coding):

• put many video compression tools (potentially overlapping functionality) in one big library,

• language syntax to specify which tools are needed in decoding a given video stream.

Why?Why?

66

Media Quality

Effective Bandwidth

Royalty Cost

• Royalty cost of delivering media at a particular quality and bandwidth.• Example surface defined by achievable [quality, bandwidth, royalty cost] triplets. • All triplets below the surface are achievable.

Media delivery cost surfaceMedia delivery cost surface

(content licensing + tool licensing)

77

Cuts from the surfaceCuts from the surface

Media Quality

Effective Bandwidth

Q

Media Quality

Effective Bandwidth

C

Royalty Cost

Royalty Cost

Media Quality

Effective Bandwidth

B

Royalty Cost

“MPEG-1”

“h.264/AVC”

88

Complicated Royalty CostsComplicated Royalty Costs

• My content costs $5 for cell phone terminals, $15 for HDTV terminals.

• My compression tool is free of charge.

• My compression tool costs $0.001 per use.

• My tool costs $0.10 per movie, $0.15 for sports, ....

• My tool costs $0.10 except when combined with all free tools, in which case it too

becomes free.

• My tool is free for not-for-profit use.

• …

99

System Level: Media ServerSystem Level: Media Server

Video segment

Flexible Encoder Bitstream

Tool set selection

Encoding parameters

Registry

I will mostly talk about

compression related tools

1010

Media Server Registry

User

Certificate can be used to verify with information from the user site (for each media segment or periodically) to ensure the legitimacy of the media.

Time instant : certificate

Time instant : Encoded media and associated certificate

System Level: Media DeliverySystem Level: Media Delivery : quality : bandwidth : tool list…

Time instant :

Time instant :

1111

Rate - Distortion - Royalty Cost Rate - Distortion - Royalty Cost Optimization SetupOptimization Setup

Problem Definition:For each of the M segments, find the set of tools to use such that distortion is minimized under total rate and total royalty cost constraints, i.e.,

s.t. ,

This is a simplification.

Optimization can get elaborate.

No reason to pay for the latest/greatest tools • if plenty of bandwidth• if simple/easy content• if cheaper tools are available• …

1212

Optimization Example Using Optimization Example Using Compression Tools Compression Tools

Table II: Rate and distortion changes with different tools. The utilized tools are subpixel accurate ME, loop filter, advanced entropy coding, and multiple reference frames respectively. The rate reduction is shown as percentage with respect to the baseline along with PSNR gain at QP=25.

Clip-1 foreman

Clip-2 akiyo

Clip-3 coast.

Clip-4 mobile

Clip-5 bus

Tool-1Rate

56.51% 55.93% 14.93% 51.56% 50.05%

ΔPSNR 0.40 0.48 0.72 0.64 0.80

Tool-2Rate

1.05% 0.37% 0.76% 0.02% 0.45%

ΔPSNR 0.05 0.25 0.01 0.06 0.06

Tool-3Rate

7.21% 4.69% 10.85% 7.81% 4.85%

ΔPSNR 0.06 0.02 0.06 0.06 0.05

Tool-4Rate

15.28% 2.45% 0.65% 19.30% 5.87%

ΔPSNR 0.18 0.06 0.02 0.25 0.12Table I: Assigned cost of using each tool for two different cases

Case-1 Case-2

Tool-1 (Subpixel MV)

10 10

Tool-2 (Loop filter)

10 0.13

Tool-3(CABAC)

10 1.55

Tool-4(Multiple Ref.)

10 1.82

• Case 1: “Flat Rate”, similar to today’s licensing.• Case 2: “Fair Rate”, each tool gets paid based on its contribution

1313

““Flat Rate”Flat Rate”

400 600 800 1000 120033

34

35

36

37

38

39

Rate (kbps)

PS

NR

(dB

)

Adaptive vs Constant Tool Selection, Case-1

C=%100Adaptive toolset, C=75%Constant toolset, C=75%Adaptive toolset, C=50%Constant toolset, C=50%Adaptive toolset, C=25%Constant toolset, C=25%

• Constant toolset: Optimized tools for the entire duration of the content.• Adaptive toolset: Optimized tools for each GOP (toolset can change every GOP = media segment)

• concatenation of 10 clips• Significant reduction in royalty costs with small loss in efficiency • Inefficient tools get cut out (lobbying for tools is useless).• Adaptive ~ constant

Quality

1414

““Fair Rate”Fair Rate”

0 500 1000 1500 200032

33

34

35

36

37

38

39

40

Rate (kbps)

PS

NR

(dB

)

Adaptive vs Constant Tool Selection, Case-2

C=%100Adaptive toolset, C=75%Constant toolset, C=75%Adaptive toolset, C=25%Constant toolset, C=25%Adaptive toolset, C=12%Constant toolset, C=12%

• Adaptive better than constant.• Significant reductions are difficult.

Quality

1515

Tool UsageTool Usage

• Adaptive allows more tools to contribute (tools that are good in niche situations get used).

Tool-1 Tool-2 Tool-3 Tool-4

C=75%Adaptive

97% 48% 98% 57%

C=75% Constant

100% 0% 100% 0%

C=25%Adaptive

69% 0% 20% 10%

C=25% Constant

100% 0% 0% 0%

C=75%Adaptive

76% 72% 98% 67%

C=75%Constant

100% 100% 0% 0%

C=25%Adaptive

29% 41% 2% 18%

C=25%Constant

0% 0% 100% 100%

“Flat Rate”

“Fair Rate”

1616

ConclusionConclusion• A system that allows practical deployment of royalty cost optimized media delivery.A system that allows practical deployment of royalty cost optimized media delivery.

• Very interesting optimization problem with sophisticated royalty costs. Very interesting optimization problem with sophisticated royalty costs.

• Content adaptive.Content adaptive.

• Increases efficiency. Allows non-standard tools to contribute. Increases efficiency. Allows non-standard tools to contribute.

• Much reduced royalties when plenty of bandwidth/resources.Much reduced royalties when plenty of bandwidth/resources.

• Flat rate: significant reductions in royalty cost possible. Flat rate: significant reductions in royalty cost possible.

• Fair rate: significant reductions difficult.Fair rate: significant reductions difficult.

• Adaptive optimization allows each tool to contribute when its niche comes.Adaptive optimization allows each tool to contribute when its niche comes.

• Can allow other resources relevant to media delivery into optimization (power consumption, Can allow other resources relevant to media delivery into optimization (power consumption,

memory usage, etc.)memory usage, etc.)

• Going forwardGoing forward::

• Optimization issues.Optimization issues.

• More tools.More tools.

• Dependencies among tools.Dependencies among tools.

• Addition of communication/networking related tools.Addition of communication/networking related tools.

• Fair rate• Allow segment based licensing

1717

• Combines game theory and economic concepts with rate-distortion Combines game theory and economic concepts with rate-distortion

• How much are customers willing to pay to get quality Q at bandwidth B? (Utilization curves).How much are customers willing to pay to get quality Q at bandwidth B? (Utilization curves).

• Optimal costs based on utilization curves and game theory.Optimal costs based on utilization curves and game theory.

• What does today’s licensing look like?What does today’s licensing look like?

• Games Games

Going forward: How should tools Going forward: How should tools be priced?be priced?

Distortion

Rate

T1 + T2: cost3(R)

T2: cost2(R)T1: cost1(R)Free

R

free regionD,R not

achievable