ARMOR- A djusting R epair and M edia Scaling with O perations R esearch for Streaming Video

1

ARMOR-Adjusting Repair and Media Scaling with Operations Research for Streaming VideoPhD Candidate: Huahui Wu - Computer Science, Worcester Poly. Inst.Committee: Prof. Mark Claypool - Computer Science, Worcester Poly. Inst. Prof. Robert Kinicki - Computer Science, Worcester Poly. Inst. Prof. Craig Wills - Computer Science, Worcester Poly. Inst. Prof. Wu-chi Feng – Computer Science, Portland Stat Univ.

Ph.D. Defense 205/01/2006

Acknowledge Prof. Claypool and Prof. Kinicki Prof. Wills Prof. Wu-Chi Feng from Portland State Univ. Faculty/Staff of Computer Science Dept., WPI Jae Chung, Feng Li, Mingzhe Li and Rui Lu User study participants Attendees today My Family


Introduction - Motivation

Net wor k Cl oud

Ser ver Cl i ent

Video Frames

Repair by Forward Error Correction (FEC)


Operations Research Concept

More Repair and More Scaling

Vid

eo Q

ualit

y

Optimal Point Adjusting Repair

and Media Scaling– Given Network and

Application Environment – For each valid FEC and

scaling combination, measure the video quality

– Find the optimal point


The DissertationRepair (FEC) Scaling Approach Publications

Media Independent

No Scaling [NOSSDAV 03] [PV 03 poster]Temporal Scaling

[TOMCCAP 05] [ACM MM 06 in Reviewing]

Quality Scaling

[NOSSDAV 05] [ACM MM 04 Demo]

Combination [NOSSDAV 06]

Media Dependent

Quality Scaling

M: Video Quality ModelA: Optimization AlgorithmU: User Study

S: SimulationI: Implementation

M A U S I

M A U S

M A

M A

M A


Outline Introduction Background Models Algorithms User Study Implementation Contributions Conclusions


Video Compression Standard

I 0 B 00 B 01 P 1 B 10 P 2 I 0

MPEG– Popular compression standard– Intra-compression and inter-compression– Three types of frames: I, P and B– Group Of Pictures (GOP)

ARMOR models MPEG dependencies


Forward Error Correction (FEC) Media-Independent FEC

– Reed-Solomon codes [Reed+ 60] ARMOR models benefits of FEC for frame transmission

1 2 K O rig in a l V id eo F ram e

1 2 K K + 1 N A fte r A d d in g F E C P ack e ts

1 K K + 1 N

1 2 K

A fte r N e tw o rk T ran sm iss io n(so m e p ack e ts a re lo s t)

A fte r R eco n s tru c tio n(w ith an y K co rrec tly rece iv ed p ack e ts)


Media Scaling Sacrifice data to fit the capacity Temporal Scaling (TS)

– Pre-Encoding Temporal Scaling– Post-encoding Temporal Scaling

I B B P B B BP B I

I B P B P B I

Raw Pi ct ur es

Af t er Encodi ng

Af t er Scal i ng

Ph.D. Defense 1005/01/2006

Media Scaling (cont.) Quality Scaling

– MPEG uses quantization in coding to save bits– Quantization Value (1~31)– For example: original data = 23, 13, 7, 3

ARMOR models both Temporal Scaling and Quality Scaling

Quantization Value

After Quantization

After DeQuantization

3 7, 4, 2, 1 21, 12, 6, 36 3, 2, 1, 0 18, 12, 6, 0

12 1, 1, 0, 0 12, 12, 0, 0

Ph.D. Defense 1105/01/2006

Video Quality Measurements Subjective Measurement

– User study, expensive, not practical Objective Measurements

– Playable Frame Rate (R)• Good for Temporal Scaling, not for Quality Scaling

– Peak Signal Noise Ratio (PSNR)• Good for Quality Scaling, not for Temporal Scaling

– Video Quality Metric (VQM) [Pinson+ 04]• By Institute for Telecommunication science• Extracts 7 perception-based features

– Only one for frame losses• Report a distortion value from 0 (no distortion) to 1 (many)

ARMOR uses both R and VQM A comprehensive user study is included

Ph.D. Defense 1205/01/2006

Outline Introduction Background Models

– Streaming Bitrate Model (cost)– Video Quality Model (benefit)

Algorithms User Study Implementation Contributions Conclusions

Ph.D. Defense 1305/01/2006

Parameters and Variables

Net wor k Cl oud

ARMOR Cl i ent

Video Frames

Repair by Forward Error Correction (FEC)

FBPBPI RNNSSS ,,,,,

Tstp RTT ,,,QSTS

BFPFIF

llSSS

,,,

Ph.D. Defense 1405/01/2006

Streaming Bitrate Model Total streaming bitrate, including video packets and FEC packets:

where G is the constant GOP rate

NPD and NBD are the numbers of transmitting P and B frames depending on Temporal Scaling level lTS

))()()(( BFBBDPFPPDIFI SSNSSNSSGB

)1( BP

F

NNRG

Ph.D. Defense 1505/01/2006

Two distortion factors– Frame Loss

• Caused by Temporal Scaling and network packet loss• Appears jerky in the video playout• Measured by Playable Frame Rate

– Quantization Distortion• Caused by a high quantization value with Quality Scaling• Appears visually as coarse granularity in every frame• Measured by VQM

Overall Quality– Distorted Playable Frame Rate

Video Quality Model - Overview

D

R

RDRD )1(

[Wu+ 05 TOMCCAP]

Ph.D. Defense 1605/01/2006

Playable Frame Rate (R) Frame Successful Transmission Probability

– Where Frame Size Frame Dependencies

Total Playable Frame Rate

FF

SS

Si

iSSiF ppiSS

q**

*

** ])1([ )(***

I 0 B 00 B 01 P 1 B 10 P 2 I 0

)),,(),,,(,,( BFPFIFBPITSBPI SSSSSSlpRRRRR

***

ˆ QSlSS

)),,(,,,( BFPFIFQSTS SSSllpR

Ph.D. Defense 1705/01/2006

Quality scaling distortion varies exponentially with the quantization level

Distorted Playable Frame Rate

Distorted Playable Frame Rate (RD )

DQSlDD ˆ [Frossard+ 01]

)),,(,,())(1()1(

BFPFIFQSTSQS

D

SSSllpRlDRDR

Ph.D. Defense 1805/01/2006

ARMOR Algorithm For each Repair and Scaling combination

• Estimate video frame sizes (SI, SP, SB)

– Compute streaming bitrate B and make sure it’s under capacity constraint T

– Use frame sizes and FEC amount to get successfully frame transmission rate (qI, qP, qB)

• Compute playable frame rate (R)• Estimate quality scaling distortion (D)

– Compute distorted playable frame rate (RD) Exhaustively search all FEC and Scaling

combination and look for the optimal quality

Ph.D. Defense 1905/01/2006


Ph.D. Defense 2005/01/2006

User Study Goals Accuracy of RD

– Correlation with user perceptual quality– Versus PSNR and VQM?

Temporal Scaling versus Quality Scaling– What are the differences?

Adjusted Repair (FEC) versus No Repair– Is Adjusted Repair an effective method for

increasing perceptual quality?

Ph.D. Defense 2105/01/2006

Video Clips Compare degraded clips to the original Original: 30 fps, no quality scaling Degraded: Combinations of 4 independent

factors (2 options each)– Video and Network environment

1. Video content: low motion (News) or high motion (Coastguard)

2. Packet loss rate: low loss (1%) or high loss (4%)– ARMOR Layer

3. Repair: adjusted repair or no repair4. Scaling: Quality Scaling or Temporal Scaling

24=16 combinations for evaluation

Ph.D. Defense 2205/01/2006

User Study Application

Two-week volunteer study

74 users, most CS undergraduate students

54321

[ITU-R BT.500-11]

Ph.D. Defense 2305/01/2006

Results – Video Quality Metrics (1)

User Score versus PSNR

Same as original clip

Much worse than original clip

Ph.D. Defense 2405/01/2006


User Scoreversus

VQM Score(1 – VQM distortion)

Ph.D. Defense 2505/01/2006


User Score versus

DistortedPlayableFrame Rate(RD)

Ph.D. Defense 2605/01/2006

Results – Scaling Methods

Temporal Scaling versus Quality Scaling

User Score ARMOR Prediction (Coastguard)

RD

30.0

22.5

15.0

7.5

0.0

Ph.D. Defense 2705/01/2006

Results – Repair Methods

Adjusted Repair versus No Repair

User Score ARMOR Prediction (Coastguard)

RD

30.0

22.5

15.0

7.5

0.0

Ph.D. Defense 2805/01/2006


Ph.D. Defense 2905/01/2006

Architecture

M P E G E n co d e r R ep a ir E n co d er U D P S en d er

N e tw o rk

A R M O RIm age S eq u e n ceR ep o sito ry

Scaling Level

MPEG Parameters

F ram esVideo and

RepairPackets

Network Parameters

S tre a m in g S e rv e r

M P E GP layer R ep a ir D eco d er U D P R ece iv e rD eco d ed

F ram es

V id eo an dR ep a ir

P ack e ts

S tre a m in g C lie n t

R awIm a ges

V id eo an dR ep a ir

P ack e ts

C lien tF eed b ac k

M ed ia S ca le r S ca ledF ram es

M P E GF ram es

P reP layer

Repair Amount

1 2 3 4

5678

1

22

3 3

Ph.D. Defense 3005/01/2006

Experiment SettingsNetwork (NistNet) Settings MPEG Encoder SettingstRTT 50 ms NP 3 frames per GOPS 1 Kbyte NB 8 frames per GOPp 0.01 to 0.04 RF 30 frames per sec

Video clip Paris – medium motion and details– two people sitting, talking, with

high-motion gestures– 1200 CIF (352x288) images– average I / P / B frame sizes:

24.24KB / 5.20 KB / 1.18 KB

Ph.D. Defense 3105/01/2006

ARMOR Analytical Results

RD

Results

ARMOR Measurement Results

RD

Ph.D. Defense 3205/01/2006

Contributions Derived a novel video quality metric

– Distorted playable frame rate Family of Video Quality Models with Repair and Scaling

– Modeled the playable frames rate– Modeled quantization distortion– Studied four ARMOR variants:

• Media Independent FEC with Temporal Scaling• Media Independent FEC with Quality Scaling• Media Independent FEC with Temporal Scaling and Quality

Scaling• Media Dependent FEC with Quality Scaling

Derived optimization algorithm to maximize the quality of streaming video

Conducted a comprehensive user study– Presented the high correlation between user score and

distorted playable frame rate Implemented a working ARMOR system

Ph.D. Defense 3305/01/2006

Conclusions Distorted playable frame rate has a high correlation

with user perceptual quality– Higher than PSNR or VQM

Adjusting repair improves video streaming quality significantly

– Better than fixed repair and no repair Quality Scaling is more effective than Temporal

Scaling– But when bandwidth is low and network loss is high,

Quality Scaling should be used with Temporal Scaling Media Dependent FEC is not as effective as Media

Independent FEC ARMOR can be implemented in a real video

streaming system and effectively improve streaming quality

34

ARMOR-Adjusting Repair and Media Scaling with Operations Research for Streaming VideoPhD Candidate: Huahui Wu - Computer Science, Worcester Poly. Inst.Committee: Prof. Mark Claypool - Computer Science, Worcester Poly. Inst. Prof. Robert Kinicki - Computer Science, Worcester Poly. Inst. Prof. Craig Wills - Computer Science, Worcester Poly. Inst. Prof. Wu-chi Feng – Computer Science, Portland Stat Univ.

Questions?

Ph.D. Defense 3505/01/2006

Future Work Study of Variance of Playable Frame Rate

Study of dynamic Group of Pictures Study of different quantization values for different types of frames

Implementation of MIQS and MITQS systems

Study of other scaling methods User study of more videos

Ph.D. Defense 3605/01/2006

Playable Frame Rate [S4]

Playable Frame Rate (PFR) of I frames

II qGR

I 0 B 00 B 01 P 1 B 10 P 2 I 0

Ph.D. Defense 3705/01/2006

Playable Frame Rate [S4] (cont.)

PFR of P frames

I 0 B 00 B 01 P 1 B 10 P 2 I 0

PIP qRR 1

iPIP qRR

i

PPP qRRii

1

Ph.D. Defense 3805/01/2006

Playable Frame Rate [S4] (cont.)

PFR of B frames

PIBPB

PBPB

NiwhenqqRR

NiwhenqRR

iij

iij

101

I 0 B 00 B 01 P 1 B 10 P 2 I 0

Ph.D. Defense 3905/01/2006

Capacity Constraint TCP-Friendly Flow [Padhye+ 00]

Bottleneck Capacity– Dial up: 56 Kbps– DSL: 1.5 Mbps (Verizon)– Cable Modem: 3 Mbps/384 Kbps (Charter)– Video is often larger than 1.5 Mbps

)321()8

33(3

2 2ppptpt

sT

RTORTT

Ph.D. Defense 4005/01/2006


User Score versus

PlayableFrame Rate(R)

Ph.D. Defense 4105/01/2006

Lines of Codes

Ph.D. Defense 4205/01/2006

Related Work DAVE (Delivery of Adaptive Video)

– Describes video content– Supports physical and semantic adaptation– Does not consider capacity constraint and

media repair Priority Drop

– Implemented SPEG for media scaling– Uses TCP as transmission protocol

Ph.D. Defense 4305/01/2006

Media Scaling (cont.) Quality Scaling (QS)

– Adaptive Quantization Level

24KB, 10KB, 5KB

Ph.D. Defense 4405/01/2006

System Layers ParametersMPEG

ARMOR

Network

FBPBPI RGNNSSS ,,,,,,

QSTSBFPFIF llSSS ,,,,

Tstp RTT ,,,

System Layers and Parameters

Documents

ARMOR- A djusting R epair and M edia Scaling with O perations R esearch for Streaming Video