ECE 5578 Multimedia Communication Lec 10a: Quality of ......QoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS ... WebRTC, and QUIC Congestion Measure and Modeling

Spring 2020: Venu: Haag 315, Time: M/W 4-5:15pm

ECE 5578 Multimedia Communication

Lec 10a: Quality of Experience (QoE)

Zhu LiDept of CSEE, UMKC

Office: FH560E, Email: [email protected], Ph: x 2346.http://l.web.umkc.edu/lizhu

Z. Li: ECE 5578 Multimedia Comm, 2020 p.1

slides created with WPS Office Linux and EqualX LaTex equation editor

Outline

QoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS Objective QoE metrics Perceptual QoE metrics

Summary


Video Communication System Coverage

Tentative Topics: QoE Metrics: Referenced, Light Reference, and Reference-less QoE

metrics MPEG Systems: File Format (MP4Box), Streaming Solution

(DASH.js), MMT Media Transport: RTP/RTSP, HTTP/WebSocket, WebRTC, and QUIC Congestion Measure and Modeling in Media Networking Video over Multiple Access Networks : Resource Pricing Solution,

DP+Lagrangian Framework FEC P2P Systems Content Identification and Info Centric Networking


QoS

QoS – Quality of Service A network centric metric Measuring the delay, loss, throughput, ..etc Does not directly translate into user experiences Typically characterized by the packet arrival and departure curves Buffer size: b(A,D, t), delay, d(A,D, t)


QoE

Quality of Experience A user centric metric, how a piece of audio/visual signal delivered by

the network looks/feels Usually a function of QoS, content, and viewing conditions

Fig credit: Touradj Ebrahimi, EPFL, ACM MM 2009


ITU STRUCTURE

3 Sectors:

• Standardization (ITU-T): promotes enabling technical, policy and regulatory frameworks to boost ICT development

• Radiocommunication (ITU-R): coordinates the shared global use of radio spectrum and geostationary satellite orbit

• Development (ITU-D): works to improve telecommunication infrastructure in the developing world


ITU-T Structure and organization

WTSA

TSAG

Study Group x

Working Party 1/x

Question 1/1

Working Party 2/x

Question 1/2

Working Party 3/x

Question 1/3

Study Group y

Working Party 1/y

Question 1/1

Working Parties …

Study Groups …


Study Group 9 OverviewLead Study Group on integrated broadband cable and

television networksResponsible for studies relating to: use of telecommunication systems for contribution, primary

distribution and secondary distribution of television, sound programmes and related data services including interactive services. use of cable and hybrid networks, primarily designed for television

and sound programme delivery to the home, as integrated broadband networks to also carry voice or other time-critical services, video on demand, interactive services, etc.


SG9 QoE metrics work

Rec. # Name Qu Title Timing

J.249 J.redref Q2/9 Perceptual video quality measurement techniques for digital cable television in the presence of a reduced reference

2009

J.340 J.ra-psnr Q2/9 Reference Algorithm for Computing Peak Signal to Noise Ratio (PSNR) of a Video Sequence with Constant Spatial Shifts and a Constant Delay

2009

J.341 J.vqhdtv-fr Q2/9 Objective perceptual multimedia video quality measurement of HDTV for digital cable television in the presence of a full reference

2010

J.bitvqm J.bitvqm Q12/9 Hybrid perceptual bitstream video quality assessment

2013

J.av-dist J.av-dist Q12/9 Methods for subjectively assessing audiovisual quality of internet video and distribution quality television, including separate assessment of video quality and audio quality

2013

J.3D-disp-req

J.3D-disp-req

Q2/9 Display requirements for 3D video quality assessment

2013


Study Group 12 Overview

‘Performance, QoS and QoE’Responsible for Recommendations on performance, quality of

service (QoS) and quality of experience (QoE) for the full spectrum of terminals, networks and services ranging from speech over fixed circuit-based networks to multimedia applications over networks that are mobile and packet based. Included in this scope are the operational aspects of performance, QoS and QoE.

A special focus is given to interoperability to ensure end-to-end users' satisfaction.

SG 12 is the Lead SG on QoS and Performance


SG12 Visual Quality Assessment

Rec. # Name Qu Title Timing

P.1201 P.NAMS Q14/12 Parametric non-intrusive assessment of audiovisual media streaming quality

2012-09

P.1202 P.NBAMS Q14/12 Parametric non-intrusive bitstream assessment of video media streaming quality

2012-09

G.1080 G.IPTV-QoE

Q13/12 Quality of experience requirements for IPTV services

2008

G.1050 Q13/12 Network model for evaluating multimedia transmission performance over the Internet Protocol

2011

G.OMVAS G.OMVAS Q13/12 Opinion model for video and audio streaming applications

2014

P.1401 P.STAT Q9/12 Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models

2012


Video Quality Experts GroupFounded 1997 ITU-T SG 12, SG 9, and ITU-R 11E (now 6C) experts Web ( www.vqeg.org );

First VQEG meeting (Turin, Italy 1997)

Primary mission: Advance the field of video quality

assessment by investigating new and advanced subjective and objective measurement techniques

VQEG does not develop or publish standards Conducts tests and reports results to

ITU and other standards organizations Tests are conducted using specifically

defined procedures (i.e., carefully developed test plans).


VQEG ProjectsCompleted

FRTV I & II (5 ITU Recommendations) Multimedia I (7 ITU Recommendations) RRNR (3 ITU Recommendations) HDTV I (2 ITU Recommendations)

Active: 3DTV (3 ITU Recommendations in progress) Joint Effort Group—JEG-Hybrid Hybrid Perceptual/Bitstream (3 ITU Recommendations in progress) Multimedia II—MM2 (1 ITU Recommendation in progress) Quality for Recognition Tasks—QART (Public Safety, Surveillance

Applications) (1 ITU Recommendation)Ramping up

High Dynamic Range Video—HDR HDTV Phase II—HDTV2 Monitoring of Audio Visual Quality by Key Indicators—MOAVI Real-Time Interactive Communications Evaluation—RICE


VQEG/Standardization Process

ITU-T &SG9, SG12, SG16

ITU-R , WP6C

Other Standards OrgsATIS, IEEE, ETSI, MPEG

VQEG

Results

Industry & Academia

Standards & Reports


Useful Linkshttp://www.itu.int/ITU-T/index.htmlhttp://www.itu.int/ITU-T/studygroups/com09/index.asphttp://www.itu.int/ITU-T/studygroups/com12/index.asphttp://www.itu.int/en/ITU-T/publications/Pages/recs.aspxhttp://www.itu.int/ITU-R/index.asp?category=study-

groups&rlink=rwp6c&lang=en

VQEG: http://www.vqeg.org


Outline

About the Project Sign UpQoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS Objective QoE metrics Perceptual QoE metrics

Summary


QoE Subjective Evaluation

MOS – Mean Opinion Score, an user study based quality evaluation

A subjective tests aiming at producing MOS is a delicate mixture of ingredients and choices:• Test/lab environment• Test material• Test methodology• Analysis of the data

credit: Touradj Ebrahimi, EPFL, ACM MM 2009


Test/lab environmentType of Monitors/Speakers and other test equipments

Lighting /Acoustic conditions

Laboratory architecture, background, …

Viewing distance /Hearing position

…


Test material

Meaningful content for the envisaged scenario/application Typical content Worst case content …

p01 p06 p10 bike cafe woman


Test methodology (I)

Single Stimulus (SS)

Non-categorical adjectival or numerical grading scale

5 Excellent 4 Good

3 Fair

2 Poor

1 Bad

5 Imperceptible

4 Perceptible but not annoying

3 Slightly annoying

2 Annoying

1 Very annoying

100

0

Excellent

Bad

Categorical adjectival grading scale: Categorical numerical grading

scale:

“Rate from 1 to 11”


Test methodology (II)

Double Stimulus Impairment Scale (DSIS) Categorical Impairment Grading Scale

5 Imperceptible

4 Perceptible but not annoying

3 Slightly annoying

2 Annoying

1 Very annoying


Test methodology (III)

Double Stimulus Continuous Quality Scale (DSCQS)

Sample 1 Sample 2

Non-categorical adjectival or numerical grading scale:

Sample 1 Sample 2

100

0

Excellent

Bad

100

0

Excellent

Bad


Test methodology (IV)

Stimulus Comparison (SC) Categorical adjectival comparison scale:

“same or different”

much worse

worse

slightly worse

the same

slightly better

better

much better

Non-categorical judgement:

Much worse

Much better


Test methodology (V)

Single Stimulus Continuous Quality Evaluation (SSCQE)

(Very annoying)

(Imperceptible)


Test methodology (VI)

Simultaneous Double Stimulus for Continuous Evaluation (SDSCE)

(Much better)

(Much worse)(Reference) (Test sequence)


Analysis of the MOS data

Improve MOS data quality• Scores distributions across subjects (testing people) is assumed

to be close to normal distribution• Outlier detection and removal• Mean Opinion Scores (MOS) and 95% confidence intervals (CIj)

Nm

MOSN

i ijjå == 1

NNtCI j

j

sa ×-= ),2/1(

mij = score by subject i for test condition j.

N = number of subjects after outliers removal.

t(1-α/2,N) = t-value corresponding to a two-tailed t-Student distribution with N-1 Degrees of Freedom (DoF) and a desired significance level α (α=0.05 in our case, 95% confidence).

σ j = s t a n d a r d d e v i a t i o n o f t h e s c o r e s distribution across subjects for test condition j.


NN

MOSMOStBA

BAobs 22 ss +

-=

What is behind a MOS?

JPEG Image Quality Assessment Study:


Relationship between estimated mean values

• Hypothesis test to find out whether the difference between two MOS values are statistically significant

Two-sided t-test:

• T-statistic:

• Decision rule to reject H0:

BA MOSMOSH =:0

BAa MOSMOSH ¹:

NN

MOSMOStBA

BAobs 22 ss

+

-=

),2/1(),2/( NttORNtt obsobs aa -><


MOS hypothesis test

JPEG

200

0 4:

2:0

JPEG

20

00

4:4:

4JP

EG

JPEG

XR

MS

JPEG

XR

PSJP

EG 2

000

4:2:

0JP

EG

2000

4:

4:4

JPEG

JPEG

XR

MS

JPEG

XR

PSJP

EG 2

000

4:2:

0JP

EG

2000

4:

4:4

JPEG

JPEG

XR

MS

JPEG

XR

PS

JPEG 2000 4:2:0

JPEG 2000 4:4:4JPEG

JPEG XR MS

JPEG XR PS

JPEG 2000 4:2:0

JPEG 2000 4:4:4JPEG

JPEG XR MS

JPEG XR PS

0.25 bpp

0.50 bpp

0.75 bpp

1.00 bpp

1.25 bpp

1.50 bpp

6

5

4

3

2

1

0Number of times

H0 is rejected


Outline

About the Project Sign UpQoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS Objective QoE metrics Perceptual QoE metrics

Summary


Objective QoE metrics

• Subjective tests are time consuming, expensive, and difficult to design

• Objective algorithms, i.e. metrics, estimating subjective MOS with high level of correlation are desired• Full reference metrics

• No reference metrics

• Reduced reference metrics

Input/Reference signal

Output/Processed signal

signalprocessing

FR METRIC



signalprocessing



signalprocessing

Features extraction

RR METRIC


PSNR - Peak Signal to Noise Ratio

PSNR def:

Widely used because of its simplicity and ease in formalizing optimization problems!

For image and video data (Y component), a correlation of circa 80% reported when compared to subjective MOS evaluation

åå= =

-=M

1y

N

1x

2ba y)](x,Imy)(x,[Im

MN1MSE

where:

M, N = image dimensions Ima , Imb = pictures to compare B= bit depth


PSNR for color images/video

Multiple channel info, several options to compute metric Weighted PSNR

Weigthed MSE:

Weighted Pixel Value PSNR:

WPSNR = w1PSNR1 + w2PSNR2 + w3PSNR3

)MSEwMSEwMSE(w1)(210log

332211

2B

10 ++-

=WPSNR_MSE

WPSNR_PIX( ) ( )[ ]åå

= =

++-++

-= M

y

N

xbbbaaa

B

)y,x(Imw)y,x(Imw)y,x(Imw)y,x(Imw)y,x(Imw)y,x(ImwMN

)(log

1 1

2332211332211

2

10 11210


PSNR for color images/video

PSNR in RGB vs YCbCro n R

component:

bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)

PSN

R(d

B)

o n G component:

o n B component:

PSN

R (d

B)

PSN

R (d

B)

on Y’ component: on Cb component:

on Cr component:

bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)


• MotivationSimulate relevant early HVS components

Referencesignal

Distortedsignal

Quality/DistortionMeasure

ChannelDecomposition

ErrorNormalization

.

.

.

ErrorPooling

Pre-processing

.

.

.

• Key featuresChannel decomposition linear frequency/orientation transforms

Frequency weighting contrast sensitivity function

Masking intra/inter channel interaction

No separation of objects and illuminance !

/1

,

= åå

l kkleE

Standard IQA Model: Error Visibility


• How to define structural information?

• How to separate structural/nonstructural information?

PhilosophyPurpose of human vision: extract structural information

HVS is highly adapted for this purposeEstimate structural information change

Classical philosophy New philosophyBottom-up Top-down

Predict Error Visibility Predict Structural Distortion

Structural Similarity

New Paradigm


Structural Similarity Measurement

The SSIM system: Image is a product of illuminance and object reflectance Try to separate the object structural info from the illuminance Full reference solution, compare image block x with y, have 3

components:o Luminance, contrast, and structure comparison


[1]

[2]

[3]

�(�,�)

�(�,�)

�(�,�)

Luminance Comparison – l(x,y)

Basic Operations Operate on image regions (can be block, or circular) For each channel, compute the region mean and variance for block x

and its reference y:

Luminance comparison function: L=dynamic range, 2B, e.g, 256 for B=8 K1, small const << 1.


�� =1� �

�=1

�� =

1� �

�=1

��

�(�, �) =2�� + (��)�

�� +��

� + (��)

Contrast Comparison – c(x,y)

Compare the illuminance dynamic range and behavior of two blocks Based on the variance of the channel

Contrast function:


�(�,�) =2�� + (��)�

�� +��

� + (��)�

�� = �1

�− 1 ��=1

�

(�� −��)��/�

�� = �1

�− 1 ��=1

�

�� −��

�/�

Structural Comparison – s(x,y)

Luminance subtraction and variance normalized comparison, supposedly removed illuminance factor, compare objects:


�(�,�) =�� + (��)�

�� + (��)�

�� = �1

�− 1 ��=1

�(�� −��)(�� −��) �

SSIM Index

Structural Similarity Measure (SSIM) General factorized form of power a, b, c:

Typically used: a=b=c=1, �� = �� , let

Then:


�� G(�,�) = [�(�,�)��(�,�)��(�,�)�]

�� G(�,�) =(2�� +��)(2�� +��)

(�� +��

� +��)(�� +��

� +��)

�� = (��)� �� = (��)�

Structural Similarity (SSIM) Index in Image Space

i

k

j

x

xi + xj + xk = 0

x - x

O

luminancechange

contrastchange

structuralchange

xi = xj = xk

),(),(),(),( yxyxyxyx sclSSIM ××=

122

12),(

CC

lyx

yx

++

+=

yx

222

22),(

CC

cyx

yx

++

+=

ssss

yx

3

3),(C

Cs

yx

xy

+

+=

sss

yx

[Wang & Bovik, IEEE Signal Processing Letters, ’02][Wang et al., IEEE Trans. Image Processing, ’04]


original image

JPEG2000 compressed

image

absolute error map

SSIM index map

Z. Li: ECE 5578 Multimedia Comm, 2020

43

original image

Gaussian noise

corrupted image

absolute error map

SSIM index map


44

original image

JPEG compressed

image

absolute error map

SSIM index map


45

MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.989

MSE=215, MSSIM=0.671 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723

Demo ImagesZ. Li: ECE 5578 Multimedia Comm,

2020

46

MOS(PSNR) MOS(MSSIM)

0.4 0.5 0.6 0.7 0.8 0.9 10

10

20

30

40

50

60

70

80

90

100

MSSIM (Gaussian window, K1 = 0.01, K2 = 0.03)

MO

S

JPEG images JPEG2000 images Fitting with Logistic Function

15 20 25 30 35 40 45 500

10

20

30

40

50

60

70

80

90

100

PSNR

MO

S

JPEG images JPEG2000 images Fitting with Logistic Function

Dataset JP2(1) JP2(2) JPG(1) JPG(2) Noise Blur Error

# of images 87 82 87 88 145 145 145

PSNR 0.934 0.895 0.902 0.914 0.987 0.774 0.881

SSIM 0.968 0.967 0.965 0.986 0.971 0.936 0.944

Validation with MOS Scores SSIM is a better predictor than PSNR


Summary

QoE is an important component in the multimedia communication system Subjective QoE study: User study generate MOS scores Objective Metrics: compare communicated content as pieces of

signals Perceptive Metrics: try to model HVS and have a better approximation

of MOS

Next Class: Reduced Reference, Non-Reference QoE metrics


Documents

ECE 5578 Multimedia Communication Lec 10a: Quality of ......QoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS ... WebRTC, and QUIC Congestion Measure and Modeling